A local, privacy-first desktop voice assistant for macOS. Hold a hotkey, speak, and get a spoken response — all running on your machine.
Built with Tauri 2 (Rust + React), powered by Whisper (STT) + Ollama (LLM) + Chatterbox (TTS).
You speak ──► Whisper transcribes ──► Ollama thinks ──► Chatterbox speaks back
(local STT) (local LLM) (local TTS)
- Hold Shift+Z — the orb appears and starts listening
- Release — your speech is transcribed via Whisper (runs natively in Rust, no server needed)
- Ollama generates a response — streamed sentence-by-sentence for low latency
- Each sentence is sent to Chatterbox TTS — audio plays back sequentially as chunks arrive
- Press Shift+X to dismiss the window
The app lives in the system tray — no dock icon, no window on launch. Just a floating orb that appears when you talk.
┌─────────────────────────────────────────────────────────┐
│ macOS System Tray │
│ ┌───────────────────────────────────┐ │
│ │ Tauri (Rust backend) │ │
│ │ ├── whisper-rs (native STT) │ │
│ │ ├── cpal (mic recording) │ │
│ │ ├── Ollama client (LLM + tools) │ │
│ │ ├── Chatterbox client (TTS) │ │
│ │ ├── RAG store (SQLite + embeds) │ │
│ │ └── Tool executors │ │
│ └───────────────┬───────────────────┘ │
│ │ events + invoke │
│ ┌───────────────▼───────────────────┐ │
│ │ React frontend │ │
│ │ ├── Orb UI (transparent overlay) │ │
│ │ ├── Chat bubbles │ │
│ │ └── Settings (tabbed window) │ │
│ └───────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
The response doesn't wait for the full LLM output. Instead:
- Ollama streams tokens
- Sentence boundary detection splits the stream (on
.!?followed by whitespace) - Each complete sentence is immediately sent to Chatterbox for TTS
- Audio chunks are emitted to the frontend with an index
- The frontend queues chunks and plays them in order
This means the user hears the first sentence while the LLM is still generating the rest.
The LLM has access to tools it can invoke mid-conversation. When you ask something that needs a tool, the flow is:
- Your message + tool definitions go to Ollama
- Ollama decides whether to call a tool or respond directly
- If it calls a tool, the backend executes it and feeds the result back
- Ollama can chain multiple tools (up to 5 rounds) before responding
- The final text response is streamed and spoken
| Tool | What It Does | How It's Used |
|---|---|---|
| Screenshot | Captures the screen via screencapture, sends the image to a vision model (llava) which describes what it sees |
"What's on my screen?" / "Is there an error?" |
| Read Clipboard | Reads clipboard text via pbpaste |
"What did I just copy?" / "Summarize what's in my clipboard" |
| Knowledge Search | Vector similarity search over ingested documents (SQLite + Ollama embeddings) | "What do my notes say about X?" |
| Open URL | Opens a URL in the default browser | "Open YouTube" / "Search Google for X" |
| Current Time | Returns the current date, time, and day of week | "What time is it?" / "What day is today?" |
| Running Apps | Lists all foreground applications via AppleScript | "What apps do I have open?" |
All tools can be individually enabled/disabled from the Settings > Tools tab.
The screenshot tool uses two models in sequence:
- Chat model (e.g. qwen3) reads your question and calls
take_screenshot(question: "Look for error messages")— it decides what to look for - Vision model (e.g. llava) receives the actual screenshot image + that question — it sees the screen and describes what's there
- The description goes back to the chat model, which formulates the spoken answer
Configure the vision model in Settings > Config > Vision Model (defaults to llava).
A local knowledge base backed by SQLite with vector embeddings via Ollama.
- Ingest — text is chunked (512 chars, 64 overlap, sentence-boundary-aware), each chunk is embedded via Ollama's
/api/embedendpoint, and stored in SQLite - Search — when the LLM calls
search_knowledge, the query is embedded and compared against all chunks using cosine similarity - Top results (above 0.3 threshold) are returned to the LLM as context
From Settings > Knowledge tab:
- Add Text — paste content with a source name
- Add File — ingest a text file from disk
- View Sources — see all ingested sources with chunk counts
- Delete — remove a source and all its chunks
Requires an embedding model pulled in Ollama (e.g. ollama pull nomic-embed-text).
The UI is a frameless transparent window with a glowing orb at the bottom-right of the screen. Chat bubbles stack upward like notifications.
| Color | State | Meaning |
|---|---|---|
| Blue (breathing) | Idle | Ready |
| Red (pulsing) | Listening | Recording your voice |
| Amber (spinning) | Processing | Transcribing speech |
| Purple (spinning) | Thinking | Waiting for LLM |
| Cyan (pulsing) | Speaking | Playing TTS audio |
| Dim red | Error | Something went wrong |
Accessible from the system tray menu. Three tabs:
- Whisper Model Path — path to a GGML whisper model file (e.g.
ggml-base.en.bin) - Ollama URL — where Ollama is running (default
http://localhost:11434) - Chat Model — Ollama model for conversation (e.g.
qwen3:4b) - Embedding Model — for RAG vector embeddings (e.g.
nomic-embed-text) - Vision Model — for screenshot description (e.g.
llava) - Chatterbox URL — TTS server address
- Voice — voice file name on the Chatterbox server
- System Prompt — personality and behavior instructions
Toggle each tool on/off. Changes take effect on the next conversation turn.
Manage the RAG knowledge base — ingest, view, and delete document sources.
All settings persist to ~/Library/Application Support/voice-assistant/config.json.
- Ollama running locally with at least a chat model pulled
ollama pull qwen3:4b # chat ollama pull nomic-embed-text # embeddings (for RAG) ollama pull llava # vision (for screenshots)
- Chatterbox TTS server running (OpenAI-compatible
/v1/audio/speechendpoint) - Whisper GGML model downloaded:
mkdir -p ~/.cache/whisper curl -L -o ~/.cache/whisper/ggml-base.en.bin \ https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin
- macOS (uses
screencapture,pbpaste,open, AppleScript)
# Install dependencies
npm install
# Development
cargo tauri dev
# Production build
cargo tauri build- Requires
CMAKE_OSX_DEPLOYMENT_TARGET=11.0for whisper-rs (handled viasrc-tauri/.cargo/config.toml) - First build compiles whisper.cpp and SQLite from source — takes a few minutes
macOSPrivateApi: trueis required intauri.conf.jsonfor transparent windows
| Layer | Technology |
|---|---|
| App framework | Tauri 2 |
| Backend | Rust |
| Frontend | React 19 + TypeScript + Vite |
| STT | whisper-rs (native, no server) |
| LLM | Ollama (local, streaming, tool calling) |
| TTS | Chatterbox (self-hosted, OpenAI-compatible) |
| Audio capture | cpal |
| Vector store | SQLite + Ollama embeddings |
| Global hotkeys | tauri-plugin-global-shortcut |
voice-assistant/
├── src/
│ ├── App.tsx # React app — Orb + Settings (URL-routed)
│ └── App.css # All styles — orb animations, settings UI
├── src-tauri/
│ ├── src/
│ │ ├── lib.rs # Tauri setup, tray, hotkeys, pipeline orchestration
│ │ ├── voice.rs # Whisper STT, Ollama streaming, Chatterbox TTS, tool defs
│ │ ├── tools.rs # Tool implementations (screenshot, clipboard, etc.)
│ │ └── rag.rs # RAG store — chunking, embedding, SQLite vector search
│ ├── Cargo.toml
│ ├── tauri.conf.json
│ └── capabilities/
│ └── default.json # Window permissions (main + settings)
└── package.json
| Shortcut | Action |
|---|---|
| Shift+Z (hold) | Push-to-talk — hold to record, release to process |
| Shift+X | Dismiss/hide the orb window |