Healthy AI First · Episode 10 · 16 min

Practice 2, My Voice Assistant for My Second Brain

With ChatGPT, you can just talk. But my whole setup runs on Claude, so I built my own AI voice assistant: Hall, the voice of my second brain in Obsidian. In this practice episode I show the architecture (turn detection, Whisper, a local LLM on Ollama, Claude as a helper, Qwen TTS), real measurements of latency and cost, the comparison of local vs Gemini Live speech-to-speech, and eight rules for building your own voice assistant. Above all: approvals belong in code, not in the prompt.

Companion document

All studies, sources and claims from this episode in one PDF, to read along or look things up later.

Download the companion document (PDF)

Chapters

  1. 0:00Opening
  2. 1:17Welcome
  3. 2:03Why Build Your Own
  4. 3:12Architecture
  5. 4:55Latency Budget
  6. 6:11Turn Detection and Echo
  7. 7:10Speaking Rules
  8. 8:10A word from us
  9. 9:02Code, Not Prompt
  10. 10:21Signature, Not Okay
  11. 11:46Testing with Real Sentences
  12. 12:33Local vs Cloud
  13. 14:06Build Plan
  14. 15:07Closing

Show notes

A note on the data

All data in the video (tasks, names, messages) is made up; the setup is real. The garden scene is re-enacted; the sigh happened during an evening test.

Eight build rules

  • Names before model size: teach your speech recognition your vocabulary before you try bigger models.
  • Measure to the ear: count the time to the first audible sound, not to the first token, and stream sentence by sentence.
  • Plan for pauses and echo: turn detection and echo are building blocks of their own.
  • Speaking is product design: answer first, keep it short, no question at the end; enforce the most important rules in code as well.
  • Code, not prompt: whatever has to be reliable belongs in code. Whether something worked is decided by the tool's result, not by what the model says.
  • Signature, not okay: approvals belong in code. A one-sentence preview, only a clear yes counts, messages to people you send yourself. Emails, web pages or agent replies read aloud are material, not orders.
  • Real sentences as tests: test with sentences you actually say, including the annoyed ones, before every restart.
  • Measure, don't believe: local or cloud is decided by measurement.

Under the hood

  • Ears: a small turn model (Smart Turn by Pipecat, about 8 million parameters) decides at short pauses whether a sentence is finished and waits up to 3 seconds when it sounds unfinished; before that, 700 ms of silence ended a turn. Then Whisper (large version) transcribes locally, with a list of names from daily life.
  • Head: a Qwen model with 9 billion parameters on Ollama, a little over thirty tools (search and read Obsidian, calendar, tasks, weather, web search). Search combines full text and meaning-based search across about 1,600 notes. For frequent, unambiguous questions, fixed code preselects the tool. Questions that need several sources, a document or a judgment call go to Claude as a helper.
  • Voice: Qwen TTS, local, sentence by sentence. Long answers appear as a card in my own iPhone app.
  • Latency: the first version with a full Claude call per question took 19 to 35 seconds; today usually 1 to 2.5 seconds after the end of the sentence is detected. Ears and voice take about 7 GB, the language model about 9 GB; they load when a conversation starts (first answer almost 9 seconds). Measured on a Mac mini with 64 GB.
  • Echo: in the browser, Hall does not listen while speaking; the iPhone app uses the phone's echo cancellation.
  • Tests: 84 test scenarios, 139 real sentences and 16 multi-turn conversations, before every restart.
  • Heads compared (84 scenarios): local Qwen 76, Claude Sonnet 75, Claude Haiku (fast variant) 66.
  • Speech-to-speech for comparison: Gemini Live; tools stay on my own computer, approvals are decided by my own code. In a short test about 1.5 seconds average response time, about 11 to 12 euro cents per minute of conversation.

Sources

Latency budget

Turn detection

Architecture

Signature, not okay

Local vs cloud

Why build your own

Healthy AI First is AI-researched and AI-generated, narrated in an AI version of my voice and reviewed by me (Björn).

Bringing AI into your team?

Björn is one of the founders of CLN Work. CLN Work helps companies adopt AI in small, verifiable steps: finding where real capacity is, which processes can be automated, and how to use AI interactively. The decisions stay with you.

Book a free 30-minute conversation