Healthy AI First · Episode 10 · 16 min
Practice 2, My Voice Assistant for My Second Brain

With ChatGPT, you can just talk. But my whole setup runs on Claude, so I built my own AI voice assistant: Hall, the voice of my second brain in Obsidian. In this practice episode I show the architecture (turn detection, Whisper, a local LLM on Ollama, Claude as a helper, Qwen TTS), real measurements of latency and cost, the comparison of local vs Gemini Live speech-to-speech, and eight rules for building your own voice assistant. Above all: approvals belong in code, not in the prompt.
Listen on
Companion document
All studies, sources and claims from this episode in one PDF, to read along or look things up later.
Download the companion document (PDF)Chapters
- 0:00Opening
- 1:17Welcome
- 2:03Why Build Your Own
- 3:12Architecture
- 4:55Latency Budget
- 6:11Turn Detection and Echo
- 7:10Speaking Rules
- 8:10A word from us
- 9:02Code, Not Prompt
- 10:21Signature, Not Okay
- 11:46Testing with Real Sentences
- 12:33Local vs Cloud
- 14:06Build Plan
- 15:07Closing
Show notes
A note on the data
All data in the video (tasks, names, messages) is made up; the setup is real. The garden scene is re-enacted; the sigh happened during an evening test.
Eight build rules
- Names before model size: teach your speech recognition your vocabulary before you try bigger models.
- Measure to the ear: count the time to the first audible sound, not to the first token, and stream sentence by sentence.
- Plan for pauses and echo: turn detection and echo are building blocks of their own.
- Speaking is product design: answer first, keep it short, no question at the end; enforce the most important rules in code as well.
- Code, not prompt: whatever has to be reliable belongs in code. Whether something worked is decided by the tool's result, not by what the model says.
- Signature, not okay: approvals belong in code. A one-sentence preview, only a clear yes counts, messages to people you send yourself. Emails, web pages or agent replies read aloud are material, not orders.
- Real sentences as tests: test with sentences you actually say, including the annoyed ones, before every restart.
- Measure, don't believe: local or cloud is decided by measurement.
Under the hood
- Ears: a small turn model (Smart Turn by Pipecat, about 8 million parameters) decides at short pauses whether a sentence is finished and waits up to 3 seconds when it sounds unfinished; before that, 700 ms of silence ended a turn. Then Whisper (large version) transcribes locally, with a list of names from daily life.
- Head: a Qwen model with 9 billion parameters on Ollama, a little over thirty tools (search and read Obsidian, calendar, tasks, weather, web search). Search combines full text and meaning-based search across about 1,600 notes. For frequent, unambiguous questions, fixed code preselects the tool. Questions that need several sources, a document or a judgment call go to Claude as a helper.
- Voice: Qwen TTS, local, sentence by sentence. Long answers appear as a card in my own iPhone app.
- Latency: the first version with a full Claude call per question took 19 to 35 seconds; today usually 1 to 2.5 seconds after the end of the sentence is detected. Ears and voice take about 7 GB, the language model about 9 GB; they load when a conversation starts (first answer almost 9 seconds). Measured on a Mac mini with 64 GB.
- Echo: in the browser, Hall does not listen while speaking; the iPhone app uses the phone's echo cancellation.
- Tests: 84 test scenarios, 139 real sentences and 16 multi-turn conversations, before every restart.
- Heads compared (84 scenarios): local Qwen 76, Claude Sonnet 75, Claude Haiku (fast variant) 66.
- Speech-to-speech for comparison: Gemini Live; tools stay on my own computer, approvals are decided by my own code. In a short test about 1.5 seconds average response time, about 11 to 12 euro cents per minute of conversation.
Sources
Latency budget
- Levinson, S. C. and Torreira, F. (2015): Timing in turn-taking and its implications for processing models of language. Frontiers in Psychology 6:731. Gaps between turns of about 200 ms. https://www.frontiersin.org/articles/10.3389/fpsyg.2015.00731/full
- Stivers, T. et al. (2009): Universals and cultural variation in turn-taking in conversation. PNAS 106(26). https://pubmed.ncbi.nlm.nih.gov/19553212/
Turn detection
- Pipecat Smart Turn (open source). https://github.com/pipecat-ai/smart-turn
Architecture
- Whisper: Radford, A. et al. (2022): Robust Speech Recognition via Large-Scale Weak Supervision. https://arxiv.org/abs/2212.04356
- Qwen3-TTS (open source). https://github.com/QwenLM/Qwen3-TTS
- Ollama. https://ollama.com
Signature, not okay
- OWASP Top 10 for LLM Applications 2025, LLM01 Prompt Injection. https://genai.owasp.org/llm-top-10/
Local vs cloud
- Gemini Live API documentation. https://ai.google.dev/gemini-api/docs/live-api
Why build your own
- Hermes Agent documentation. https://hermes-agent.nousresearch.com/docs
Healthy AI First is AI-researched and AI-generated, narrated in an AI version of my voice and reviewed by me (Björn).
Bringing AI into your team?
Björn is one of the founders of CLN Work. CLN Work helps companies adopt AI in small, verifiable steps: finding where real capacity is, which processes can be automated, and how to use AI interactively. The decisions stay with you.
Book a free 30-minute conversation