Design a Mobile Voice Assistant: Wake Word and On-Device ASR

Updated July 26, 2026 · techinterview.org

Voice assistants (Siri, Google Assistant, Alexa) are deceptively complex mobile system design topics. Always-listening for a wake word, on-device automatic speech recognition for low-latency commands, intent parsing, action routing — all on a battery-constrained mobile device. The 2026 reality has shifted significantly toward on-device processing for both privacy and latency.

Functional requirements

• Detect wake word (“Hey Siri”, “OK Google”) in low-power mode. This is the always-on entry point, and interviewers want to hear that it runs on a dedicated low-power path rather than the main CPU. A match should only arm the rest of the pipeline, not trigger an action directly.
• Transcribe spoken command (ASR). Convert the audio captured after the wake word into text. Be ready to discuss streaming transcription, where partial results appear as the user speaks, versus waiting for end-of-speech before returning a final transcript.
• Parse intent (what the user wants). Map free-form text to a structured command with slots (action plus parameters), and define the fallback path for when no intent matches with enough confidence.
• Execute the action (set timer, send message, query knowledge). Route each intent to the right handler — a system service, a third-party app, or a knowledge backend — and handle actions that need confirmation (“send this to whom?”) or that fail partway.
• Speak response (TTS) where appropriate. Not every command needs audio; setting a timer can just update the UI. Call out when spoken feedback matters (hands-free, driving) versus when a silent on-screen confirmation is enough.

The always-listening problem

The phone listens 24/7 for the wake word. Concerns:

• Battery drain. Running a full ASR model or keeping the main CPU awake continuously would empty the battery in hours, so the always-on stage has to stay tiny and fixed-cost.
• Privacy (false triggers send audio to cloud). A misfire can stream real conversation off the device, so the design goal is to keep audio local until a wake word is actually confirmed.

Mitigations:

• Dedicated low-power audio chip listens for the wake word (Apple A-series and Google Tensor have these). The chip also buffers the last second or two of audio, so the start of the phrase isn’t clipped in the moment the main processor is waking up.
• Always-listening at <1mA. That is roughly two orders of magnitude below what the main application processor draws, which is why the detector lives on a separate always-on domain instead of the main CPU.
• Wake-word detection runs entirely on-device. No audio is sent anywhere during the listening phase, so a phone in airplane mode still hears its wake word.
• Only after wake-word match does the main processor wake up. A common answer is a two-stage cascade: a tiny model on the always-on chip triggers cheaply, then a slightly larger confirmation model on the main CPU rejects false positives before the microphone stream is opened for the command.

Wake word detection

Tiny ML model trained specifically for the wake phrase. Inputs: short audio buffer. Output: probability of match.

Threshold tuning: too low → false triggers; too high → user repeats themselves. Manufacturers tune this aggressively.

Personalization: the model can be fine-tuned on the user’s voice to reduce false-accept-on-other-people.

On-device ASR

Once wake word matches, the next ~10 seconds of audio are processed:

• Modern phones run Whisper-derived or proprietary ASR models on-device. These are quantized and shrunk to fit in memory and run within the phone’s power budget, which is the trade-off against the larger, more accurate models a server can host.
• Latency: sub-second for short commands. Streaming the transcript as the user talks hides most of that latency, so the assistant can start acting before the sentence is fully finished.
• Privacy: audio never leaves the device for most commands. This is the point interviewers usually push on — be specific that “most” means the on-device path handles common commands, and only defined exceptions escalate.

Cloud fallback: for long-form queries or low-confidence transcriptions, ASR can run in the cloud.

Intent parsing

Transcribed text → structured intent. Modern systems use LLM-based intent classification:

• “Set a timer for 5 minutes” → SetTimer(duration=5min)
• “What is the weather” → GetWeather(location=current)
• “Tell me a joke” → TellJoke

For complex queries, route to the LLM for general-purpose response.

Action execution

Each intent maps to an action:

• System actions (set timer, alarm, calendar). These hit built-in services directly and are the lowest-latency path, since there’s no third-party app to launch or knowledge lookup to wait on.
• App actions (play song in Spotify, send message in WhatsApp). The assistant hands a structured request to the app; the hard parts are disambiguating targets (“message which contact?”) and knowing which apps have registered a handler for the intent.
• Knowledge queries (search the web, summarize). These are the ones that usually leave the device, so be clear about what data goes to the cloud and how the spoken answer is condensed from a longer result.

App developers register actions via platform APIs (App Intents on iOS, Slices on Android).

Text-to-speech (TTS)

Modern TTS produces natural-sounding voices on-device. Latency: 100–300ms for typical sentences.

Voice cloning concerns: most platforms restrict to predefined voices to prevent misuse.

Privacy

The strongest argument for on-device:

• Audio never leaves the device for most commands
• No cloud has a transcript of your private conversations
• Users can opt-out of cloud processing entirely (with reduced functionality)

Battery

• Wake word: continuous, <1mA
• Active session: bursty, ~50–200mA for a few seconds
• Aggregate impact: typically <2% per day

Frequently Asked Questions

Why is on-device ASR better than cloud?

Lower latency, better privacy, works offline. Cloud was needed for accuracy a few years ago; on-device models have closed most of the gap.

Can I make my own voice assistant for an app?

Possible but not recommended. Use platform APIs (Siri/Google Assistant integration) for system-level voice control. Custom voice for in-app commands.

How does the wake word handle background noise?

The model is trained on noisy data. Modern phones have multi-microphone arrays for beamforming and noise rejection.