Voice AI: ASR, TTS & Realtime

Question 1

Order the stages of a cascaded voice agent:

Accepted Answer

VAD → ASR → LLM → TTS — Detect end of speech, transcribe it, generate the reply, then speak it — with every stage streaming into the next.

Question 2

What does VAD do?

Accepted Answer

Detects whether anyone is currently speaking — Voice activity detection runs continuously and feeds both endpointing (user finished) and barge-in (user interrupted).

Question 3

"Endpointing" is the decision that:

Accepted Answer

The user has finished their turn — Too eager cuts people off mid-thought; too patient adds dead air. Semantic endpointing asks the LLM whether the utterance sounds complete.

Question 4

Barge-in handling requires:

Accepted Answer

Stopping speech synthesis within ~100 ms once the caller starts speaking — Users forgive wrong answers faster than being talked over — yield the floor instantly and treat the new audio as the next turn.

Question 5

Beyond roughly what silence does a voice reply start to feel broken?

Accepted Answer

Roughly 1–2.5 s, well past typical conversational pause norms — Human turn-gaps run ~250 ms; ~1 s is the conversational bar and by ~2.5 s callers assume the line dropped.

Question 6

The key latency metric for voice UX is:

Accepted Answer

Time to first audio — Perception is about when speech starts, not ends — ship the first syllable fast and the user grants you seconds for the rest.

Question 7

Why must every cascade stage stream rather than run to completion?

Accepted Answer

Each stage adds delay; overlapping stages keeps total under 1.2 s — Partial ASR prefills the LLM, the first sentence feeds TTS mid-generation — queued stages blow the budget even if each is fast.

Question 8

A "speculative first phrase" ("Sure —") exists to:

Accepted Answer

Buy LLM generation time by speaking before the answer is ready — A cheap acknowledgment buys hundreds of milliseconds of perceived responsiveness while the answer generates.

Question 9

Native speech-to-speech models differ from cascades by:

Accepted Answer

Processing and generating audio tokens within a single model — One model listens and speaks — no text hop — which is what preserves prosody and cuts latency.

Question 10

What does a cascade LOSE at each text hop?

Accepted Answer

Vocal cues: tone, hesitation, emotion, and overlap — Text flattens how something was said; a native model can hear sarcasm or urgency and respond with matching prosody.

Key insight

A spoken turn, end to end

Test your knowledge

Order the stages of a cascaded voice agent:

The short version

In more depth

Voice AI: ASR, TTS & Realtime

Cascaded pipeline

Native speech-to-speech

The numbers that matter

Interview Questions

References & Further Reading

Love this visualization?