Voice AI latency: why sub-second matters and how to actually get there

In text, latency is a courtesy. In voice, it's the entire experience. Humans expect a reply within a few hundred milliseconds of finishing a sentence; stretch that past a second and the silence starts to mean something — the line dropped, I wasn't heard, is anyone there? Get latency wrong and it doesn't matter how good the agent's answer is, because the conversation has already broken.

The latency budget

A spoken reply travels through a chain, and every link spends time:

• Hearing the end of a turn. The system has to notice the caller stopped talking — too eager and it interrupts, too patient and it feels slow.
• Understanding the speech. Turning audio into something the agent can reason about.
• Reasoning and tool calls. The actual thinking, plus any lookups the agent needs to answer.
• Speaking. Turning the response back into natural audio.

Add those up naively and you're well over a second before the caller hears a word. The engineering challenge is that the budget is fixed by human expectation — you don't get to ask people to be more patient — so you have to hide the work, not just do it faster.

How you actually hit it

• Stream everything. Don't wait for a complete response to start speaking. Begin the reply as the first words are ready, and keep generating while the caller hears the start. This single technique buys more than any other.
• Overlap the stages. Understanding, reasoning, and speaking shouldn't run strictly one-after-another. The more they pipeline, the less the caller waits.
• Keep the reasoning path short. The fewer hops between hearing and speaking, the tighter the loop. Latency is a reason to keep the architecture lean, not just an afterthought.
• Handle interruptions instantly. When the caller barges in, everything in flight has to stop now. Snappy interruption handling is as much a part of "feeling fast" as raw response time.

Latency shapes the architecture

The deeper lesson is that sub-second voice isn't a setting you turn on — it's a constraint that shapes every decision, from how you stream to how you keep the reasoning-to-speech path lean. It's also why we run voice on the same agent brain as chat: the intelligence is shared, and the voice-specific work lives in the thin, latency-critical transport layer where it belongs.

Fast enough that the caller forgets they're talking to software — that's the bar. You can hear where we've landed on the voice page.