A voice agent is judged on a timescale no text product faces. A chat interface can think for three seconds and nobody minds, because a spinner is a socially acceptable way to say "working on it." Speech has no spinner. Silence is all the user gets, and silence from a device on a kitchen counter does not read as thinking. It reads as broken.
We built a real-time voice agent for a healthcare product running on a device in the user's home. One constraint shaped everything: the people using it speak slowly, pause in the middle of sentences, and cannot fall back to typing when the voice path fails. There is no keyboard escape hatch. If the conversation breaks, the product breaks.
The stack: LiveKit, AssemblyAI, GPT-4.1, Cartesia, Silero
The pipeline runs on the LiveKit Agents framework, which handles WebRTC transport, worker process lifecycle, and orchestration between recognition, model, and synthesis.
Speech to text is AssemblyAI universal streaming. Streaming is not optional here. A batch API that waits for the complete utterance adds the length of that utterance to your latency, so the slower someone speaks, the worse their experience gets, which inverts the product for these users. The production-grade streaming field is small, AssemblyAI and Deepgram being the two most teams evaluate, and either beats a batch API on felt responsiveness.
Reasoning runs on a GPT-4.1 class model, the fast variant. Voice punishes large models harder than text does, because the user listens to silence while tokens generate.
Speech synthesis is Cartesia. We started on a different provider and switched mid-build, which was a small change only because the framework isolates synthesis behind a plugin interface. Keep every provider behind an abstraction you control, even when you are sure of the choice.
Voice activity detection is Silero, running locally rather than as a network call, with a transformer turn detector on top. Tracing exports to Langfuse over OpenTelemetry, and Sentry carries errors and performance.
Where the latency actually goes in a voice agent
The obvious build is a straight line: audio in, speech to text, LLM, synthesis, audio out. Every hop is defensible alone. Added together they spend the conversational budget before you write a line of product logic. The trap is that most of the latency you can control is not in the model call, which is where teams instinctively look. It is in everything wrapped around it. Three changes moved our numbers most, and none of them touched the model.
Prewarm at the worker process, not the session. Loading the Silero VAD ONNX model costs 100 to 500 milliseconds, and paying that per session opens every conversation with dead air. LiveKit runs a prewarm function once per worker before any job is assigned, so the VAD model and the memory client are built ahead of time and every session inherits them warm.
Make session startup concurrent. Loading long-term memory is a round trip to a separate service, and the naive version awaits it, then sets up tracing, constructs the session, registers RPC handlers, and finally greets the user. We start the memory load as a task at the top and await it only where the first reply needs it. The rest of session construction runs inside that window for free.
Move writes off the critical path. Classifying and persisting something worth remembering is real work, and none of it needs to happen before the agent answers. Those writes are background tasks. The user never waits on a database round trip, and a failed write lands in the logs rather than in the conversation.
The shape of the fix is the same each time. Find the work sitting between the user's last word and the first audio back, then move it earlier, in parallel, or after.
Turn detection and endpointing: the tuning that defines the product
The setting that changes a voice agent most is when it decides you have stopped talking. Slightly wrong in one direction and it interrupts people mid-thought. Slightly wrong in the other and it sits there while the user wonders whether it heard them.
Two systems cooperate. Voice activity detection answers the acoustic question of whether there is speech right now. The turn detector answers the semantic question of whether the sentence sounds finished. Running both means a trailing "and, um..." keeps the floor with the user even though the audio went quiet.
We ended up with a 400 millisecond minimum endpointing delay and a 4 second maximum. The maximum is the interesting one: when the turn detector suspects the user is not done, we wait up to four full seconds before assuming otherwise. That is an eternity in a consumer assistant and exactly right for someone pausing to find a word. On the VAD side, a 40 millisecond minimum speech duration, a 350 millisecond silence window, an activation threshold slightly below default at 0.42, and 600 milliseconds of prefix padding so the front of an utterance is never clipped.
Barge-in is tuned the opposite way. One word and 10 milliseconds of speech are enough to cut the agent off. When responses are long and the user has changed their mind, being easy to interrupt beats finishing the sentence politely.
None of these values are universal. They encode a belief about how this population speaks, arrived at by listening to recorded sessions rather than by reading documentation.
Keeping voice and touch in sync with one authoritative state machine
The device has a screen. Users can speak or tap, both have to work, and the two have to agree at all times. This is where multimodal voice products quietly fall apart: the interface keeps its own state, the agent keeps its own, and after a few turns they disagree in a way neither side can detect.
We made the backend authoritative and left the interface no state to disagree with. Agent to interface runs one way over a LiveKit data channel on a dedicated topic, carrying typed commands: show a panel, update its data, close it, change the view mode. Interface to agent runs over RPC, one registered method routing to per-feature handlers. Both directions converge on the same state machine, which validates every transition against an explicit table and raises on anything invalid.
A tap and a spoken command are therefore the same event by the time they reach the logic. There is no second code path, and an impossible state fails loudly in the logs instead of producing a screen that contradicts what the agent is saying.
Four failure modes that only appear in production
Each of these was invisible in local testing and obvious within a day of real use.
- The duplicate action race. Once voice and touch reach the same handler, a user can answer the same prompt twice, by speaking and then tapping, or by tapping again while the agent is still talking. The first version interrupted itself and answered twice. The fix has two parts: the handler returns a duplicate flag, and the call that interrupts the agent's speech moves to after the duplicate check. Ordering an interrupt before a validity check reads as correct in review and sounds terrible in the room.
- The metadata race at session start. The agent joins the room before the device's participant metadata is guaranteed to have arrived, and that metadata carries device identity, timezone, and which screen to open. Blocking forever is not an option and neither is crashing. We poll on a bounded window, a quarter second apart for up to six seconds, then continue in a degraded but working state and log it.
- Memory service degradation. The long-term memory preload tries a bulk fetch, falls back to semantic search, then falls back to an empty context. Empty is a legitimate outcome. An agent that fails to greet you because a memory service is slow is far worse than one that greets you without remembering your dog's name.
- Synthesis reading things never meant to be read aloud. A temperature of 66°F becomes unintelligible and a latency figure in ms gets read as a word. Every synthesized string now passes through a normalization step that expands unit abbreviations into spoken English. It is a twenty line function, invisible in code review, and you only find it by putting on headphones and listening to your own product.
Observability and evaluation
Through LiveKit's OpenTelemetry integration, every model call, transcription, synthesis call, and tool invocation exports to Langfuse as a span tied to a session and a device. Sentry carries the piece that matters most here: a transaction measuring time to first speech, opened at session start and closed when the first audio actually plays, with CPU profiling underneath. When a session opens slowly, the profile says which part of startup ate the time, which is a different question from which model call was slow.
The honest gap is that general purpose LLM tracing has no native concept of voice. It shows each turn between agent and user well, and gives you no first class metrics for what determines whether a voice product feels good: end-of-turn detection latency, wake word latency, the distribution of interruption events. We instrumented those ourselves, and it remains the clearest missing piece in observability tooling for voice.
Evaluation is scenario-based. Each scenario is YAML: a name, tags, an ordered list of user messages, and criteria in plain English. A runner plays it against a real session, captures the transcript along with the interface commands and tool calls produced, and hands all of it to a judge model that returns PASS or FAIL per criterion with a reason. Every scenario runs several times, because a voice agent is nondeterministic and one green run proves little.
The detail that earns the most is negative criteria. Alongside "responded warmly" we assert "did not call any function tools" and "did not send any interface commands." The most common regression in an agent with a rich tool surface is a correct answer accompanied by a tool call or a screen change that should never have fired, and only an explicit negative assertion catches it.
What we would keep
Treat startup as part of the latency budget, because the work done before the first word is as visible to the user as the model call and usually easier to fix. Tune turn detection to your actual users by listening to real sessions, since framework defaults encode assumptions about how fast people talk that may not describe yours. Give the backend sole authority over state, because the moment the interface keeps its own copy the two begin to drift invisibly. And write evaluations in the language of the product requirement, including the behavior that must not happen.
Voice is one of the few places in applied AI where engineering discipline still decides the outcome more than model choice does. The models are good enough. Whether the product feels alive comes down to milliseconds, state, and the willingness to sit and listen to the thing talk.
If you are building a real-time voice product and want the latency budget, the turn detection behavior, or the evaluation approach reviewed by people who have shipped one, book a call. The build side is on our LiveKit development page, and running it on your own servers on self-hosted LiveKit.
