Your voice AI can understand every word, choose the right answer, and still feel broken. The failure often appears in the silence after the caller finishes: nothing happens, the caller starts speaking again, and the agent finally answers over them.
If you own this product, the useful question is not simply whether the model is fast. You need to know when the system decides the caller has finished, how long the first meaningful response takes to reach the caller, whether the agent can be interrupted, and what happens when any part of that sequence is wrong.
Measure the silence the caller actually experiences
API response time is not conversational latency. A speech recognition service can be fast while the agent still waits too long to commit the turn. A language model can produce its first token quickly while text-to-speech buffers a complete sentence. A media gateway can receive audio promptly while network jitter delays playback.
The caller experiences one continuous interval: the time between the last meaningful sound in their turn and the first audible evidence that the agent has taken the next turn. Instrument that interval end to end, then break it into stages.
- Last user speech: the final voiced audio frame belonging to the caller, excluding background noise.
- Endpoint candidate: the moment the system first suspects that the turn has ended.
- Endpoint commitment: the moment the system decides to respond rather than continue listening.
- Transcript ready: the point at which the agent has enough stable text to act.
- Response generation: the interval from the agent request to usable response text.
- Speech generation: the interval from usable text to the first playable audio chunk.
- Delivery: the path from outbound audio to playback on the caller’s channel.
Use the same trace identifier across those timestamps. Measure from a shared clock where possible; otherwise, synchronize the systems before comparing their timings. Without a connected trace, each provider can show an attractive dashboard while the caller still waits on the accumulated critical path.
You also need two versions of the top-line metric. Perceived start latency ends at the first audible agent audio. Meaningful start latency ends at the first audio that advances the caller’s task. A generic acknowledgment may improve the first measure without improving the second. Tracking both prevents a stream of empty filler from looking like a performance gain.
Human speakers commonly exchange turns across a gap of roughly 200 milliseconds. That makes conversational timing unusually sensitive: a small delay can change whether silence feels normal, hesitant, or broken. One voice AI vendor, Smallest.ai, classifies the following ranges as experiential anchors. These ranges are useful hypotheses, not universal service-level objectives; channel quality, language, task complexity, and the caller’s speaking pattern all affect the result.
| Reference band | Likely experience | What to do with it |
|---|---|---|
| Under 100 ms | Near-immediate response | Check that speed is not coming from premature endpoint decisions that interrupt the caller. |
| 150-250 ms | A short pause that can fit normal turn-taking | Use it as an initial target for simple, common turns, then validate it with actual callers. |
| 300-400 ms | Noticeable hesitation | Inspect repeat attempts and double-talk, not just the latency trace. |
| 500 ms-1 second | The caller is clearly waiting | Find the serial wait, or provide a truthful acknowledgment when real downstream work is unavoidable. |
| Over 1 second | Silence can be interpreted as failure | Do not accept this silently on a common path; add recovery behavior and investigate the tail. |
The ranges are deliberately not exhaustive, which is another reason to use them as design anchors rather than contractual truth. The ITU’s sub-150-millisecond recommendation for high-quality interactive voice is also not a ready-made bot response target. Network transport delay and the time an AI needs to detect, understand, and answer a turn are different measurements. Define the starting and ending event before putting any number into a requirement.
Key takeaways
- Measure from the caller’s last speech to audible playback, not from one internal API request to another.
- Track first audible audio and first meaningful audio separately so filler cannot game the latency metric.
- Treat endpoint detection, response generation, synthesis, transport, and playback as one trace.
- Review the latency distribution and its recurring tail by turn type; an overall average hides the interactions most likely to fail.
- Pair every speed metric with cutoffs, double-talk, repeat attempts, task completion, and escalation.
Treat end-of-turn detection as a product decision
Silence does not reliably mean that a person has finished. It can mean the caller is searching for a word, checking a number, taking a breath, or preparing a correction. Voice activity detection can tell you that speech energy stopped. It cannot, by itself, tell you that the conversational turn is complete.
This creates the central turn-taking tradeoff. Commit too slowly and the agent feels unresponsive. Commit too quickly and it answers during a natural pause. The second failure is often worse than the first because the system is no longer merely slow; it appears not to listen.
A single silence timeout is therefore a weak product policy. Use several forms of evidence when deciding whether to take the turn:
- Acoustic evidence: whether speech has stopped and whether the audio resembles a trailing or completed utterance.
- Lexical evidence: whether the provisional transcript ends on a complete thought, a conjunction, an unfinished number, or another continuation cue.
- Semantic evidence: whether the agent has enough information to satisfy the current intent.
- Dialogue state: whether the agent asked for a constrained slot, invited an open explanation, or is waiting for confirmation.
- Caller behavior: whether the speaker resumes, corrects a value, or explicitly indicates completion.
Make the endpoint policy conditional on the turn. A yes-or-no confirmation can often be committed faster than an account of a complicated support problem. A caller spelling a name or reading a sequence needs room to pause and self-correct. An open-ended discovery question should not inherit the same endpoint behavior as a constrained data field.
The implementation should preserve a candidate state between listening and commitment. During that state, the system can stabilize transcription and prepare likely downstream work without irreversibly taking the turn. If the caller resumes, the agent should cancel speculative work and return to listening. This is where cancelability matters as much as raw speed.
Barge-in is part of latency, not an optional feature
A fast agent that cannot stop talking is not conversational. Callers interrupt to correct a mistake, skip an explanation, answer an anticipated question, or redirect the task. Barge-in must work across the entire stack, not only at the audio player.
- Keep speech recognition active while the agent is speaking.
- Suppress or distinguish the agent’s own audio so it is not mistaken for caller speech.
- Stop outbound speech when a real interruption is detected.
- Cancel model generation, retrieval, or tool work that the interruption made obsolete.
- Track how much of the previous response was actually sent so the next turn does not assume the caller heard unsent content.
- Reconstruct the dialogue state before answering the interruption.
Do not treat every sound as a barge-in. A brief acknowledgment from the caller may mean continue, while a correction means stop. Test both. Otherwise, improving interruption sensitivity can create an agent that repeatedly abandons useful answers because of a backchannel, room noise, or echo.
Filler speech is not a substitute for this machinery. A short acknowledgment can be appropriate when a real tool operation has begun and the caller needs proof that the request was accepted. Repeating a generic preamble on every turn only moves dead time from silence into speech. It may even make barge-in harder because the caller has to interrupt language that adds no value.
Budget the critical path, then remove serial waits
Once the end of the turn is defined, latency becomes a budget. The budget must cover every stage that blocks the first meaningful audio. Assign each stage an owner, a start event, an end event, and an acceptable failure behavior.
| Stage | What to measure | Useful product and engineering levers | Common measurement trap |
|---|---|---|---|
| Audio capture and transport | User audio arrival, gaps, jitter, and channel metadata | Test real phone and device paths, place services near the media path, and avoid unnecessary buffering | Testing only with a clean local microphone |
| End-of-turn decision | Last user speech, endpoint candidate, and endpoint commitment | Adapt the policy by intent, language, transcript stability, and dialogue state | Counting the timeout as if it were model latency |
| Speech recognition | Partial transcript, stable transcript, and final transcript timing | Stream partial results and act on stable information without discarding correction handling | Reporting recognition speed after the complete recording has arrived |
| Context, retrieval, and tools | Every lookup and tool call on the blocking path | Run independent work concurrently, prefetch safe context, cache stable data, and define timeouts and fallbacks | Hiding tool time inside a single orchestration number |
| Language model | Request start, first usable output, and completion | Stream output, route by task, constrain unnecessary verbosity, and cancel obsolete generations | Optimizing full-response completion when speech can begin earlier |
| Speech synthesis | Text enqueue, first playable chunk, and subsequent audio continuity | Synthesize stable phrase chunks incrementally and avoid waiting for the entire answer | Measuring synthesis on prewritten text instead of live streamed output |
| Outbound playback | First chunk sent and the closest available client or gateway playback event | Control buffering, observe packet delivery, and test the actual destination channel | Stopping the clock when audio leaves the synthesis service |
Indicative estimates for a conventionally chained setup put speech-to-text at 100-300 milliseconds, model generation at 200-800 milliseconds, and text-to-speech at 100-400 milliseconds. Treat these vendor-provided ranges as an illustration of serial accumulation, not as a forecast for your product. Model choice, prompt and context size, utterance length, hardware, region, network, synthesis settings, and tool use can all change the result.
Do not add the median from each provider’s dashboard and call the result your expected median. Those component observations may come from different requests and different load conditions. Build end-to-end traces from the same turns. That shows the true critical path and exposes whether the slow tail comes from one stage, a retry, a cold path, or several small waits that only become visible when combined.
The largest gains usually come from removing unnecessary serialization. Stream caller audio into recognition instead of waiting for a completed recording. Begin safe context preparation while the transcript stabilizes. Run independent lookups concurrently. Send stable model output to speech synthesis without waiting for the full answer. Start playback from playable chunks. Cancel work as soon as a new turn invalidates it.
Each optimization has a correctness boundary. Starting from an unstable transcript can answer the wrong question. Synthesizing at arbitrary token boundaries can produce awkward prosody. Aggressive caching can return stale information. Parallel tool calls can perform unnecessary or conflicting work. Add these techniques one at a time and keep correctness, turn integrity, and recovery as guardrails.
Evaluate control, not the number of vendors
Owning an integrated voice stack can reduce handoffs and make cross-layer tuning easier. It is not automatically the right build decision. A composable stack can also work if its interfaces support streaming, timestamps, cancellation, interruption, regional placement, and useful failure signals.
During procurement, ask every provider to demonstrate the controls your turn manager needs:
- Can you access partial and stable transcription events, or only final text?
- Can endpoint behavior be tuned by language, channel, and intent?
- Can generation and synthesis be canceled immediately after an interruption?
- Does the service expose first-output and completion timestamps for each request?
- Can audio be streamed in and out without full-utterance buffering?
- Can you correlate provider events with your own end-to-end trace?
- What does the slow tail look like under realistic concurrency and in the regions you serve?
- What happens when a dependency times out after the agent has already acknowledged the request?
A low benchmark number without these controls can trap the product team. You may be able to prove that a component is fast while remaining unable to tune the conversation that callers experience.
Test conversations, not isolated response times
A clean request followed by a clean answer is the easiest case and the least revealing evaluation. Your acceptance suite should force the system to decide who owns the floor, recover from an incorrect assumption, and keep the caller oriented during slower work.
Build a reusable set of live and replayed scenarios from the turn shapes your product must handle:
- A crisp completion: the caller gives a short, complete answer and expects an immediate next turn.
- A mid-thought pause: the caller pauses after a continuation cue, then completes the sentence.
- A hesitation: the caller searches for a value without surrendering the turn.
- A self-correction: the caller replaces a name, date, or other field before finishing.
- An early barge-in: the caller interrupts near the beginning of the agent’s response.
- A late barge-in: the caller interrupts after receiving part of a substantive answer.
- A backchannel: the caller makes a brief acknowledgment that should not necessarily stop the agent.
- A slow tool path: a downstream operation succeeds after a meaningful wait.
- A tool failure: the dependency times out or returns an unusable result after the agent has taken the turn.
- A difficult audio path: the representative channel includes noise, echo, packet variation, or another real operating condition.
Recorded replay makes builds comparable, but it cannot show how a person adapts to delay. A caller may speak more slowly, repeat a question, shorten an explanation, or stop trusting the agent after an awkward exchange. Include live task sessions so you can observe those feedback loops.
For each eligible turn, capture the following measures with the same trace:
- Endpoint lag: the interval between the last user speech and the agent’s commitment to respond.
- False cutoff: a committed endpoint that occurs while the caller intended to continue.
- Delayed endpoint: an avoidable wait after the caller clearly surrendered the turn.
- Audible start latency: the interval to the first agent audio.
- Meaningful start latency: the interval to audio that advances the task.
- Interruption stop latency: how long the agent continues speaking after a valid barge-in.
- False interruption: a response abandoned because a non-interrupting sound was misclassified.
- Recovery success: whether the agent correctly incorporates the interruption and continues from the right state.
- User repair: repetitions, restarts, explicit checks that the agent is present, and corrections caused by the interaction.
- Task outcome: completion, escalation, abandonment, and answer correctness.
Segment the results before drawing a conclusion. Separate simple turns from tool-backed turns, first turns from later turns, clean channels from difficult channels, and the languages and speaking patterns represented in your caller population. A single overall number can improve because traffic shifted toward easier cases while the experience deteriorated for a valuable or vulnerable segment.
Listen to the traces around the slow tail and every false cutoff. The waveform should show when user speech ended, when the endpoint was committed, when each dependency ran, and when agent audio began. That review tells you whether to tune turn detection, remove a blocking call, change response construction, or repair transport. A latency chart alone cannot make that diagnosis.
If you run an A/B test, choose a task outcome as the primary decision metric and treat timing, correctness, cutoffs, and interruption recovery as guardrails. A faster variant that creates more double-talk is not a win. Neither is a smoother variant that begins speaking quickly but completes fewer tasks because its first response is generic.
Make the launch decision with a latency scorecard
One latency target cannot represent every conversational path. A simple confirmation, an open-ended explanation, and a tool-backed transaction have different work on the critical path. Define a budget for each high-value turn class and give each one an explicit recovery policy.
For a common, simple turn, the first meaningful response should be the target. For a genuinely slow operation, separate time to acknowledgment from time to completion. The acknowledgment should confirm what is happening only after the operation has actually started. It should not imply success, and it should not repeat on a loop while the dependency remains stuck.
| Launch dimension | Evidence to require | Reason to block or limit rollout |
|---|---|---|
| Common-turn timing | Production-like end-to-end traces meet the chosen target across the typical range and recurring tail | Frequent silent gaps cause callers to restart or check whether the agent is present |
| Turn integrity | Pauses, continuations, corrections, and constrained answers work for representative callers | Faster endpoint settings cut off valid speech |
| Barge-in | The agent stops, cancels obsolete work, and recovers the correct dialogue state | The system talks over corrections or continues using invalidated context |
| Slow operations | The caller receives a truthful status cue, a bounded recovery path, and a clear result | The agent fills time without advancing the task or acknowledging failure |
| Observability | A single trace explains endpoint, recognition, model, tool, synthesis, and playback timing | The team can see that a call was slow but cannot locate the delay |
| Outcome quality | Timing improvements preserve correctness, completion, and escalation behavior | The latency metric improves while the customer outcome deteriorates |
I would not approve a voice AI launch from a median latency number alone. The decision needs evidence that the system preserves the caller’s turn, handles the slow tail, accepts interruption, and remains diagnosable when a dependency degrades. Those are product capabilities, not infrastructure details.
Start with one common turn in the real delivery channel. Trace the caller’s last speech through the first meaningful audio, label every repeat and overlap, and identify the stage controlling the tail. Improve that path, then rerun it with a mid-thought pause, an interruption, and a slow dependency. When the system can keep the floor, surrender it, and recover it deliberately, you have evidence of a conversation rather than a fast sequence of APIs.
References
