The Voice AI Trend in 2026: Controllability vs Latency
Editorial note. Aissist.io has not officially launched a voice AI product. We are, however, very interested in this area. Everything below comes from two places: our team's prior experience building real-time conversational systems, and our own recent hands-on exploration of the current generation of voice models. We are publishing it as a working study rather than a product pitch, and we will update it as the technology moves.
TL;DR — The defining voice AI trend of 2026 is not a new model release — it is that every meaningful architecture decision still reduces to one tradeoff: controllability versus latency. A componentized ASR→LLM→TTS pipeline exposes an output at every seam — somewhere to filter, redact, enforce policy and log — and makes you fight for the milliseconds. An end-to-end speech-to-speech model gives you sub-second response and takes away almost everywhere you might have inserted control. Fine-tuning an end-to-end model is currently the most feasible way to get both, at the price of compute and engineering iteration. Filler words and typing sounds buy tolerance, not a genuine conversation. We expect end-to-end to win eventually — but only once it can accept context mid-conversation and emit more than voice.
Ask where voice AI is heading and you get a list of model launches. The more useful answer is structural: every meaningful decision in voice AI architecture still reduces to one tradeoff, controllability versus latency. You can have a system you can inspect, filter, govern and customize at every step — and it will be slow. Or you can have a system that answers in under a second and feels human — and you will have very few places left to insert control. As of August 2026, no production architecture escapes this. Choosing a voice AI architecture is really choosing which side of that tradeoff your use case can afford to lose on.
Why Voice AI Latency Is a Hard Constraint, Not a Nice-to-Have
Short answer: Humans expect a reply in about 200 ms. No voice agent is close, so the working rule is stay under two seconds and get under one if you can.
Text automation is forgiving. A support chat that takes four seconds to reply reads as thoughtful. Voice has no such slack, because humans have a hard-wired expectation for how fast a turn should come back. Conversation analysis puts the modal gap between speaker turns at roughly 200 milliseconds (Levinson & Torreira), with median gaps across ten languages falling between 0 and 300 ms (Stivers et al., 2009), as summarized in the Journal of Cognition.
No voice agent hits 200 ms. The working targets the industry has settled on are an order of magnitude looser: stay under two seconds, and get under one second if you can. Twilio's latency guide sets a mouth-to-ear turn gap target of 1,115 ms with an upper limit of 1,400 ms (Twilio, November 2025), and the widely circulated Voice AI and Voice Agents primer names 1,500 ms voice-to-voice as the number to aim at (updated June 2026).
Cross the two-second line and the experience degrades in a way users describe as awkward rather than slow. They talk over the agent. They repeat themselves. They ask if anyone is there.
The Tricks That Fill the Gap — and Why They Only Half Work
Because the budget is so tight, the field has developed a set of latency-masking techniques. OpenAI's realtime prompting guide recommends short spoken preambles — "I'll check that order now" — explicitly to run audio and a tool call in parallel, "masking latency" (OpenAI). LiveKit ships background audio clips for exactly this purpose, including KEYBOARD_TYPING and OFFICE_AMBIENCE, played while the agent is thinking (LiveKit docs). Filler words, backchannels and simulated typing are now standard equipment.
They do measurably help — but narrowly. A 54-participant study presented at CUI '25 found that natural fillers (a thinking gesture plus a line like "Hmm, let's see…") significantly improved perceived response time at 4.0 s and 6.5 s delays, while artificial indicators such as spinners and processing tones "did not significantly affect user experience" at all. Crucially, the natural fillers' gains on every broader dimension the study measured — engagement, good impression, discomfort, competence, willingness to interact again — "were not significant" (Maslych et al., CUI '25). An earlier study of conversational fillers in virtual agents reported outright "mixed attitudes" (Pfeifer & Bickmore, IVA 2009).
That matches our own read. Masking buys a few hundred milliseconds of tolerance. It does not produce a genuine conversational experience. Architecture does.
Componentized Voice AI: Control at the Cost of Latency
Short answer: Chaining ASR, an LLM and TTS costs you serial latency and buys you an intervention point at every seam.
The componentized — or cascaded — architecture chains at least three parts: ASR for speech recognition, an NLP/LLM layer for reasoning, and TTS for speech synthesis. In our assessment it remains the way most production voice agents are built.
Its problem is arithmetic. Every component adds serial latency inside a budget that was already tight. Twilio's per-component targets are STT 350 ms, LLM time-to-first-token 375 ms, and TTS time-to-first-byte 100 ms. The Voice AI and Voice Agents primer itemizes a full round trip at 1,293 ms — transcription and endpointing 300 ms, LLM time-to-first-byte 650 ms, TTS 120 ms, and the rest lost to encoding, jitter buffering and network hops.
Meeting that budget takes real engineering. Two levers matter most:
Detecting end-of-turn earlier. Naive voice activity detection waits for silence, and the wait is expensive — LiveKit notes that "a silence timeout set to 800 ms adds nearly a full second to every single response before the pipeline even starts" (LiveKit, February 2026). Model-based turn detection replaces the timer with a prediction: LiveKit's transformer end-of-utterance model runs in about 50 ms, and LiveKit reports that pairing it with VAD cuts unintentional interruptions by 85% versus VAD alone (LiveKit, December 2024), and Pipecat's Smart Turn v3 is an 8 MB model doing 12 ms CPU inference across 23 languages (Daily, September 2025). Both encode the same insight: grammatically complete is not conversationally complete.
Running work in parallel. Stream partial transcripts into the LLM, synthesize the first sentence before the last is generated, and speculatively draft responses while the user is still speaking. Research on input-time speculative decoding reports roughly 2× average latency reduction on conversational benchmarks (PredGen, arXiv 2506.15556).
What you buy with that effort is control. A componentized pipeline exposes an output at every seam, and every seam is a place to intervene: redact PII before it reaches the model, enforce policy on the text before it is spoken, filter and compress context so the downstream model sees less and answers faster, log a transcript for every turn. That transcript is the raw material for observability, quality scoring, and the governance and audit artifacts a compliance team will ask for. Customization is straightforward because there is somewhere to put it — the same reason our text-side multi-agent platform is built as inspectable stages rather than one opaque call, and the same principle behind evaluable AI.
End-to-End Speech-to-Speech: Speed at the Cost of Control
Short answer: One model, audio in and audio out, gets you under a second — and leaves you a small prompt as your only control surface.
End-to-end models take audio in and emit audio out, skipping the text round trip entirely. The latency benefit is real and large. Kyutai's Moshi reports a theoretical latency of 160 ms and 200 ms in practice (arXiv 2410.00037). Qwen3-Omni reports a theoretical end-to-end first-packet latency of 234 ms in cold-start, audio-only settings — 547 ms with video (arXiv 2509.17765). With a good, stable network, a speech-to-speech agent can land comfortably inside one second — close enough to human timing that the interaction stops feeling like a transaction.
The cost is controllability, and it shows up in four places.
Prompting is the only real control surface, and it is small. You can load context at the start of the session, but a large system prompt inflates latency — the entire reason you chose this architecture. OpenAI's realtime context window moved from 32K to 128K tokens in May 2026 (OpenAI), which helps, but the practical per-session budget stays far below what a cascaded pipeline feeds a text LLM per turn.
Instruction-following is less reliable than in text mode. The Voice AI and Voice Agents primer states it flatly: "Speech-to-speech models do not follow instructions or call tools as reliably as text-mode LLMs." OpenAI's own prompting guide warns that gpt-realtime-2 "follows instructions more literally than earlier realtime models. Prompts that worked well on older models may need tuning." Our experiments line up with both: prompt-based control here is workable but flaky, and it fails in ways that are hard to reproduce.
The output surface is narrow. These models emit voice and some text — not the structured side-channels an operational system needs, such as reliable language detection, caller identity resolution, sentiment scores or routing decisions. OpenAI's realtime model cards list function calling as supported but structured outputs as not supported (gpt-realtime-2 model card). Adding those back means bolting on a parallel pipeline, which reintroduces the latency you were avoiding.
Cost runs higher. As of August 2026, realtime audio on gpt-realtime-2 is billed at $32 per million input tokens and $64 per million output tokens (model card); the same primer estimates a realtime-API agent at "3 to 5 times more expensive" than a text-LLM equivalent.
Fine-Tuning an End-to-End Model: The Most Feasible Path to Good CX With Control
Short answer: Move the behavior into the weights instead of fighting the model with prompts — and pay for it in compute and engineering iterations.
The third option, and in our assessment the most promising near-term one, is to fine-tune an end-to-end model for a specific use case. Instead of fighting a general model with prompts at runtime, you move the behavior into the weights: your tone, your escalation rules, your domain vocabulary, your refusal patterns. You keep sub-second latency and recover much of the control you gave up.
Two caveats matter before anyone plans around it.
The first is availability. As of August 2026 the major closed realtime APIs do not offer fine-tuning — both gpt-realtime and gpt-realtime-2 model cards list fine-tuning as not supported. The practical path today runs through open-weight models. Ultravox (MIT licensed) supports training, but only the audio adapter is trainable; the underlying LLM and audio encoder stay frozen. Its own documentation gives a useful cost anchor: v0.4 took 2–3 hours on 8×H100 GPUs for 14K training steps (fixie-ai/ultravox). Qwen3-Omni ships under Apache 2.0 and the team has fine-tuned it themselves, though the repository does not document a general fine-tuning workflow (QwenLM/Qwen3-Omni).
The second is operating cost — compute and engineering both. Fine-tuning is not a configuration change; it needs data collection, evaluation harnesses, and a team that can run the loop, and every iteration takes time. For a high-volume, narrow use case that arithmetic works. For a long tail of low-volume flows it usually does not, which is why we expect hybrid deployments — cascaded pipelines for regulated or infrequent paths, a tuned end-to-end model for the high-traffic conversation — to be the common shape rather than a clean choice among the three.
Demand for this is not hypothetical: 46% of business leaders surveyed cited fine-tuning models as key to greater voice AI adoption (Deepgram / Opus Research, State of Voice AI 2025). The same survey found 80% already use traditional voice agent systems while only 21% are very satisfied with them.
Cascaded vs Speech-to-Speech: The Three Voice AI Architectures Compared
Short answer: Cascaded for regulated and policy-heavy flows, end-to-end for latency-critical conversation, fine-tuned end-to-end for high-volume CX you can afford to train.
| Componentized (ASR→LLM→TTS) | End-to-end speech-to-speech | Fine-tuned end-to-end | |
|---|---|---|---|
| Typical latency | Tight budget; ~1.3 s round trip is good | Sub-second achievable on a good network | Sub-second, retained |
| Control surface | Many — an output at every seam | Prompt at session start, and little else | Weights plus a small prompt |
| Customization | Straightforward | Limited and unreliable | Strong, within the tuned domain |
| Non-voice outputs | Native (transcripts, language ID, caller data) | Voice plus limited text; no structured outputs | Still constrained |
| Observability & audit | Full text trail per turn | Sparse | Sparse |
| Cost profile | Per-component, generally lower | ~3–5× a text-LLM agent | Inference plus training and iteration |
| Best fit | Regulated, complex, policy-heavy flows | Latency-critical, low-stakes conversation | High-volume, narrow, CX-critical use cases |
The Voice AI Trend for 2027 and Beyond
Short answer: End-to-end wins — but only the version that can take context mid-conversation and emit more than voice.
Multiple teams are pushing this envelope continuously, and the gap between the two architectures is closing from both ends — cascaded pipelines keep shaving milliseconds through better turn detection and parallelism, while end-to-end models keep gaining instruction following and tool use.
We still believe end-to-end is the direction. But the version that wins will not be the one that is merely fastest. It will be the one that adds two capabilities the current generation lacks: the ability to inject context mid-conversation rather than only at session start, and the ability to customize the output beyond voice — structured fields, routing signals, confidence, language identity. Those two changes would close most of the controllability gap without giving back the latency advantage, and the direction of travel already points there — OpenAI's realtime context window widened from 32K to 128K in a single release cycle, and in our own testing instruction-following in realtime models has improved markedly over the past year.
Gartner expects agentic AI to autonomously resolve 80% of common customer service issues by 2029 (Gartner, March 2025). Voice is the harder half of that forecast, and the architecture question is why.
Summary
Voice AI architecture is a tradeoff between controllability and latency, forced by a response budget of roughly two seconds and ideally under one. Componentized pipelines give you control at every seam and make you work for the milliseconds. End-to-end models give you the milliseconds and take away almost everywhere you might have inserted control. Fine-tuning an end-to-end model is currently the most feasible way to get both, at the price of compute and engineering iteration. Latency masking — fillers, typing sounds, spoken preambles — buys a few hundred milliseconds of tolerance and nothing more.
We have not shipped voice at Aissist.io. We are watching this area closely, running our own experiments, and we will update this study as the picture changes.
Comparing notes on voice, or already running agentic AI on text? See how AgentMesh™ resolves service and sales end-to-end on the stack you already run. Book a consultation →
Frequently Asked Questions
What is a good latency for a voice AI agent?
Stay under two seconds end-to-end, and target under one second where you can. Twilio publishes a mouth-to-ear target of 1,115 ms with a 1,400 ms upper limit, and a widely used industry primer names 1,500 ms voice-to-voice as the goal. For reference, humans in natural conversation take turns with a modal gap of about 200 ms — no production voice agent is close to that yet.
What is the difference between cascaded and speech-to-speech voice AI architecture?
A cascaded (componentized) architecture chains separate ASR, LLM and TTS components, passing text between them. A speech-to-speech architecture uses one model that takes audio in and emits audio out. Cascaded is slower but exposes text at every stage, which is where control, filtering and logging live. Speech-to-speech is faster but offers almost no intervention points.
Which voice AI architecture do most production agents actually use?
In our assessment, componentized ASR→LLM→TTS pipelines still account for the majority of production deployments, because most enterprise use cases need the control, transcripts and audit trail that the text layer provides. End-to-end models are gaining ground in latency-critical, lower-stakes conversation, and hybrid deployments that route different flows to different architectures are becoming common.
Can you fine-tune a speech-to-speech model?
Not on the major closed realtime APIs as of August 2026 — OpenAI's gpt-realtime and gpt-realtime-2 model cards both list fine-tuning as unsupported. Fine-tuning is available on open-weight models such as Ultravox (MIT) and Qwen3-Omni (Apache 2.0), though Ultravox trains only the audio adapter while the LLM and encoder stay frozen. Expect meaningful engineering effort per iteration.
What is turn detection in voice AI, and how is it different from VAD?
Voice activity detection simply measures whether someone is speaking and waits for silence — a timer, and an expensive one, since an 800 ms silence threshold adds nearly a second before the pipeline even starts. Turn detection uses a model to predict whether the speaker has actually finished their thought. LiveKit's transformer model runs in about 50 ms, and pairing it with VAD reduces unintentional interruptions by 85% versus VAD alone.
Do filler words and typing sounds actually make voice AI feel faster?
Somewhat, and only for perceived response time. A 54-participant study found natural fillers significantly improved perceived responsiveness at 4.0 s and 6.5 s delays, while artificial indicators like spinners and processing tones showed no significant improvement. The same study found no significant gain on any other experience dimension. Masking buys tolerance; it does not create a genuine conversational feel.
Why can't you just put a long system prompt in an end-to-end voice model?
Because prompt size directly costs latency, which is the reason you chose an end-to-end model. Context windows have widened — OpenAI's realtime models moved from 32K to 128K tokens in May 2026 — but the practical per-session budget stays small, and instruction-following in speech-to-speech mode is less reliable than in text mode, so a longer prompt does not proportionally buy more control.
How much more expensive is an end-to-end voice agent?
As of August 2026, realtime audio pricing on gpt-realtime-2 is $32 per million input tokens and $64 per million output tokens. One widely cited industry estimate puts a realtime-API agent at roughly three to five times the cost of a comparable text-LLM agent. Fine-tuned open-weight deployments trade that per-token premium for training compute and ongoing engineering cost.
Can end-to-end voice models do language detection or caller identification?
Not natively as a structured output. These models emit voice and some text; OpenAI's realtime model cards support function calling but not structured outputs. Getting reliable language identification, caller ID resolution, sentiment or routing signals usually means running a parallel pipeline alongside the model, which adds back some of the complexity and latency the architecture was meant to avoid.
