Audio and voice models: the 2026 landscape, from signal processing to full-duplex

Why this subject matters now

For most of the field's history a voice interface was a pipeline of independent services wired in series. A voice-activity detector decided when the user had stopped talking, a speech-to-text model transcribed the buffered audio, a language model read the transcript and produced a reply, and a text-to-speech model spoke the reply back. This cascade, sometimes called the STT to LLM to TTS pattern, is turn-based by construction. Each stage waits for the previous one to finish, the model cannot hear the user while it is speaking, and the whole thing has a felt latency that is the sum of the parts plus the time spent deciding the user was actually done. It works, most production voice agents in 2026 are still built this way, and a practitioner has to know how to squeeze it. But it is a fundamentally different object from what arrived most recently.

The change that matters is the move from that turn-based cascade to native full-duplex interaction. A full-duplex model listens and speaks at the same time. It can be interrupted mid-sentence and stop cleanly (barge-in), it can emit the small acknowledgments a human listener makes while the other person is still talking (backchannels, the spoken equivalent of nodding), and it decides when to take the turn from the audio itself rather than from a separate silence timer. Kyutai's Moshi, released in open form in 2024, was the first widely available system built this way. OpenAI's Realtime API reached general availability in August 2025 with gpt-realtime, a single speech-to-speech model rather than a cascade, and through 2026 that line grew to include higher-reasoning and live-translation variants. Around May 2026 Thinking Machines Lab announced its Interaction Models, a family designed from the ground up to process audio, video, and text in small continuous chunks and to listen while speaking. Full-duplex is the current frontier, and the reason it is hard is not raw speed. It is that the model has to maintain a running policy over the shared channel, deciding every fraction of a second whether to speak, keep speaking, stop, or listen.

What a practitioner is expected to know today, then, is broader than a recognizer. It is the whole stack: the features and codecs that turn a waveform into something a transformer can model, the three generations of ASR and the three of TTS and how each fixed the one before it, the difference between a cascade and a native speech-to-speech model and a full-duplex model, how to fine-tune and evaluate any of them without fooling yourself, and, because so much of this is now a commercial decision, how the labs and products actually differ once the marketing is stripped away. The modeling internals of the recognizers and codecs named here live on the speech and spoken language page and the transformer backbone they hand tokens to lives on the language models from scratch page. This page is the landscape and the comparison.

TURN-BASED CASCADE (still the common production pattern)

  user speaks --> VAD/endpoint --> STT --> LLM --> TTS --> audio out
                  waits for silence     transcript  reply   speech
                  |______________ serial, latencies add ______________|

NATIVE SPEECH-TO-SPEECH (gpt-realtime, one model)

  user audio  --> [ single audio-in / audio-out model ] --> audio out
                  no intermediate text round-trip required

FULL-DUPLEX (Moshi, Thinking Machines Interaction Models)

  user audio  ===============================>  continuous input stream
  model audio  <==============================  continuous output stream
                both directions live at once: barge-in, backchannels,
                model-decided turn-taking from the audio itself

Signal processing then versus now

Every audio model, however modern, still begins by turning pressure into numbers and numbers into a representation. What changed is how much of that representation is hand-designed and how much is learned. The classic pipeline and the modern stack solve the same problems in the same order, and seeing them side by side is the fastest way to understand why the field moved.

The classic pipeline

The traditional front end frames the waveform into overlapping windows of about 25 ms every 10 ms, takes a short-time Fourier transform of each frame, warps the power spectrum onto the perceptual mel scale with a bank of triangular filters, takes the logarithm, and, for classical recognizers, decorrelates the result with a discrete cosine transform to produce mel-frequency cepstral coefficients. The mel warp is the one formula worth carrying between pages, because it encodes the roughly logarithmic pitch resolution of human hearing,

and it is derived in full, with the triangular filterbank and the reason log-mel replaced MFCCs for neural models, on the speech page. Those cepstral features fed a hidden-Markov-model acoustic recognizer whose states emitted Gaussian mixtures and whose transitions were decoded by Viterbi against a hand-built pronunciation lexicon and an n-gram language model. Synthesis ran the other direction with equally hand-built machinery: concatenative TTS stitched together recorded units of speech chosen to minimize a join cost, and parametric TTS drove a source-filter vocoder, an impulse or noise source shaped by a time-varying filter modeling the vocal tract, from statistically predicted parameters. The signal-processing derivations underneath all of this, sampling and aliasing, the DFT and FFT, windows and leakage, linear prediction and the source-filter model, are on the audio signal processing page. Here the point is only that the entire pipeline was a chain of separately designed and separately trained parts.

The learned stack

The modern stack keeps the same skeleton and makes the interior learnable. The mel spectrogram survives as an input, but self-supervised encoders such as wav2vec 2.0 and HuBERT often skip even the filterbank and learn a convolutional front end from the raw waveform, pretrained on hundreds of thousands of hours of unlabeled audio so that a small labeled set suffices to fine-tune a recognizer. The most consequential new component has no classical analogue that quite fits: the neural audio codec. A codec such as SoundStream, EnCodec, Descript's DAC, or Kyutai's Mimi is an autoencoder that compresses a waveform into a short sequence of discrete tokens and reconstructs it, trained adversarially against multi-resolution spectral losses. Once audio is a sequence of tokens from a fixed vocabulary, a transformer can model it with exactly the machinery it uses for text, and that single fact is what unlocked audio language models, in-context voice cloning, and speech-to-speech systems. The quantizer that makes those tokens is residual vector quantization, and its bitrate is a formula worth being able to write down. A codec that emits frames at rate \( f_r \) hertz using \( Q \) stacked codebooks of \( V \) entries each spends \( \log_2 V \) bits per codebook index, so

The residual structure is what makes the codebooks cheap. The first codebook quantizes the latent vector, the second quantizes the error the first one left behind, the third quantizes the error after that, and so on, so \( Q \) codebooks of \( V \) entries represent \( V^Q \) distinct combinations while storing only \( Q \cdot V \) code vectors. With \( Q = 8 \) and \( V = 1024 \) that is \( 1024^8 = 2^{80} \) effective points from 8192 stored vectors, and because the codebooks are ordered coarse to fine, keeping a prefix of them gives a lower-bitrate reconstruction for free. Residual vector quantization is derived in more detail on the speech page; the survey point is that discrete audio tokens are the bridge that let the language-model program absorb speech.

The model families, on a timeline

The field has four families that a practitioner keeps separate in their head: recognizers that turn audio into text, codecs that turn audio into tokens, synthesizers that turn text into audio, and language models that operate on audio tokens directly, including the speech-to-speech and full-duplex systems. They evolved in parallel and now feed each other, the codec makes the tokens the audio language model consumes, and the same self-supervised features improve both recognition and synthesis.

                 2006-15        2016-19          2020-22            2023-24           2025-26
ASR       HMM-GMM --> CTC/DeepSpeech --> LAS/RNN-T --> wav2vec2/HuBERT --> Whisper --> streaming Nova/Flux
CODEC                                        SoundStream --> EnCodec --> DAC / Mimi (12.5 Hz)
TTS  concat/param --> Tacotron/Tacotron2 --> FastSpeech/VITS --> VALL-E/StyleTTS2 --> Sonic/Octave/Aura-2
S2S / DUPLEX                                       AudioLM/SpeechGPT --> Moshi/GPT-4o voice --> gpt-realtime
                                                                          Thinking Machines Interaction Models

Automatic speech recognition

Recognition passed through three regimes. The first was the HMM-GMM pipeline described above, a lexicon and a forced alignment and an n-gram rescorer, powerful but a specialist craft. The second was end-to-end neural training: Baidu's DeepSpeech applied connectionist temporal classification to drop the alignment, Listen-Attend-and-Spell recast recognition as attention-based sequence-to-sequence translation, and the RNN transducer added an autoregressive prediction network that let a streaming recognizer model output dependencies. The Conformer married convolution and self-attention in the encoder and became the default acoustic backbone for production streaming systems. All three losses, CTC, attention, and RNN-T, along with their forward-backward recurrences and gradients, are derived on the speech page and are not repeated here.

The third regime is the one to internalize now. wav2vec 2.0 and HuBERT showed that a contrastive or masked-prediction objective on unlabeled audio produces representations good enough that a tiny labeled set fine-tunes a strong recognizer, which collapsed the cost of building an ASR system for a new language. Whisper then took the opposite bet, weak supervision at scale, training a plain encoder-decoder transformer on 680,000 hours of noisy multilingual web audio and getting robustness across accents and acoustic conditions that clean academic corpora never produced. Whisper is open weights and has become the default batch transcriber, but it is not streaming, its encoder consumes a 30-second window and its decoder can hallucinate text in silence. Production streaming recognition is a separate design point, and that is where the commercial ASR vendors live: Deepgram's Nova line for low-latency streaming transcription, its Flux model marketed for conversational turn detection, and OpenAI's gpt-realtime-whisper for streaming speech to text inside the Realtime API. The distinction between a robust offline recognizer and a low-latency streaming one is the single most common source of confusion when people pick an ASR model.

Neural audio codecs

The codec is the quiet enabler of everything downstream. SoundStream introduced the end-to-end neural codec with residual vector quantization and adversarial training, EnCodec refined it and became the widely used open implementation, and Descript's DAC pushed reconstruction quality higher with an improved residual-quantized-GAN recipe. Kyutai's Mimi is the codec built specifically for streaming full-duplex dialogue, and its defining choice is a very low frame rate, on the order of 12.5 Hz, with a first codebook distilled to carry semantic content so that a language model reasoning over its tokens sees a short sequence. Codecs are small, on the order of tens of millions of parameters, which is worth stating because people assume the impressive audio quality implies a large model. It does not, the codec is a compact autoencoder and the intelligence lives in whatever consumes its tokens. The reason discrete tokens mattered so much is the one from the previous section, once audio is a token stream a text transformer models it unchanged, and that is what turned speech from a separate discipline into another modality inside the language-model program.

Text-to-speech

Synthesis has its own three-generation arc, and each generation removed a piece of hand-built machinery. The classical era was concatenative and parametric, high effort and limited flexibility. The first neural generation was Tacotron and Tacotron 2, sequence-to-sequence models with attention that predicted a mel spectrogram from characters and inverted it with a neural vocoder, WaveNet at first and later the far faster HiFi-GAN. Tacotron inherited the attention-instability failure of sequence-to-sequence models, occasionally skipping or repeating words, which FastSpeech fixed by predicting an explicit per-phoneme duration and expanding the sequence with a length regulator, making synthesis non-autoregressive and parallel and robust. VITS then folded the acoustic model and vocoder into one end-to-end model trained with a variational objective, normalizing flows, and an adversarial loss, producing waveforms directly from text without a separate spectrogram stage.

The current generation treats speech synthesis as conditional generation over audio tokens or as a continuous generative process. VALL-E reframed TTS as a neural-codec language-modeling problem: it predicts EnCodec-style tokens autoregressively, and because a short reference clip can be provided as the prompt, it performs in-context voice cloning from a few seconds of a target speaker, a capability the earlier generations did not have. A parallel line uses diffusion and flow matching to generate the acoustic representation, and StyleTTS 2 combined a style diffusion model with adversarial training against a large speech language model to reach human-level naturalness on read speech. The connection to image and audio diffusion is on the diffusion and large vision models page. Commercially this is where the most visible products sit. ElevenLabs is generally regarded as the most expressive and has the broadest voice library and cloning tools; Cartesia's Sonic family targets extremely low time-to-first-audio and is notable for being built on state-space models rather than transformers, from the same researchers behind the Mamba architecture, which is covered on the sequence models and state spaces page; Deepgram's Aura and Aura-2 target low-latency conversational TTS for agents; and Hume's Octave emphasizes emotional prosody and expressive control. The exact model sizes and architectures of all of these commercial systems are undisclosed.

Audio and speech language models, speech-to-speech, and full-duplex

The fourth family is the one moving fastest, and it is worth being precise about the three architectures people conflate. A cascade is the STT to LLM to TTS pipeline: three models, text in the middle, turn-based. A native speech-to-speech model takes audio in and produces audio out with a single model and no intermediate text round-trip, which removes the serial handoffs and lets the model preserve tone, emotion, and timing that a transcript throws away. A full-duplex model goes further and runs both directions continuously, so it can be interrupted, can backchannel, and decides turn-taking from the audio stream itself. These are genuinely different systems, and a faster cascade is not a full-duplex model no matter how low its latency gets.

The research lineage runs from AudioLM, which showed that a language model over codec tokens plus semantic tokens could continue speech and even piano audio with long-term coherence, through SpeechGPT and Qwen2-Audio, which coupled audio understanding to a text LLM, to Moshi, the first open full-duplex speech-text model. Moshi's design is instructive: it models the user stream and its own stream jointly at Mimi's low frame rate, keeps an internal text monologue aligned to its speech, and therefore can speak and listen at once with sub-second theoretical latency, all with an open roughly 7-billion-parameter backbone built on the Helium language model. OpenAI's GPT-4o voice mode and then the Realtime API brought native speech-to-speech to a mass audience, and gpt-realtime, generally available since late August 2025, added tool calls, the Model Context Protocol, image input, and SIP phone calling so that a single model can run a phone agent. Through 2026 OpenAI reported extending the family with a higher-reasoning GPT-Realtime-2, a GPT-Realtime-Translate for live translation across many languages, and a GPT-Realtime-Whisper for streaming transcription. The sizes of all of these are undisclosed.

The 2026 arrival worth watching is Thinking Machines Lab, founded in February 2025 by Mira Murati with a group of former OpenAI researchers, which announced its Interaction Models around May 2026. The first model is reported as TML-Interaction-Small, described as a mixture-of-experts model with roughly 276 billion total and about 12 billion active parameters that processes audio, video, and text together in continuous chunks of about 200 milliseconds and is architected to listen while speaking, that is, native full-duplex. Every quantitative claim about it here is company-reported and, as of this writing, not independently verified: the lab reported roughly 0.40 seconds of turn-taking latency against about 1.18 seconds for a gpt-realtime-class baseline, and a substantially higher score on an internal full-duplex benchmark it calls FD-bench. As of mid-2026 it is a research preview with limited access. Treat those figures as a vendor press claim, interesting as a direction, not as a measured result, exactly the posture the product-comparison section argues for in general.

Music and general audio, briefly

Speech is not the whole of audio, and the generative music and sound-effect models share the same building blocks, a neural codec plus a language model or a diffusion process. MusicGen, part of Meta's AudioCraft, is a single-stage transformer over EnCodec tokens that generates music from text and melody conditioning. Suno is the most prominent commercial text-to-song product, generating full songs with vocals, and Stable Audio from Stability applies latent diffusion to audio for music and sound design. These are surveyed in more depth, with the diffusion mechanics, on the diffusion and computational music analysis pages; the point here is only that the taxonomy generalizes, discrete tokens or a diffusion latent, plus a generative model, spans speech, music, and general audio alike.

Model sizes: what is known and what is not

Parameter counts are where surveys most often invent numbers, so this table is deliberately conservative. The open models have counts from their weights or model cards; the paper models have counts their authors reported; the commercial ASR and TTS and speech-to-speech models almost all have undisclosed sizes, and the honest entry is to say so rather than guess. Reading a vendor's latency or quality and inferring a size from it is not possible, a small well-trained model can beat a large one, and the codec that produces excellent audio is tiny.

The pattern is clear. Every model whose size is known is either open or described in a paper, and every closed commercial voice model has an undisclosed size. When a system-design conversation needs a number for a closed model, the correct move is to measure the behavior that actually matters, latency, throughput, quality on your audio, and treat the parameter count as unavailable.

How to fine-tune these models

Fine-tuning an audio model is mostly a data-and-evaluation problem wearing a training-loop costume. The training mechanics are standard, the traps are in the data preparation and the metrics, so the code here is deliberately paired with the caveats.

Fine-tuning Whisper

Whisper fine-tuning is the most common request, usually to adapt it to an accent, a domain vocabulary, or a low-resource language. The recipe is to load the model and its processor, build a dataset that pairs audio resampled to 16 kHz with normalized transcripts, compute log-mel input features and tokenized labels, and train with a sequence-to-sequence trainer. Full fine-tuning of large-v3 is heavy, so parameter-efficient fine-tuning with LoRA is the usual choice: it freezes the base weights and trains small low-rank adapters, which cuts memory by an order of magnitude and, importantly, reduces the catastrophic-forgetting risk that full fine-tuning carries. That risk is real, a Whisper model fine-tuned hard on one narrow domain can lose the broad robustness that made it worth starting from, so the held-out evaluation must include out-of-domain audio, not just more of the training distribution. Fine-tuning helps most when the target has systematic vocabulary or acoustic characteristics the base model mishandles; it helps least, and can hurt, when the base model is already strong and the fine-tuning set is small and narrow.

import torch
from datasets import load_dataset, Audio
from transformers import (WhisperProcessor, WhisperForConditionalGeneration,
                          Seq2SeqTrainer, Seq2SeqTrainingArguments)

# 1. model + processor. "small" is a sane starting point for adaptation.
name = "openai/whisper-small"
processor = WhisperProcessor.from_pretrained(name, language="en", task="transcribe")
model = WhisperForConditionalGeneration.from_pretrained(name)
model.generation_config.language = "en"
model.generation_config.task = "transcribe"

# 2. data: resample audio to 16 kHz, normalise the reference text yourself.
ds = load_dataset("your/dataset").cast_column("audio", Audio(sampling_rate=16000))

def prepare(batch):
    a = batch["audio"]
    # log-mel features, shape (80, 3000) for a 30 s window
    batch["input_features"] = processor.feature_extractor(
        a["array"], sampling_rate=16000).input_features[0]
    batch["labels"] = processor.tokenizer(batch["text"]).input_ids
    return batch

ds = ds.map(prepare, remove_columns=ds["train"].column_names)

# 3. train. small LR, short schedule: you are nudging, not retraining.
args = Seq2SeqTrainingArguments(
    output_dir="whisper-ft", per_device_train_batch_size=16,
    learning_rate=1e-5, warmup_steps=200, max_steps=2000,
    fp16=torch.cuda.is_available(), predict_with_generate=True,
    eval_strategy="steps", eval_steps=500)

trainer = Seq2SeqTrainer(model=model, args=args,
    train_dataset=ds["train"], eval_dataset=ds["test"],
    tokenizer=processor.feature_extractor)
trainer.train()
# ALWAYS evaluate on held-out, out-of-domain audio too, not just in-domain.

import torch
from transformers import WhisperForConditionalGeneration
from peft import LoraConfig, get_peft_model

# Parameter-efficient fine-tuning: freeze the base, train low-rank adapters
# on the attention projections. ~1% of the parameters are trainable, memory
# drops by roughly an order of magnitude, and forgetting risk is lower.
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3")

lora = LoraConfig(
    r=32, lora_alpha=64, lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"],  # attention query/value projections
    bias="none")

model = get_peft_model(model, lora)
model.print_trainable_parameters()
# trainable params: ~15M || all params: ~1.55B || trainable%: ~1.0

# The rest of the loop is identical to the Trainer version above; you save
# only the adapter (tens of MB) and load it on top of the frozen base at
# inference. Merge with model.merge_and_unload() if you want a single model.

Voice cloning and TTS adaptation

Cloning a voice comes in two grades that are worth distinguishing because they trade off effort against quality and consent. Instant or zero-shot cloning, the mode VALL-E introduced and ElevenLabs offers as instant voice cloning, conditions a pretrained model on a few seconds of reference audio with no training, fast but limited in fidelity and expressiveness. Professional or fine-tuned cloning trains or adapts the model on many minutes to hours of a consenting speaker and reaches much higher fidelity, at the cost of data collection and compute. Open options let you do this yourself: Coqui's XTTS supports multilingual zero-shot cloning from a short clip, and StyleTTS 2 can be fine-tuned on a target speaker for high-quality results. The engineering caveat is consent and misuse, cloning a voice from public audio without permission is both an ethical and increasingly a legal problem, and the reputable commercial services gate professional cloning behind voice-verification steps for exactly this reason.

Adapting audio language models, and the evaluation traps

Audio language models and speech-to-speech models adapt with the same LoRA machinery as text LLMs, applied to the transformer backbone while the codec stays frozen, since the codec is a fixed tokenizer. The harder part is always evaluation, and audio has more ways to fool you than most modalities. Recognition is scored by word error rate and character error rate, edit distances against a reference, and the trap there is that an error that is catastrophic for the task, a wrong digit in a dollar amount, counts exactly the same as a harmless one, a dropped plural. Synthesis is scored by mean opinion score, a 1-to-5 human naturalness rating, and its unreliability is notorious: MOS is not comparable across studies, listener pools, or even sessions, it saturates near the top so two good systems are hard to separate, and it is easily gamed by audio normalization. Voice cloning adds speaker-similarity metrics, cosine similarity between speaker embeddings, which reward matching the timbre but say nothing about whether the prosody is right. The single most important discipline across all of these is a genuinely held-out evaluation, held-out speakers for cloning and held-out domains for recognition, because a model evaluated on speakers or domains it saw in training will report numbers it cannot reproduce in deployment.

A codec round trip, to make the tokens concrete

The fastest way to internalize what a neural codec does is to encode and decode a clip and look at the token tensor. The PyTorch tab uses EnCodec through its library; the JAX tab implements the residual vector-quantization step itself, which is the one piece of genuinely new algorithm in the codec, so its mechanics are visible rather than hidden behind an API.

import torch, torchaudio
from encodec import EncodecModel

model = EncodecModel.encodec_model_24khz()
model.set_target_bandwidth(6.0)          # 6 kbps -> 8 codebooks

wav, sr = torchaudio.load("clip.wav")    # (channels, samples)
if sr != 24000:
    wav = torchaudio.functional.resample(wav, sr, 24000)
wav = wav.mean(0, keepdim=True)[None]    # (1, 1, samples), mono

with torch.no_grad():
    encoded = model.encode(wav)          # list of (codes, scale)
    codes = encoded[0][0]                # (1, Q=8, T_frames), integer tokens
    recon = model.decode(encoded)        # (1, 1, samples), reconstruction

frames = codes.shape[-1]
seconds = wav.shape[-1] / 24000
print("frame rate:", round(frames / seconds))          # ~75 Hz
print("codebooks:", codes.shape[1])                     # 8
print("token rate:", round(codes.shape[1] * frames / seconds))  # ~600 tokens/s
# codes is exactly what an audio language model consumes as its vocabulary.

import jax, jax.numpy as jnp

def rvq_encode(x, codebooks):
    # x:         (T, D) latent frames from the encoder
    # codebooks: (Q, V, D) stacked codebooks, coarse to fine
    # returns:   indices (T, Q) and the quantized reconstruction (T, D)
    residual = x
    indices = []
    quantized = jnp.zeros_like(x)
    for q in range(codebooks.shape[0]):
        cb = codebooks[q]                              # (V, D)
        # nearest entry to the current residual, per frame
        d2 = jnp.sum((residual[:, None, :] - cb[None]) ** 2, axis=-1)  # (T, V)
        idx = jnp.argmin(d2, axis=-1)                  # (T,)
        picked = cb[idx]                               # (T, D)
        quantized = quantized + picked
        residual = residual - picked                   # quantize the leftover next
        indices.append(idx)
    return jnp.stack(indices, axis=-1), quantized

# Q codebooks of V entries encode V**Q combinations from Q*V stored vectors:
# with Q=8, V=1024 that is 1024**8 == 2**80 points from 8192 vectors.

Comparing the labs and products

This is the section a practitioner actually reaches for, and the first thing to say is a warning. Vendor benchmarks are marketing. Every latency number, quality claim, and head-to-head chart a vendor publishes was produced under conditions the vendor chose, on inputs the vendor chose, with a definition of the metric the vendor chose. There is a well-argued case in the field that these numbers are close to useless for cross-vendor decisions, because time-to-first-audio measured on a warm connection in one region tells you almost nothing about the round-trip latency your users will feel, and quality measured on the vendor's demo script tells you nothing about your accents and your domain. The only number that means anything is the one you measure on your own workload, with your audio, from your users' network, at your concurrency. Read the table below as a map of what each product is for, not as a leaderboard.

The orchestration layer is a separate category and a frequent source of confusion. Vapi and Retell are not speech models. They are orchestration platforms that wire a speech-to-text model, a language model, and a text-to-speech model together into a deployable voice agent, and they handle the parts that are tedious to build yourself: telephony and SIP trunking, turn detection and endpointing, barge-in handling, function calling, and call analytics. When someone says they "use Vapi's voice model" they are confusing the conductor for the orchestra; the actual speech quality and latency come from whichever ASR and TTS models Vapi is configured to call. LiveKit Agents and Pipecat occupy the same orchestration category as open-source frameworks, LiveKit built on its WebRTC transport and Pipecat as a Python pipeline framework, and both let you assemble a cascade or plug in a native speech-to-speech model. Choosing an orchestrator is a different decision from choosing a model, and conflating the two leads to buying the wrong thing.

A recommended production stack for a sub-700ms voice agent

The pattern that most reliably hits interactive latency in 2026 is still a well-tuned cascade, with the option to move to a native speech-to-speech model where its qualities are worth the tradeoffs. A workable default is a streaming ASR model with fast, confident endpointing (Deepgram's Nova or Flux, or gpt-realtime-whisper), a small-to-mid language model with low time-to-first-token and streaming output, and a low-latency streaming TTS with a very short time-to-first-audio (Cartesia Sonic, Deepgram Aura-2, or ElevenLabs where expressiveness matters more than the last few milliseconds), assembled with an orchestrator such as Pipecat or LiveKit Agents that overlaps the stages, begins TTS on the first sentence while the LLM is still generating, and handles barge-in by cancelling in-flight speech the moment the user starts talking. The single highest-leverage tuning knob is endpointing, deciding the user is done, because it is often the largest and most controllable term in the budget and because being too eager cuts users off while being too patient adds dead air.

The tradeoff against a native speech-to-speech model such as gpt-realtime is real in both directions. The cascade gives you full control, you can swap any component, log the intermediate transcript, apply guardrails to the text, and mix vendors, and it is often cheaper. The native model gives you things the cascade structurally cannot, it preserves tone and emotion that the transcript discards, it responds faster because there is no serial handoff, and with a full-duplex model it can handle interruptions and backchannels naturally. The decision is not which is better in the abstract but which failure modes you can tolerate: a cascade that occasionally mis-transcribes and replies to the wrong thing, or a native model that is more opaque and harder to constrain. Whichever you choose, the closing instruction from the comparison table stands, measure it on your own workload before you commit.

Open source to read

These are real repositories, chosen for what they teach, with the first file worth opening. Reading the codec and full-duplex repositories in particular does more to build intuition than any survey paragraph.

• openai/whisper : the reference ASR model and decoding heuristics. Open whisper/decoding.py to see temperature fallback and the compression-ratio and no-speech thresholds that suppress hallucination.
• SYSTRAN/faster-whisper : a CTranslate2 reimplementation that is several times faster with lower memory; what you actually deploy for batch transcription. Start at faster_whisper/transcribe.py.
• m-bain/whisperX : Whisper plus forced alignment for accurate word-level timestamps and speaker diarization. Open whisperx/alignment.py to see how the phoneme alignment is bolted on.
• coqui-ai/TTS : the broad open TTS toolkit, home of XTTS for multilingual zero-shot cloning. Open TTS/tts/models/xtts.py for the cloning path.
• yl4579/StyleTTS2 : human-level TTS via style diffusion and adversarial training against a speech LM. Open models.py to see the style predictor and the diffusion sampler.
• facebookresearch/encodec : the widely used neural codec; the clearest place to read residual vector quantization. Open encodec/quantization/core_vq.py.
• descriptinc/descript-audio-codec : DAC, higher-fidelity codec with an improved RVQ-GAN recipe. Open dac/model/dac.py to compare its quantizer with EnCodec's.
• kyutai-labs/moshi : the open full-duplex speech-text model and the Mimi codec; the single best repo for understanding listen-while-speaking. Open moshi/models/lm.py and the Mimi model under moshi/models/.
• facebookresearch/audiocraft : MusicGen and AudioGen for generative music and sound. Open audiocraft/models/musicgen.py.
• NVIDIA/NeMo : production-grade toolkit for Conformer and transducer ASR and TTS at scale. Open the ASR model configs under examples/asr/ to see real training recipes.
• espnet/espnet : the research reference for end-to-end speech, hybrid CTC/attention and everything around it. Open a recipe under egs2/ for a full pipeline.
• speechbrain/speechbrain : a readable PyTorch toolkit spanning ASR, speaker recognition, and enhancement. Open a recipe under recipes/.
• snakers4/silero-vad : the small, fast voice-activity detector nearly every voice agent uses for endpointing. Open the usage example in the README; the model is tiny and worth understanding.
• pipecat-ai/pipecat : the open orchestration framework for real-time voice agents; the best code for seeing how a cascade is actually wired and overlapped. Open the examples directory and one of the pipeline scripts.

Common misconceptions

"A vendor's lower latency number means a faster agent." The number a vendor advertises is usually time-to-first-audio measured on a warm local connection. The latency a user feels is the full round trip, endpointing plus STT plus LLM plus TTS plus real network, and Problem 5 shows the network term alone can swamp a model's advantage. Measure end to end on your users' devices.

"Vapi (or Retell) is a voice model." Both are orchestration platforms that wire together someone else's STT, LLM, and TTS into a deployable agent and handle telephony, turn-taking, and analytics. The speech quality and latency come from the underlying models they call, not from the orchestrator. The same distinction applies to LiveKit Agents and Pipecat.

"Mean opinion score is a reliable quality metric." MOS is a 1-to-5 human rating that is not comparable across studies, listener pools, or sessions, saturates near the top so two good systems are hard to separate, and is sensitive to loudness normalization and playback conditions. It is a coarse sanity check, not a leaderboard, and cross-paper MOS comparisons are close to meaningless.

"Full-duplex is just faster turn-taking." Lowering a cascade's latency does not make it full-duplex. Full-duplex means the model runs both directions of the channel continuously and can be interrupted, can emit backchannels while listening, and decides turn-taking from the audio itself. It is a different architecture with a running policy over the shared channel, not a speed tweak.

"A bigger TTS model is always better." Size is not disclosed for most commercial TTS, and where it is known a compact model often wins on the axes that matter for agents, latency and consistency. A neural codec that produces excellent audio is only tens of millions of parameters. Expressiveness comes from training data and objective, not from raw scale.

"Whisper is a streaming recognizer." Whisper is an offline encoder-decoder that consumes a 30-second window and is prone to hallucinating in silence. Real-time transcription needs a streaming model such as Deepgram Nova or gpt-realtime-whisper, or a chunked wrapper with careful handling; using vanilla Whisper in a live pipeline invites latency and repetition problems.

"Discrete audio tokens are lossy in a way that ruins quality." A residual vector quantizer with enough codebooks reconstructs audio at high fidelity, and the tokenization is exactly what let text transformers model speech at all. The tradeoff is bitrate against sequence length, not a hard quality ceiling, and streaming systems deliberately choose a low frame rate to keep the sequence short.

Self-check

References

1. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C. and Sutskever, I. (2022). Robust speech recognition via large-scale weak supervision (Whisper). arXiv:2212.04356
2. Baevski, A., Zhou, H., Mohamed, A. and Auli, M. (2020). wav2vec 2.0: a framework for self-supervised learning of speech representations. NeurIPS. arXiv:2006.11477
3. Hsu, W.-N., Bolte, B., Tsai, Y.-H., Lakhotia, K., Salakhutdinov, R. and Mohamed, A. (2021). HuBERT: self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM TASLP. arXiv:2106.07447
4. Gulati, A. et al. (2020). Conformer: convolution-augmented transformer for speech recognition. Interspeech. arXiv:2005.08100
5. Shen, J. et al. (2018). Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions (Tacotron 2). ICASSP. arXiv:1712.05884
6. Ren, Y. et al. (2019). FastSpeech: fast, robust and controllable text to speech. NeurIPS. arXiv:1905.09263
7. Kim, J., Kong, J. and Son, J. (2021). Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech (VITS). ICML. arXiv:2106.06103
8. Wang, C. et al. (2023). Neural codec language models are zero-shot text-to-speech synthesizers (VALL-E). arXiv:2301.02111
9. Li, Y. A., Han, C., Raghavan, V., Mischler, G. and Mesgarani, N. (2023). StyleTTS 2: towards human-level text-to-speech through style diffusion and adversarial training with large speech language models. NeurIPS. arXiv:2306.07691
10. Borsos, Z. et al. (2022). AudioLM: a language modeling approach to audio generation. IEEE/ACM TASLP. arXiv:2209.03143
11. Zeghidour, N., Luebs, A., Omran, A., Skoglund, J. and Tagliasacchi, M. (2021). SoundStream: an end-to-end neural audio codec. IEEE/ACM TASLP. arXiv:2107.03312
12. Défossez, A., Copet, J., Synnaeve, G. and Adi, Y. (2022). High fidelity neural audio compression (EnCodec). TMLR. arXiv:2210.13438
13. Kumar, R., Seetharaman, P., Luebs, A., Kumar, I. and Kumar, K. (2023). High-fidelity audio compression with improved RVQGAN (Descript Audio Codec). arXiv:2306.06546
14. Défossez, A., Mazaré, L., Orsini, M., Royer, A. et al. (2024). Moshi: a speech-text foundation model for real-time dialogue (Kyutai). arXiv:2410.00037
15. Copet, J., Kreuk, F., Gat, I., Remez, T. et al. (2023). Simple and controllable music generation (MusicGen). NeurIPS. arXiv:2306.05284
16. Gu, A. and Dao, T. (2023). Mamba: linear-time sequence modeling with selective state spaces. The state-space line of work behind Cartesia's Sonic. arXiv:2312.00752
17. OpenAI (2025). Introducing next-generation audio models in the API (gpt-4o-transcribe and gpt-4o-mini-tts). openai.com/index/introducing-our-next-generation-audio-models
18. OpenAI (2025). Introducing gpt-realtime and Realtime API general availability. openai.com/index/introducing-gpt-realtime
19. ElevenLabs. Product and API documentation for expressive TTS and voice cloning. elevenlabs.io/docs
20. Cartesia. Sonic real-time TTS documentation. docs.cartesia.ai
21. Deepgram. Nova, Flux, and Aura documentation for streaming ASR and TTS. developers.deepgram.com
22. The Decoder (2026). Reporting on Thinking Machines Lab's Interaction Models launch; used as secondary source since no formal paper exists. the-decoder.com
23. DeepLearning.AI, The Batch (2026). Coverage of the Thinking Machines Interaction Models announcement. deeplearning.ai/the-batch