Speech: signal processing, CTC, and end-to-end spoken language models

Why this subject matters now

For two decades a speech recognizer was a pipeline of separately trained parts: a feature extractor producing mel-frequency cepstral coefficients, a Gaussian mixture model scoring each frame against context-dependent phone states, a hidden Markov model gluing the states into words through a pronunciation lexicon, and an n-gram language model rescoring the lattice. Each part had its own objective and its own failure modes, and building one was a specialist craft. That world is essentially gone from new systems. What a practitioner is expected to know today is a different stack: a self-supervised encoder pretrained on hundreds of thousands of hours of unlabeled audio, a loss such as connectionist temporal classification or a transducer that aligns audio to text without a lexicon, and, increasingly, a discrete audio codec that turns a waveform into a short sequence of tokens a transformer can model exactly as it models text. Speech has become another modality inside the language-model program rather than a separate discipline.

The ideas did not vanish, though, they were absorbed, and the reasons they existed still shape the replacements. The alignment problem that the HMM solved with dynamic programming is the same problem CTC solves with a differentiable forward-backward pass; the conditional-independence assumption that makes CTC tractable is exactly the weakness the transducer's joint network repairs; the mel filterbank that fed the GMM still feeds the neural encoder, minus the final cepstral step. A strong practitioner is asked why the blank symbol matters, why log-mel replaced MFCCs when the model became a neural net, what residual vector quantization buys over a single codebook, and why a non-autoregressive text-to-speech model needs an explicit duration predictor. Those are questions about the material here. The end-to-end training of the decoder-only transformer that a speech LM ultimately hands its tokens to lives on the language-models-from-scratch page; the linguistic and neural machinery of language modeling proper is on the neural NLP page. The pure digital-signal-processing of the FFT, filter design, and sampling theory will get its own page; here spectral features are derived only to the depth the recognizer needs.

1990s-2012          2006-2015              2015-2020            2020-now
HMM-GMM             CTC / neural           attention +          self-supervised +
pipeline            acoustic models        transducer           discrete codecs
↓                   ↓                      ↓                    ↓
MFCC + GMM +        deep nets replace      LAS, Whisper,        wav2vec 2.0, HuBERT,
HMM + n-gram LM,    the GMM; CTC drops     RNN-T stream         EnCodec tokens, audio
forced alignment    the forced alignment   on device           LMs and neural TTS

From waveform to features

Framing and windowing

A digitized utterance is a sequence of amplitude samples \( x[0], x[1], \dots, x[N-1] \), taken at a sampling rate \( f_s \); telephone speech is sampled at 8 kHz and most modern systems at 16 kHz, which by the Nyquist criterion resolves frequencies up to \( f_s/2 \), 8 kHz, enough for the phonetically important band. Speech is not stationary: the vocal tract reconfigures every few tens of milliseconds as one phone gives way to the next. But on a short enough interval it is approximately stationary, so the first move is always to cut the signal into overlapping frames. A frame length of \( 25\,\text{ms} \) and a hop (frame shift) of \( 10\,\text{ms} \) are near-universal. At 16 kHz that is a window of \( L = 0.025 \times 16000 = 400 \) samples advanced by \( H = 0.010 \times 16000 = 160 \) samples, so consecutive frames overlap by 240 samples, or 60 percent.

Chopping the signal with a rectangular boundary is a multiplication by a boxcar in time, which is a convolution with a sinc in frequency, and the sinc's tall side lobes smear energy across the spectrum (spectral leakage). A tapered window suppresses the discontinuity at the frame edges. The Hann window

and the closely related Hamming window are the standard choices; both fall smoothly to (near) zero at the edges, trading a slightly wider main lobe for far lower side lobes. The framed, windowed signal is \( x_t[n] = w[n]\, x[tH + n] \), one length-\( L \) vector per frame index \( t \).

The short-time Fourier transform

Each frame is mapped to frequency by the discrete Fourier transform. With an FFT size \( K \ge L \) (the frame is zero-padded to \( K \), commonly \( K = 512 \) for a 400-sample frame),

The collection \( \{X_t[k]\} \) over all frames \( t \) and bins \( k \) is the short-time Fourier transform (STFT), and \( |X_t[k]|^2 \) is the power spectrogram, an image with time on one axis and frequency on the other. Because \( x_t \) is real, \( X_t[k] = \overline{X_t[K-k]} \), so only the \( K/2 + 1 \) non-negative frequencies carry information: for \( K = 512 \) that is 257 bins spanning 0 to \( f_s/2 \). The number of frames for a signal of \( N \) samples with no padding is \( \lfloor (N-L)/H \rfloor + 1 \); libraries that center each frame by reflecting the signal at the boundaries instead report \( 1 + \lfloor N/H \rfloor \). Both counts appear in Problem 1.

The mel filterbank, derived

A 257-bin power spectrum is finer than perception warrants and finer than a model needs. Human pitch perception is roughly logarithmic: the perceived distance from 200 to 400 Hz is about the same as from 2000 to 4000 Hz, a doubling either way, not the same number of hertz. The mel scale (Stevens, Volkmann and Newman, 1937) encodes this. The standard analytic form used in nearly every toolkit is

linear below roughly 1 kHz and logarithmic above, with the constants chosen so that 1000 Hz maps near 1000 mel. To build a bank of \( B \) filters spanning \( [f_{\min}, f_{\max}] \), place \( B+2 \) points equally spaced on the mel axis, map them back to hertz, and snap each to the nearest FFT bin:

Filter \( i \) (for \( i = 1,\dots,B \)) is a triangle rising from bin \( b_{i-1} \) to a peak of 1 at \( b_i \) and falling back to 0 at \( b_{i+1} \), so adjacent filters share a center and overlap by half:

The mel energies are the filtered power sums \( E_i = \sum_{k} H_i[k]\, |X_t[k]|^2 \), and the log-mel spectrogram is \( \log(E_i + \varepsilon) \) with a small floor \( \varepsilon \) to keep the logarithm finite in silence. The logarithm matters for two reasons: it matches the roughly logarithmic loudness response, and it turns the multiplicative interaction of a source spectrum with the vocal-tract filter into an additive one, which is what makes the next step, the cepstrum, work. A typical bank has \( B = 80 \) filters for neural systems, or 23 to 40 for classical MFCC front ends.

MFCCs, and why log-mel replaced them for neural models

The log-mel energies of neighboring filters are highly correlated, because the triangular filters overlap and because the vocal-tract transfer function is smooth across frequency. A diagonal-covariance Gaussian, the workhorse of the GMM acoustic model, badly mismodels correlated features, so classical front ends decorrelate the log-mel vector with a discrete cosine transform:

The result is the mel-frequency cepstral coefficients. Keeping only the first 13 coefficients is a low-pass lifter on the cepstral axis: the low-order coefficients capture the slowly varying spectral envelope (the vocal-tract shape that identifies the phone), while the discarded high-order coefficients hold the fast ripple of the pitch harmonics, which is speaker and pitch information the phone classifier does not want. The DCT also approximately diagonalizes the covariance, which is exactly what the diagonal-Gaussian GMM needed. Deltas and delta-deltas, finite differences across frames, were appended to inject the local dynamics a frame-independent GMM could not see.

A neural network wants none of this. A convolutional or transformer encoder models correlations across frequency and time directly, so the decorrelating DCT throws away structure the network could have used, and stacked frames make explicit delta features redundant. The DCT is also a fixed linear map; folding it into the first learned layer costs nothing and can only help. The field therefore moved back one step: modern acoustic models consume 80-dimensional log-mel spectrograms (and self-supervised models such as wav2vec 2.0 skip even the filterbank and learn a convolutional front end from the raw waveform). MFCCs survive in low-resource, low-compute, and speaker/keyword tasks where a compact decorrelated feature and a simple classifier still make sense.

The HMM-GMM recognizer

Before end-to-end training there was the hidden Markov model, and it is worth deriving because the alignment problem it addresses is the problem every later loss inherits. Speech has an unknown correspondence between the frame sequence and the label sequence: a single phone spans a variable, unknown number of frames, and that duration changes with speaking rate. The HMM models an utterance as a Markov chain of hidden states (context-dependent sub-phone states), each emitting one observation per frame, so the alignment becomes a hidden variable to be marginalized or maximized over.

The three problems

An HMM is \( \lambda = (\pi, A, B) \): an initial distribution \( \pi_i = \P(q_1 = i) \), a transition matrix \( A_{ij} = \P(q_{t+1} = j \mid q_t = i) \), and emission densities \( b_j(o_t) = \P(o_t \mid q_t = j) \), which in an HMM-GMM are Gaussian mixtures over the acoustic feature vector. Rabiner's 1989 tutorial organizes everything into three questions. Evaluation: given \( \lambda \) and an observation sequence \( O \), compute the likelihood \( \P(O \mid \lambda) \); solved by the forward algorithm, a dynamic program over the trellis with \( \alpha_t(j) = \P(o_1{:}t, q_t = j) \) and the recurrence \( \alpha_{t+1}(j) = \big[\sum_i \alpha_t(i) A_{ij}\big] b_j(o_{t+1}) \). Decoding: find the single most likely state path \( q^\star = \argmax_q \P(q \mid O, \lambda) \); solved by Viterbi, which replaces the sum with a max and stores back-pointers. Learning: re-estimate \( \lambda \) to maximize \( \P(O \mid \lambda) \); solved by Baum-Welch, the expectation-maximization instance whose E-step is the forward-backward pass and whose M-step reweights transitions and re-fits the Gaussians with the posterior state occupancies \( \gamma_t(j) = \alpha_t(j)\beta_t(j)/\P(O) \).

Viterbi

Viterbi carries the probability of the best path ending in each state, \( \delta_t(j) = \max_{q_1:q_{t-1}} \P(q_1{:}t{-}1, q_t = j, o_1{:}t) \), with the recurrence

and the best final state \( \argmax_j \delta_T(j) \) is traced back through the stored \( \psi \). In a real recognizer this trellis is the composition of the acoustic HMM with the pronunciation lexicon and the language-model automaton, and the search is beam-pruned, but the recurrence is exactly the one worked in Problem 2.

Connectionist temporal classification

The HMM works but is cumbersome: it needs a pronunciation lexicon, a forced alignment to bootstrap the state boundaries, and a separately trained acoustic model. Connectionist temporal classification (Graves, Fernández, Gomez and Schmidhuber, 2006) removes the alignment entirely. It trains a single network that emits, at each of \( T \) input frames, a distribution over an alphabet augmented with one extra symbol, and it defines a loss that sums over all alignments consistent with the target transcription. Nothing outside the network is trained, and no frame-level labels are ever needed.

Paths, the blank, and the collapse

Let the label alphabet be \( \mathcal{A} \) and augment it with a blank symbol \( \varnothing \), giving \( \mathcal{A}' = \mathcal{A} \cup \{\varnothing\} \). The network outputs a probability \( y^t_k \) for each symbol \( k \in \mathcal{A}' \) at each frame \( t \), normalized by a softmax. A path \( \pi \in \mathcal{A}'^{\,T} \) is one choice of symbol per frame, and under the CTC conditional-independence assumption its probability is simply the product across frames,

The many-to-one map \( \mathcal{B} \) collapses a path to a label sequence by first merging runs of identical symbols and then deleting blanks: \( \mathcal{B}(a\,a\,\varnothing\,a\,b) = a\,a\,b \) and \( \mathcal{B}(\varnothing\,a\,\varnothing\,\varnothing\,b) = a\,b \). The probability of a target labelling \( \ell \) is the total mass of every path that collapses to it,

and the CTC loss is \( -\log \P(\ell \mid x) \). The blank is not cosmetic. It gives the network a way to say "no new label here", which is what lets a single label persist across many frames without emitting duplicates, and, crucially, it is the only way to emit a genuine double letter: to produce \( a\,a \) the path must place a blank between the two \( a \)'s, as in \( a\,\varnothing\,a \), because two adjacent \( a \)'s in a path collapse to one. Without a blank the model could neither hold a label over a long segment nor spell a repeated character.

The forward-backward recurrence, derived

Enumerating \( \mathcal{B}^{-1}(\ell) \) is intractable; there are exponentially many paths. The trick, exactly as in the HMM, is a dynamic program over a modified label sequence. Build \( \ell' \) by inserting a blank between every pair of labels and at both ends, so a target of length \( U \) becomes \( \ell' \) of length \( S = 2U + 1 \). For \( \ell = a\,b \), \( \ell' = \varnothing\, a\, \varnothing\, b\, \varnothing \). Every valid path corresponds to a monotone left-to-right walk through the trellis whose rows are the positions of \( \ell' \) and whose columns are the frames. Define the forward variable

the total probability of all path prefixes that have consumed exactly the first \( s \) symbols of \( \ell' \) by frame \( t \). The allowed moves into node \( s \) at frame \( t \) are: stay on the same symbol \( (s \to s) \); advance by one \( (s{-}1 \to s) \); or, only when the current symbol is a non-blank that differs from the symbol two positions back, skip the intervening blank \( (s{-}2 \to s) \). The skip is forbidden when \( \ell'_s = \varnothing \) (you cannot skip over a real label) and when \( \ell'_s = \ell'_{s-2} \) (a blank between two identical labels is mandatory, exactly the double-letter rule). This gives the recurrence

with initial conditions \( \alpha_1(1) = y^1_{\varnothing} \), \( \alpha_1(2) = y^1_{\ell'_2} \), and \( \alpha_1(s) = 0 \) for \( s > 2 \), since a path can begin only in the leading blank or the first real label. The total likelihood is the mass that has consumed all of \( \ell' \) by the last frame, arriving in either the final label or the final blank:

The backward variable \( \beta_t(s) \), the probability of completing \( \ell' \) from \( s \) onward, obeys the mirror-image recurrence, and the product \( \alpha_t(s)\beta_t(s) \) is the probability of all complete paths that pass through node \( s \) at frame \( t \), which is what the gradient needs.

The loss gradient

Because a path's probability contains \( y^t_{\ell'_s} \) exactly once at node \( (s,t) \), the product \( \alpha_t(s)\beta_t(s) \) contains that factor twice, so dividing once recovers the path mass through the node: \( \alpha_t(s)\beta_t(s)/y^t_{\ell'_s} \) sums the probabilities of all paths through \( (s,t) \), and \( \P(\ell \mid x) = \sum_s \alpha_t(s)\beta_t(s)/y^t_{\ell'_s} \) for any fixed frame \( t \). Differentiating with respect to the softmax output \( y^t_k \) picks out only the nodes whose symbol is \( k \), the set \( \mathrm{lab}(\ell', k) = \{ s : \ell'_s = k \} \), giving

Pushing this through the softmax, whose Jacobian is \( \partial y^t_k / \partial a^t_j = y^t_k(\delta_{kj} - y^t_j) \), the gradient of the loss \( \mathcal{L} = -\log \P(\ell \mid x) \) with respect to the pre-softmax logit \( a^t_k \) collapses to the clean form

This is exactly the structure of softmax cross-entropy, output minus target, but the target is not a one-hot label; it is the posterior occupancy \( \sum_s \alpha_t(s)\beta_t(s)/\P(\ell\mid x) \), the expected number of times symbol \( k \) is aligned to frame \( t \) under all valid alignments. CTC thus manufactures its own soft targets from the forward-backward pass and trains against them, which is why it needs no external alignment.

The conditional-independence assumption

The tractability comes at a price. \( \P(\pi \mid x) = \prod_t y^t_{\pi_t} \) assumes the per-frame outputs are conditionally independent given the input, so CTC cannot model that the letter after \( q \) is almost always \( u \): the acoustic evidence must carry that on its own. In practice CTC systems recover the missing language modeling by decoding with an external n-gram or neural LM through a beam search (prefix-beam or WFST composition). The transducer, next, keeps the CTC alignment machinery but adds an internal autoregressive path over the output, so it does not need the assumption.

The RNN transducer

The RNN transducer (Graves, 2012) keeps CTC's monotonic alignment lattice but removes the conditional-independence assumption by adding a small autoregressive language model over the emitted symbols. It has three parts: an encoder (or transcription network) that maps the acoustic frames to \( f_t \), a prediction network that is an autoregressive RNN over the previously emitted non-blank labels producing \( g_u \), and a joint network that combines them,

over the alphabet plus blank. The lattice is now two-dimensional, indexed by frame \( t \) and output position \( u \): emitting a non-blank advances \( u \); emitting a blank advances \( t \). The forward variable \( \alpha(t, u) \) satisfies

where \( \varnothing(t,u) \) is the blank probability and \( y(t,u) \) the probability of the next target symbol at node \( (t,u) \), and the total likelihood is \( \alpha(T, U)\,\varnothing(T, U) \). Because the prediction network conditions on the emitted history, the transducer captures the \( q \to u \) dependence CTC cannot, and because the recurrence is still a monotone forward pass, it streams: it can emit output as audio arrives without seeing the whole utterance, which is why it is the dominant loss for on-device and low-latency recognition. The cost is memory, the joint tensor is \( T \times U \times |\mathcal{A}'| \), which motivated function-merging and pruned-transducer implementations discussed in the practice section.

Attention encoder-decoders and large-scale supervision

Listen, Attend and Spell

A different route drops the monotonic lattice altogether and treats recognition as sequence-to-sequence translation from audio to text. Listen, Attend and Spell (Chan, Jaitley, Le and Vinyals, 2016) pairs a pyramidal BiLSTM encoder (the listener, which downsamples the frame rate so the decoder attends over a shorter sequence) with an attention decoder (the speller) that autoregressively emits characters, at each step forming a context vector

and predicting the next character from \( s_i \) and \( c_i \). Unlike CTC and the transducer, the attention is unconstrained and can look anywhere, which makes the model expressive but also prone to failures peculiar to speech: skipping or repeating whole segments when the soft alignment loses monotonicity. Chorowski, Bahdanau, Serdyuk, Cho and Bengio (2015) addressed exactly this with location-aware attention that conditions each step's scores on the previous alignment, and with windowing and monotonicity penalties, foreshadowing the hybrid CTC/attention training that stabilizes modern encoder-decoders by adding a CTC loss on the encoder as an alignment regularizer.

Whisper and weak supervision at scale

Whisper (Radford, Kim, Xu, Brockman, McLeavey and Sutskever, 2022) is the attention encoder-decoder taken to its data-centric conclusion: a plain transformer trained on 680,000 hours of weakly labeled, multilingual, multitask audio scraped from the web, with no self-supervised pretraining stage. The decoder is prompted with special tokens that select the task (transcribe versus translate), the language, and whether to predict timestamps, so a single model does multilingual recognition, X-to-English speech translation, language identification, and voice-activity detection. The lesson Whisper drove home is that scale and diversity of supervision buy robustness: it degraded far less than models trained on clean academic corpora when moved to noisy, accented, out-of-distribution audio, without any dataset-specific fine-tuning. It also inherited the encoder-decoder's failure modes, hallucinated text in silence and repetition loops, which its decoding heuristics (temperature fallback, compression-ratio and log-probability thresholds, no-speech detection) exist to suppress.

Self-supervised speech representations

wav2vec 2.0

Labeled speech is scarce and expensive; unlabeled speech is effectively unlimited. wav2vec 2.0 (Baevski, Zhou, Mohamed and Auli, 2020) learns representations from raw audio with no transcripts, then fine-tunes on a small labeled set with CTC. A convolutional feature encoder maps the waveform to latent vectors \( z_t \) at about a 20 ms stride; a span of these is masked (as in BERT), and a transformer context network produces contextual representations \( c_t \) over the masked and unmasked positions. The self-supervised objective is contrastive: at each masked step the model must identify the true quantized latent \( q_t \) among distractors sampled from other masked steps,

with \( Q_t \) the set of one positive and \( K \) negatives and \( \kappa \) a temperature. Predicting raw continuous latents would be ill-posed, so the targets are discretized. The quantization uses product quantization with a Gumbel-softmax: the latent is split into \( G \) groups, each mapped by a differentiable soft argmax to one of \( V \) codebook entries, and the chosen entries are concatenated, giving \( V^G \) possible discrete units while keeping the selection trainable through the straight-through Gumbel estimator. A diversity penalty on the codebook usage keeps all entries active. Fine-tuned on 10 minutes of labeled speech, wav2vec 2.0 reached word error rates that previously required hundreds of hours, which is the result that made self-supervised pretraining standard.

HuBERT and the predictive alternative

HuBERT (Hsu, Bolte, Tsai, Lakhotia, Salakhutdinov and Mohamed, 2021) reaches similar quality with a simpler recipe that sidesteps the contrastive machinery. It first clusters MFCC (later, learned) features with k-means to assign every frame a discrete pseudo-label, then trains a BERT-style masked-prediction model to guess the cluster id of masked frames, and iterates: the improved representations are re-clustered to produce better targets for the next round. The predictive-versus-contrastive split, and the closely related BEST-RQ approach that replaces learned quantization with a random-projection codebook, is one of the live design axes in self-supervised speech, and both feed directly into the discrete-token view that audio language models depend on.

Speaker diarization

Recognition answers what was said; diarization answers who spoke when, partitioning an audio stream into speaker-homogeneous segments without knowing the speakers in advance. The classical pipeline extracts a fixed-dimensional speaker embedding (an i-vector, or a neural x-vector from a time-delay network trained to discriminate speakers) from short windows, then clusters the embeddings, historically with agglomerative hierarchical clustering under a probabilistic linear discriminant analysis similarity, more recently with spectral clustering. The clustering approach cannot represent overlapping speech, which is where end-to-end neural diarization reframes the task as per-speaker voice-activity detection with a permutation-invariant training loss, allowing two speakers to be active in the same frame. Diarization composes with recognition to produce speaker-attributed transcripts, the "who said what" output that meeting and call-center systems need.

Neural audio codecs and audio language models

A speech waveform at 16 kHz is 16,000 floating-point numbers per second, far too many for a transformer to model directly. A neural audio codec compresses it into a short sequence of discrete tokens that a language model can then treat exactly like text. SoundStream (Zeghidour, Luebs, Omran, Skoglund and Tagliasacchi, 2021) and EnCodec (Défossez, Copet, Synnaeve and Adi, 2022) are convolutional encoder-decoder autoencoders with a vector-quantized bottleneck, trained with a combination of reconstruction and adversarial losses to sound natural at very low bitrates.

Residual vector quantization

A single codebook of \( V \) entries carries only \( \log_2 V \) bits per frame, not nearly enough for high-fidelity audio; a codebook large enough for the needed bitrate would be astronomically big and impossible to train. Residual vector quantization (RVQ) stacks \( Q \) small codebooks so that each quantizes the residual error left by the previous ones. With encoder output \( z \), set the initial residual \( r_0 = z \), and for \( q = 1, \dots, Q \) choose the nearest entry \( e_q = \operatorname{VQ}_q(r_{q-1}) \) and subtract it, \( r_q = r_{q-1} - e_q \); the reconstruction is \( \hat z = \sum_{q=1}^{Q} e_q \). With \( Q \) codebooks of \( V \) entries each, RVQ represents \( V^Q \) points using only \( Q V \) vectors and \( Q \log_2 V \) bits per frame, a coarse-to-fine description that also degrades gracefully: keeping the first few codebooks already reconstructs a usable signal, which enables bitrate scalability. The output of a codec is therefore not one token stream but \( Q \) parallel streams, and how an audio LM factorizes across them (flat, coarse-then-fine as in AudioLM, or delayed-interleaved as in MusicGen) is the central modeling choice for generative audio.

waveform  ->  conv encoder  ->  z  ->  RVQ (Q codebooks)  ->  Q token streams
16 kHz         downsample        latent    r0=z                      c^1_t ... c^Q_t
                                           e_q = VQ_q(r_{q-1})              |
                                           r_q = r_{q-1} - e_q             v
                                           zhat = sum e_q          language model
                                                                   over discrete tokens
                                                                          |
waveform  <-  conv decoder  <-  zhat  <-  lookup + sum  <-------------- sampled tokens

Text-to-speech

Synthesis runs the pipeline in reverse: text to a spectrogram to a waveform. The modern decomposition is an acoustic model that predicts a log-mel spectrogram from text, followed by a vocoder that inverts the spectrogram to audio, because predicting 80 mel channels at a 10 ms frame rate is far easier than predicting 16,000 raw samples per second directly.

Tacotron 2 and attention alignment

Tacotron 2 (Shen and colleagues, 2018) is an attention encoder-decoder that maps a character sequence to a mel spectrogram, one frame at a time, exactly the attention mechanism of the ASR decoder run in the generative direction. Its central difficulty is the same monotonicity problem: because text and speech proceed left to right together, a good alignment is a near-diagonal band, and when the soft attention wanders the output stutters, skips words, or babbles. This is why location-sensitive attention and, later, monotonic and forward-attention variants were needed. Tacotron 2 predicts the spectrogram and a separate stop token, and its autoregressive frame-by-frame generation makes it accurate but slow and occasionally unstable, the two problems the next design targets.

FastSpeech: non-autoregressive synthesis with explicit durations

FastSpeech (Ren and colleagues, 2019) and FastSpeech 2 (Ren and colleagues, 2020) replace learned attention with an explicit duration model. A duration predictor outputs, for each input phoneme, how many mel frames it should occupy; a length regulator then expands the phoneme sequence by repeating each hidden state that many times, producing a frame-level sequence the decoder converts to a spectrogram in a single parallel pass. Because there is no autoregression and no soft attention, the model is an order of magnitude faster and cannot skip or repeat: the alignment is enforced by construction rather than learned and hoped for. The durations for training come from an external aligner (a teacher's attention, or a forced alignment), and FastSpeech 2 adds explicit pitch and energy predictors so the same text can be rendered with controllable prosody. The trade is that the one-to-many nature of speech, one sentence has many valid renderings, is handled by these explicit variance predictors rather than by a flexible attention, which is why later systems reintroduce stochasticity through flows and diffusion.

Vocoders: WaveNet to HiFi-GAN to diffusion

The vocoder turns the mel spectrogram into a waveform. WaveNet (van den Oord and colleagues, 2016) was the breakthrough: an autoregressive model over raw samples, \( \P(x) = \prod_n \P(x_n \mid x_{<n}) \), built from dilated causal convolutions whose receptive field grows exponentially with depth, reaching thousands of samples of context while staying tractable. It sounded unprecedentedly natural and was unusably slow, generating one sample at a time. Parallel WaveNet (van den Oord and colleagues, 2018) fixed the speed with probability density distillation: a parallel inverse-autoregressive-flow student is trained to match the autoregressive teacher's distribution, generating a whole utterance in one pass. The field then moved to GAN vocoders: HiFi-GAN (Kong, Kim and Bae, 2020) is a fully convolutional generator with multi-period and multi-scale discriminators that judge the waveform at several temporal resolutions, reaching near-WaveNet quality at orders-of-magnitude higher speed, which is why it became the default neural vocoder. The current frontier adds diffusion vocoders (DiffWave, WaveGrad), which invert the spectrogram by learning to denoise, trading a few sampling steps for high fidelity and stable training, and end-to-end models such as VITS that fold the acoustic model and vocoder into one variational, adversarially trained network.

Evaluation: word error rate and its limits

Recognition quality is reported as word error rate, the normalized Levenshtein (edit) distance between the hypothesis and the reference transcript at the word level,

where \( S, D, I \) are the counts of each edit in the minimum-cost alignment and \( N \) is the reference length. The minimum edit count comes from the same dynamic program as string edit distance, with a table \( D[i,j] \) giving the cost of aligning the first \( i \) reference words to the first \( j \) hypothesis words:

with \( D[i,0] = i \) and \( D[0,j] = j \). WER can exceed 1 (a hypothesis longer than the reference can accrue more insertions than there are reference words), and it weights every word equally, which is its central defect: deleting "not" and misspelling a name cost the same one substitution, though one inverts the meaning and the other barely dents it. WER is also sensitive to text normalization, casing, punctuation, numbers spelled out versus digits, and it ignores everything a downstream task cares about, which is why speech-translation and spoken-language-understanding systems report task metrics (BLEU, intent accuracy) in addition to, or instead of, WER.

Implementation

The four computations worked above are implemented below and cross-checked. First the CTC forward pass in log space, the numerically stable form used in every real trainer, in PyTorch and JAX; its output on the Problem 3 inputs is \( \log \P = \log 0.486 = -0.72155 \), matching torch.nn.functional.ctc_loss.

import torch

NEG = -1e30  # log(0)

def log_add(a, b):
    # numerically stable log(exp(a) + exp(b))
    m = torch.maximum(a, b)
    m = torch.where(torch.isinf(m), torch.zeros_like(m), m)
    return m + torch.log(torch.exp(a - m) + torch.exp(b - m))

def ctc_forward_logprob(log_y, target, blank):
    # log_y: (T, C) log-softmax outputs; target: (U,) label ids; returns log P(target | x)
    T, C = log_y.shape
    U = target.shape[0]
    S = 2 * U + 1
    ext = torch.full((S,), blank, dtype=torch.long)     # blank, l1, blank, l2, ...
    ext[1::2] = target
    a = torch.full((S,), NEG)                            # alpha at current frame, log space
    a[0] = log_y[0, blank]
    a[1] = log_y[0, ext[1]]
    for t in range(1, T):
        prev = a.clone()
        a = torch.full((S,), NEG)
        for s in range(S):
            acc = prev[s]
            if s - 1 >= 0:
                acc = log_add(acc, prev[s - 1])
            # skip transition only across a mandatory blank between distinct labels
            if s - 2 >= 0 and ext[s] != blank and ext[s] != ext[s - 2]:
                acc = log_add(acc, prev[s - 2])
            a[s] = acc + log_y[t, ext[s]]
    return log_add(a[S - 1], a[S - 2])

y = torch.tensor([[0.6, 0.1, 0.3],
                  [0.2, 0.5, 0.3],
                  [0.1, 0.6, 0.3]])                      # (T=3, C=3), symbols a,b,blank
log_y = torch.log(y)
target = torch.tensor([0, 1])                            # "a b"
lp = ctc_forward_logprob(log_y, target, blank=2)
print(float(lp), float(torch.exp(lp)))                   # -0.72155  0.48600

# cross-check against the library implementation
ref = torch.nn.functional.ctc_loss(
    log_y.unsqueeze(1), target,
    torch.tensor([3]), torch.tensor([2]), blank=2, reduction='none')
print(float(-ref))                                       # -0.72155

import jax, jax.numpy as jnp
from jax.scipy.special import logsumexp

NEG = -1e30

def ctc_forward_logprob(log_y, target, blank):
    # log_y: (T, C) log-softmax outputs; target: (U,) label ids
    T, C = log_y.shape
    U = target.shape[0]
    S = 2 * U + 1
    ext = jnp.full((S,), blank, dtype=jnp.int32).at[1::2].set(target)  # blank, l1, blank, ...
    # precompute which rows allow the skip (distinct-label, non-blank)
    can_skip = jnp.array([(s >= 2) and (int(ext[s]) != blank)
                          and (int(ext[s]) != int(ext[s - 2])) for s in range(S)])

    a0 = jnp.full((S,), NEG).at[0].set(log_y[0, ext[0]]).at[1].set(log_y[0, ext[1]])

    def step(a_prev, log_yt):
        stay = a_prev
        adv = jnp.concatenate([jnp.array([NEG]), a_prev[:-1]])            # alpha(s-1)
        skp = jnp.concatenate([jnp.array([NEG, NEG]), a_prev[:-2]])       # alpha(s-2)
        skp = jnp.where(can_skip, skp, NEG)
        terms = jnp.stack([stay, adv, skp], axis=0)                       # (3, S)
        a = logsumexp(terms, axis=0) + log_yt[ext]
        return a, a

    a_last, _ = jax.lax.scan(step, a0, log_y[1:])
    return logsumexp(jnp.array([a_last[S - 1], a_last[S - 2]]))

y = jnp.array([[0.6, 0.1, 0.3],
               [0.2, 0.5, 0.3],
               [0.1, 0.6, 0.3]])                                          # (T=3, C=3)
lp = ctc_forward_logprob(jnp.log(y), jnp.array([0, 1]), blank=2)
print(float(lp), float(jnp.exp(lp)))                                      # -0.72155  0.48600

Next the log-mel feature extractor: frame, window, STFT, apply the derived triangular mel filterbank, and take the log. Running the PyTorch version on a batch of sixteen one-second clips on an NVIDIA H100 80GB HBM3 (132 SMs, PyTorch 2.7, CUDA 12.8) produces an output of shape \( (16, 80, 101) \), the 101 frames matching the centered frame count from Problem 1, in 0.093 ms per batch after warmup.

import torch, math

def hz_to_mel(f): return 2595.0 * math.log10(1.0 + f / 700.0)
def mel_to_hz(m): return 700.0 * (10.0 ** (m / 2595.0) - 1.0)

def mel_filterbank(sr=16000, n_fft=400, n_mels=80, fmin=0.0, fmax=8000.0, device='cuda'):
    # returns (n_mels, n_fft//2 + 1) triangular filters
    m_pts = torch.linspace(hz_to_mel(fmin), hz_to_mel(fmax), n_mels + 2)
    hz_pts = torch.tensor([mel_to_hz(m.item()) for m in m_pts])
    bins = torch.floor((n_fft + 1) * hz_pts / sr).long()          # snap to FFT bins
    fb = torch.zeros(n_mels, n_fft // 2 + 1, device=device)
    for i in range(1, n_mels + 1):
        l, c, r = bins[i - 1].item(), bins[i].item(), bins[i + 1].item()
        for k in range(l, c):
            if c > l: fb[i - 1, k] = (k - l) / (c - l)             # rising edge
        for k in range(c, r):
            if r > c: fb[i - 1, k] = (r - k) / (r - c)             # falling edge
    return fb

def log_mel(x, sr=16000, n_fft=400, hop=160, n_mels=80, device='cuda'):
    # x: (B, N) waveform -> (B, n_mels, frames)
    win = torch.hann_window(n_fft, device=device)
    S = torch.stft(x, n_fft, hop, win_length=n_fft, window=win, return_complex=True)
    power = S.abs() ** 2                                           # (B, n_fft//2+1, frames)
    fb = mel_filterbank(sr, n_fft, n_mels, device=device)          # (n_mels, n_fft//2+1)
    mel = torch.matmul(fb, power)                                  # (B, n_mels, frames)
    return torch.log(mel + 1e-6)

x = torch.randn(16, 16000, device='cuda')                         # 16 clips of 1 s
feats = log_mel(x)
print(tuple(feats.shape))                                         # (16, 80, 101)

import jax, jax.numpy as jnp
import numpy as np, math

def hz_to_mel(f): return 2595.0 * math.log10(1.0 + f / 700.0)
def mel_to_hz(m): return 700.0 * (10.0 ** (m / 2595.0) - 1.0)

def mel_filterbank(sr=16000, n_fft=400, n_mels=80, fmin=0.0, fmax=8000.0):
    m_pts = np.linspace(hz_to_mel(fmin), hz_to_mel(fmax), n_mels + 2)
    hz_pts = np.array([mel_to_hz(m) for m in m_pts])
    bins = np.floor((n_fft + 1) * hz_pts / sr).astype(int)
    fb = np.zeros((n_mels, n_fft // 2 + 1))
    for i in range(1, n_mels + 1):
        l, c, r = bins[i - 1], bins[i], bins[i + 1]
        if c > l: fb[i - 1, l:c] = (np.arange(l, c) - l) / (c - l)
        if r > c: fb[i - 1, c:r] = (r - np.arange(c, r)) / (r - c)
    return jnp.asarray(fb)

def frame(x, n_fft, hop):
    N = x.shape[-1]
    n = 1 + (N - n_fft) // hop
    idx = jnp.arange(n_fft)[None, :] + hop * jnp.arange(n)[:, None]
    return x[..., idx]                                            # (B, frames, n_fft)

def log_mel(x, sr=16000, n_fft=400, hop=160, n_mels=80):
    win = jnp.hanning(n_fft)
    frames = frame(x, n_fft, hop) * win                          # (B, frames, n_fft)
    spec = jnp.fft.rfft(frames, axis=-1)                         # (B, frames, n_fft//2+1)
    power = jnp.abs(spec) ** 2
    fb = mel_filterbank(sr, n_fft, n_mels)                       # (n_mels, n_fft//2+1)
    mel = jnp.einsum('btf,mf->bmt', power, fb)                   # (B, n_mels, frames)
    return jnp.log(mel + 1e-6)

x = jnp.asarray(np.random.randn(16, 16000))
feats = log_mel(x)
print(feats.shape)                                              # (16, 80, 98)  (uncentered)

Finally the two dynamic programs in plain Python: word error rate by edit distance with a traceback that labels each edit, and Viterbi for the HMM of Problem 2.

def word_error_rate(ref, hyp):
    r, h = ref.split(), hyp.split()
    n, m = len(r), len(h)
    D = [[0] * (m + 1) for _ in range(n + 1)]
    for i in range(n + 1): D[i][0] = i          # i deletions
    for j in range(m + 1): D[0][j] = j          # j insertions
    for i in range(1, n + 1):
        for j in range(1, m + 1):
            sub = D[i - 1][j - 1] + (r[i - 1] != h[j - 1])
            D[i][j] = min(D[i - 1][j] + 1,       # deletion
                          D[i][j - 1] + 1,       # insertion
                          sub)                   # match or substitution
    # traceback to count S, D, I
    i, j, S, De, I = n, m, 0, 0, 0
    while i > 0 or j > 0:
        if i > 0 and j > 0 and D[i][j] == D[i-1][j-1] + (r[i-1] != h[j-1]):
            S += (r[i-1] != h[j-1]); i, j = i-1, j-1
        elif i > 0 and D[i][j] == D[i-1][j] + 1:
            De += 1; i -= 1
        else:
            I += 1; j -= 1
    return (S + De + I) / n, (S, De, I)

wer, (S, De, I) = word_error_rate("the quick brown fox jumps over",
                                  "the quick fox jumped over the")
print(wer, S, De, I)                             # 0.5 1 1 1


def viterbi(pi, A, B, obs):
    # pi: (Ns,), A: (Ns, Ns), B: (Ns, No) emission, obs: list of symbol ids
    Ns, T = len(pi), len(obs)
    delta = [[0.0] * Ns for _ in range(T)]
    psi = [[0] * Ns for _ in range(T)]
    for j in range(Ns):
        delta[0][j] = pi[j] * B[j][obs[0]]
    for t in range(1, T):
        for j in range(Ns):
            scores = [delta[t-1][i] * A[i][j] for i in range(Ns)]
            psi[t][j] = max(range(Ns), key=lambda i: scores[i])
            delta[t][j] = scores[psi[t][j]] * B[j][obs[t]]
    last = max(range(Ns), key=lambda j: delta[T-1][j])
    path = [last]
    for t in range(T - 1, 0, -1):
        last = psi[t][last]; path.append(last)
    return path[::-1], max(delta[T-1])

pi = [0.6, 0.4]
A  = [[0.7, 0.3], [0.4, 0.6]]
B  = [[0.5, 0.4, 0.1], [0.1, 0.3, 0.6]]          # states A, B over o1,o2,o3
path, p = viterbi(pi, A, B, [0, 2, 2])           # observe o1, o3, o3
print(path, p)                                    # [0, 1, 1]  0.01944  (A -> B -> B)

How it is done in practice

A deployed recognizer is more engineering than any single loss. Production front ends run 80-channel log-mel at a 10 ms stride with global or per-utterance cepstral mean-variance normalization, and SpecAugment, masking random time and frequency bands of the spectrogram during training, is the single most reliable regularizer for speech, cheap and applied everywhere. Encoders are Conformers, transformer blocks interleaved with convolutions to capture both global and local structure, with the frame rate subsampled by a factor of four to eight before the transformer to cut the sequence length. Streaming systems overwhelmingly use the transducer, because it emits incrementally with bounded latency; batch and offline systems favor attention encoder-decoders or hybrid CTC/attention for accuracy. The two are often combined: a shared encoder feeds a CTC head for fast first-pass alignment and streaming, and an attention or transducer head for the accurate result, with the CTC branch also serving as an alignment regularizer that keeps attention monotonic during training.

Decoding rarely trusts the acoustic model alone. Weighted finite-state transducers compose the acoustic model, a lexicon, and an n-gram language model into one searchable graph, and neural-LM rescoring of the resulting lattice or n-best list adds the long-range language knowledge CTC's independence assumption omits. Transducer training in particular is memory-bound, as Problem 6 quantified, so the standard trainers fuse the joint network and loss and prune the lattice around a CTC alignment, cutting the \( T \times U \) tensor to a narrow band. On the synthesis side, latency and stability push toward the non-autoregressive FastSpeech family with a HiFi-GAN vocoder for real-time use, while the highest-fidelity offline systems use diffusion or flow-based decoders. The transformation of the field into "audio as tokens" is now the dominant industrial direction: an EnCodec-style codec turns speech into discrete units, a decoder-only transformer models them, and a single model does recognition, synthesis, translation, and dialogue, the architecture behind the current generation of speech-to-speech assistants.

The current research frontier

The most active line is the audio language model: treat discrete audio tokens exactly like text tokens and let one transformer generate them. AudioLM (Google, 2022) established the coarse-to-fine recipe, modeling semantic tokens from a self-supervised model and then acoustic tokens from a neural codec; VALL-E (Microsoft, 2023) cast text-to-speech as codec-token language modeling with in-context voice cloning from a three-second prompt; and MusicGen (Meta, 2023) applied the delayed-codebook-interleaving pattern to music. The generative-versus-discriminative codec question, whether to optimize a codec for reconstruction or for downstream language modeling, is unsettled: the Descript Audio Codec pushed reconstruction quality at very low token rates, while semantic-token codecs (SpeechTokenizer, Mimi in the Moshi full-duplex dialogue system from Kyutai) deliberately align the first codebook with self-supervised semantic features so the LM's early tokens carry meaning.

A second thread is scale and multilinguality. Meta's MMS extended self-supervised pretraining and CTC recognition to over a thousand languages; OWSM (from CMU and the ESPnet community) reproduced a Whisper-style model with fully open data and code; and Universal Speech Model and SeamlessM4T pushed massively multilingual recognition and direct speech-to-speech translation. A third is efficiency and streaming: chunked and cache-based Conformers, and the return of recurrent and state-space encoders (linear-attention and SSM variants) for constant-memory streaming, connect this material to the sequence-models and state-spaces page. A fourth is full-duplex spoken dialogue, models that listen and speak simultaneously rather than in strict turns, which forces the recognition, generation, and dialogue policy into a single streaming model and is where much of the speech-LM research energy now sits.

Open source to read

Each of these repays a focused read; the file listed is the right first door.

openai/whisper — the reference attention encoder-decoder for weakly supervised multilingual recognition. Open whisper/decoding.py to see the special-token task prompting and the temperature-fallback, compression-ratio, and no-speech heuristics that tame the decoder's hallucination and repetition modes.

facebookresearch/fairseq — the original wav2vec 2.0 and HuBERT implementations. Start under fairseq/models/wav2vec/wav2vec2.py: the convolutional feature encoder, the masking, the Gumbel product quantizer, and the contrastive loss are all there in one file.

speechbrain/speechbrain — the most readable end-to-end toolkit, with recipes for CTC, transducer, attention ASR, diarization, and TTS in a uniform style. Start at speechbrain/nnet/losses.py and the CTC and transducer recipes to connect the losses above to trainable systems.

NVIDIA/NeMo — production-grade Conformer-CTC and Conformer-Transducer with the memory-optimized transducer loss. Open nemo/collections/asr/parts/numba/rnnt_loss/ to see the fused, pruned transducer loss that makes the \( T \times U \) tensor of Problem 6 tractable.

k2-fsa/icefall — recipes built on the k2 differentiable-WFST library, the cleanest place to see modern pruned transducers and lattice-based decoding. Start at an egs/librispeech pruned-transducer recipe and follow it into the k2 loss.

coqui-ai/TTS — a broad synthesis toolkit with Tacotron 2, FastSpeech, VITS, and multiple vocoders side by side. Start at TTS/tts/models/ to compare the autoregressive-attention and duration-based acoustic models directly.

descriptinc/descript-audio-codec — a compact, high-quality neural codec. Open dac/nn/quantize.py for a clean residual-vector-quantization implementation, exactly the RVQ recurrence derived above, and the right starting point for understanding audio-LM tokenizers.

Common misconceptions

"The CTC blank is just padding for variable-length outputs." The blank is a real emission with two structural jobs: it lets one label persist over many frames without producing duplicates, and it is the only way to spell a repeated symbol, since a path must place a blank between two identical labels or the collapse merges them. Remove it and CTC can neither hold a label nor write "aa".

"CTC and the transducer are basically the same loss." They share the monotonic forward pass, but CTC assumes the per-frame outputs are conditionally independent given the audio, so it cannot model that the letter after "q" is usually "u"; the transducer's prediction network is an autoregressive LM over the emitted tokens, so it can. That is why transducers need no external LM to sound fluent and CTC systems usually do.

"MFCCs are strictly better features than log-mel spectrograms." The extra DCT step in MFCC exists to decorrelate features for a diagonal-covariance Gaussian; a neural network models cross-frequency correlation itself, so the DCT discards structure the net could use. Log-mel replaced MFCC precisely when the acoustic model became a neural net. MFCCs remain sensible for GMMs and compact low-compute classifiers.

"A lower word error rate always means a better system for the task." WER weights every word equally, so deleting "not" and mis-transcribing a hesitation cost the same. It is sensitive to text normalization and cannot see meaning; two systems with identical WER can differ sharply on the errors a downstream application actually cares about, which is why speech translation and understanding report task metrics alongside it.

"wav2vec 2.0 needs no labels at all." Its pretraining needs no labels, but it is not a recognizer until it is fine-tuned with a labeled loss, usually CTC. The achievement is the amount of labeled data the fine-tune needs, ten minutes rather than hundreds of hours, not the elimination of supervision.

"Attention-based ASR always aligns audio to text correctly." Unconstrained attention has no built-in monotonicity, so it can skip or repeat entire segments; the babbling and looping failures of early attention recognizers and of Whisper in silence are exactly this. Location-aware attention, monotonic variants, and a CTC alignment loss on the encoder exist to force the near-diagonal alignment that CTC and the transducer get for free.

"A neural vocoder is a nice-to-have; Griffin-Lim is basically fine." Griffin-Lim reconstructs a waveform from magnitude by iteratively guessing the discarded phase and sounds robotic and buzzy. The jump from Griffin-Lim to WaveNet and then HiFi-GAN is most of the perceived-quality gap in modern TTS; the vocoder is where naturalness is won or lost.

"A single large codebook would work as well as residual vector quantization." The codebook needed for high-fidelity audio at a useful bitrate would have astronomically many entries and be untrainable, because each entry is updated only when it is selected. RVQ stacks small codebooks that quantize successive residuals, reaching \( V^Q \) effective points with \( QV \) trainable vectors and, as a bonus, giving bitrate scalability by keeping a prefix of the codebooks.

Self-check

References

1. Jurafsky, D. and Martin, J. H. Speech and Language Processing, 3rd edition draft. The standard reference for acoustic features, HMMs, CTC, and evaluation; freely available online.
2. Rabiner, L. R. (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE 77(2), 257-286. doi:10.1109/5.18626
3. Graves, A., Fernández, S., Gomez, F. and Schmidhuber, J. (2006). Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. ICML.
4. Graves, A. (2012). Sequence transduction with recurrent neural networks. ICML Representation Learning Workshop. arXiv:1211.3711
5. Chan, W., Jaitly, N., Le, Q. and Vinyals, O. (2016). Listen, attend and spell: a neural network for large vocabulary conversational speech recognition. ICASSP. arXiv:1508.01211
6. Chorowski, J., Bahdanau, D., Serdyuk, D., Cho, K. and Bengio, Y. (2015). Attention-based models for speech recognition. NeurIPS. arXiv:1506.07503
7. Baevski, A., Zhou, H., Mohamed, A. and Auli, M. (2020). wav2vec 2.0: a framework for self-supervised learning of speech representations. NeurIPS. arXiv:2006.11477
8. Hsu, W.-N., Bolte, B., Tsai, Y.-H., Lakhotia, K., Salakhutdinov, R. and Mohamed, A. (2021). HuBERT: self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM TASLP. arXiv:2106.07447
9. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C. and Sutskever, I. (2022). Robust speech recognition via large-scale weak supervision. Whisper technical report. arXiv:2212.04356
10. Zeghidour, N., Luebs, A., Omran, A., Skoglund, J. and Tagliasacchi, M. (2021). SoundStream: an end-to-end neural audio codec. IEEE/ACM TASLP. arXiv:2107.03312
11. Défossez, A., Copet, J., Synnaeve, G. and Adi, Y. (2022). High fidelity neural audio compression (EnCodec). Transactions on Machine Learning Research. arXiv:2210.13438
12. Shen, J. et al. (2018). Natural TTS synthesis by conditioning WaveNet on mel spectrogram predictions (Tacotron 2). ICASSP. arXiv:1712.05884
13. Ren, Y. et al. (2019). FastSpeech: fast, robust and controllable text to speech. NeurIPS. arXiv:1905.09263
14. Ren, Y. et al. (2020). FastSpeech 2: fast and high-quality end-to-end text to speech. ICLR 2021. arXiv:2006.04558
15. van den Oord, A. et al. (2016). WaveNet: a generative model for raw audio. arXiv preprint. arXiv:1609.03499
16. van den Oord, A. et al. (2018). Parallel WaveNet: fast high-fidelity speech synthesis. ICML. arXiv:1711.10433
17. Kong, J., Kim, J. and Bae, J. (2020). HiFi-GAN: generative adversarial networks for efficient and high fidelity speech synthesis. NeurIPS. arXiv:2010.05646