Speech Recognition

How end-to-end models solve alignment

Two ideas dominate, and modern systems often combine them:

• CTC (Connectionist Temporal Classification). The model predicts a label for every frame, including a special "blank" symbol, then collapses repeats and blanks. This lets it output "hello" from a hundred frames without anyone hand-labeling which frames were the "h". CTC sums over all valid frame-to-text alignments during training, so the model never needs a single ground-truth alignment.
• Attention-based sequence-to-sequence. A decoder emits one token at a time, using attention to look back at whichever encoder frames are relevant for the token it is producing now. Alignment becomes something the attention mechanism learns, rather than an explicit step.

Large modern ASR models (trained on very large, weakly supervised datasets spanning many languages and acoustic conditions) push word error rates on clean speech to within a few points of human transcribers, and are robust enough to work out of the box on everyday audio.

Word error rate

The standard metric is word error rate (WER): how many word-level edits are needed to turn the model's output into the reference transcript, normalized by the reference length. It counts three kinds of error, substitutions ($S$, a wrong word), deletions ($D$, a missed word), and insertions ($I$, an added word), against $N$, the number of words in the reference:

$\text{WER} = \frac{S + D + I}{N}$

Worked example. Reference (5 words): "the cat sat on the". Model output: "the cat sat in". That is one substitution (on becomes in) and one deletion (the final the is missing), with no insertions:

$\text{WER} = \frac{1 + 1 + 0}{5} = 0.40 = 40\%$

Two things to note. WER can exceed 100% (many insertions on a short reference), and a low aggregate WER can still hide serious failures on specific words, which is why the number is a starting point, not the whole story.

Where recognition still fails

Even strong systems degrade sharply under conditions that humans handle easily:

• Noise and reverberation. Background chatter, music, wind, or a boomy room mask the speech and drive WER up.
• Accents and dialects. Models underperform on accents underrepresented in training data, a real fairness problem, not just an accuracy one.
• Overlapping speakers. Two people talking at once is the "cocktail party problem"; without speaker separation, transcripts garble.
• Rare and domain-specific words. Names, medical or legal terms, and product jargon are often outside the model's learned vocabulary and get mangled.
• Code-switching. Mixing languages mid-sentence confuses models trained to expect one language at a time.

Further reading

• Sequence Modeling With CTC, a clear, visual explanation of the alignment problem and CTC.
• Robust Speech Recognition via Large-Scale Weak Supervision (Whisper), the paper behind a widely used, robust multilingual ASR model.
• Speech and Language Processing (Jurafsky & Martin), ASR chapter, WER, acoustic modeling, and decoding in depth.
• NIST: speech recognition evaluation tools, how WER is measured in standardized evaluations.