Spectrograms & Features

The result is a matrix: one axis is time (one column per frame), the other is frequency (one row per frequency bin), and each cell's value is how much of that frequency was present in that frame.

Reading a spectrogram

A spectrogram is displayed as an image, and once you can read it, speech becomes legible at a glance:

• Horizontal axis, time. Left to right is the clip playing forward.
• Vertical axis, frequency. Low pitches at the bottom, high at the top.
• Color/brightness, energy. Bright means a lot of that frequency at that moment; dark means little. Magnitudes are almost always shown on a decibel (log) scale, because loudness spans an enormous range.

Vowels appear as stacked horizontal bands (the resonant frequencies, or formants, of the vocal tract); the hiss of an "s" or "f" shows as a broad smear of high-frequency energy; silence is a dark column. This is exactly why turning audio into an image is so powerful: it lets the same convolutional and transformer architectures that conquered vision do the heavy lifting on sound.

The time–frequency tradeoff

The one dial you cannot escape is window length, and it forces a genuine tradeoff rooted in the physics of waves:

$\Delta t \cdot \Delta f \gtrsim \text{constant}$

• A long window measures frequency precisely (fine $\Delta f$) but blurs when things happened (coarse $\Delta t$), good for a sustained musical note, bad for fast consonants.
• A short window pins down timing (fine $\Delta t$) but smears frequency (coarse $\Delta f$), good for percussive onsets, bad for resolving close pitches.

You cannot make both arbitrarily precise at once, sharpening time necessarily blurs frequency and vice versa. Speech pipelines typically settle near a 25 ms window with a 10 ms hop as a well-tested compromise for the timescale of phonemes.

The mel scale: matching human hearing

Humans do not hear pitch linearly. The gap between 200 Hz and 400 Hz sounds like a large jump, while 5,000 Hz to 5,200 Hz, the same 200 Hz, is barely noticeable. Our perception of pitch is roughly logarithmic. The mel scale warps frequency to match, spacing perceived-equal pitches equally:

$m = 2595 \, \log_{10}\!\left(1 + \frac{f}{700}\right)$

A mel spectrogram applies a bank of triangular filters spaced on this scale, collapsing the hundreds of linear frequency bins into ~80 perceptually meaningful ones. This throws away detail the ear ignores and keeps what it cares about, which is why the mel spectrogram, not the raw STFT, is the standard input to speech-recognition and text-to-speech models. Taking a further step (a discrete cosine transform of the log-mel energies) yields the classic MFCC features that dominated speech systems before deep learning.

Further reading

• librosa: Feature extraction, practical STFT, mel-spectrogram, and MFCC computation in Python.
• The Short-Time Fourier Transform, definition, windowing, and the time–frequency resolution tradeoff.
• Mel scale, the perceptual frequency scale and its formula.
• Speech and Language Processing (Jurafsky & Martin), Ch. on ASR features, spectrograms, mel filterbanks, and MFCCs in the context of speech systems.