Audio & Multimodal

Text-to-speech, music generation, image captioning, and visual question answering look like unrelated problems. They share one move: convert the signal, a sound, an image, a sentence, into a sequence of discrete tokens (or a compact array of numbers), let a transformer predict that sequence, then decode it back into the target medium. Master that pattern and most of modern generative audio and multimodal AI stops looking mysterious.

Representing audio so a model can generate it

You almost never model a raw waveform directly. Audio at CD quality is 44,100 samples per second per channel; a few seconds is hundreds of thousands of values, far too long for a transformer to predict sample by sample. So the field uses two compressed representations.

• Spectrogram: transform the waveform into a mel-spectrogram, a time-by-frequency image showing how much energy sits at each frequency over time. A model generates this image-like array, and a separate vocoder (a neural vocoder such as HiFi-GAN) converts the spectrogram back into a waveform.
• Neural codec tokens: an autoencoder such as SoundStream or EnCodec compresses each short audio frame into a few integer codes using residual vector quantization (RVQ), a stack of codebooks where each one quantizes the residual left by the previous. A model generates these discrete tokens (autoregressively, like a language model, or by diffusion), and the codec's decoder reconstructs the waveform.

Text-to-speech conditions generation on encoded text plus a speaker embedding; music and sound-effect generation condition on a text prompt.

Two audio representations, compared