Autoregressive vs Diffusion

Each factor $p(x_t \mid x_{<t})$ is a distribution over the next element given the prefix $x_{<t}$. Generation is therefore inherently sequential, you cannot sample element $t$ until element $t-1$ exists. This maps naturally onto discrete sequences such as text and code, where GPT-style transformers dominate.

A diffusion model instead defines a forward process that gradually corrupts data into Gaussian noise, then learns to reverse it. During training the network sees a noised sample $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon$ and predicts the noise $\epsilon$. At generation time you start from $x_T \sim \mathcal{N}(0, I)$ and apply the learned denoiser repeatedly, each pass removing a little noise. Every pixel is updated in parallel, yet you still need many sequential denoising steps. This fits continuous data, images, audio, video, extremely well.

The core difference, explained three ways

Tracing a sentence, autoregressively

1. 1PromptThe model receives a start context (a begin-of-sequence marker or prompt).
2. 2Emit 'The'It samples the first token from the distribution over the vocabulary. Prefix so far: (empty).
3. 3Emit 'cat'The token 'The' is fed back in; the next distribution is conditioned on 'The'.
4. 4Emit 'sat'Now conditioned on 'The cat'. The prefix, and the KV cache, has grown by one.
5. 5Emit 'down'Conditioned on 'The cat sat'. Each new token depends strictly on all prior tokens.

Tracing an image, by diffusion

1. 1Step T: pure noiseBegin with an image of random Gaussian static, no structure at all.
2. 2Step T-1: coarse blobsThe denoiser predicts and subtracts noise; vague color regions appear across the whole frame at once.
3. 3Middle steps: shapesRepeated denoising sharpens edges and layout until a subject becomes recognizable.
4. 4Step 0: final imageAfter the last step the sample is clean. No pixel was 'finished' before the others, they refined together.

Autoregressive versus diffusion

AR and diffusion are not the only options. GANs generate in a single forward pass by pitting a generator against a discriminator, and VAEs learn a compressed latent distribution to sample from; both predate today's diffusion dominance for images. Hybrids are common, latent diffusion runs the denoising process inside a VAE's latent space, and some systems generate discrete tokens autoregressively, then decode them with diffusion.

Further reading

• Ho, Jain, Abbeel, Denoising Diffusion Probabilistic Models: https://arxiv.org/abs/2006.11239
• Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models: https://arxiv.org/abs/2112.10752
• Vaswani et al., Attention Is All You Need: https://arxiv.org/abs/1706.03762