How Diffusion Works

Here $\bar{\alpha}_t$ comes from the noise schedule, a predetermined sequence that controls how fast signal is replaced by noise. Early steps barely perturb the image; late steps leave almost pure static. Because the schedule is known and fixed, we always know exactly how much noise was added at each step, which gives us a perfect training target.

The reverse process: a network that predicts noise

Generation runs that process backward. A neural network, usually a U-Net, or in newer systems a diffusion transformer, looks at a noisy sample $x_t$ and the step index $t$, and predicts the noise $\epsilon$ that was mixed in. We subtract a scaled portion of that prediction to recover a slightly cleaner sample, then repeat. Starting from pure noise and iterating down turns random static into a coherent image.

1. 1Start from noiseSample a latent of pure Gaussian static, this is the starting canvas.
2. 2Predict the noiseThe network estimates which part of the current latent is noise.
3. 3Take one denoising stepSubtract a scaled fraction of that predicted noise to get a slightly cleaner latent.
4. 4RepeatLoop for the chosen number of sampler steps, moving from high noise toward low noise.
5. 5DecodeIn latent diffusion, a VAE decoder turns the final latent into a full-resolution image.

The training objective

Training is remarkably simple. Take a clean image, pick a random step $t$, sample noise $\epsilon$, build $x_t$ with the forward formula, and ask the network to predict that noise. The loss is plain mean-squared error between the true and predicted noise:

$\mathcal{L}=\mathbb{E}_{x_0,\epsilon,t}\big[\lVert \epsilon - \epsilon_\theta(x_t,t)\rVert^2\big]$

There is no adversarial game and no discriminator, just regression against a known target, which is why diffusion training is far more stable than GAN training.

Classifier-free guidance

To follow a text prompt, the network is trained to predict noise both with a caption and without one (the caption randomly dropped). At sampling time we combine the two predictions and extrapolate away from the unconditional one:

$\hat{\epsilon} = \epsilon_\theta(x_t, t, \varnothing) + s\big(\epsilon_\theta(x_t, t, c) - \epsilon_\theta(x_t, t, \varnothing)\big)$

The guidance scale $s$ controls prompt adherence. At $s=1$ you get the plain conditional prediction; larger $s$ pushes the sample harder toward the text.

One denoising step, by the numbers

Take a 2D latent for illustration. Suppose the current noisy latent is $x_t = [1.60, -0.80]$ and the network predicts noise $\epsilon_\theta = [0.50, -0.30]$. With an illustrative step coefficient $c = 0.2$ (the real DDPM update also rescales by schedule terms, but the essence is: subtract predicted noise):

$x_{t-1} = x_t - c\,\epsilon_\theta = [\,1.60 - 0.10,\; -0.80 + 0.06\,] = [1.50, -0.74]$

The latent moved a little closer to a clean sample. Now add text guidance. Say the unconditional prediction is $[0.50, -0.30]$ and the conditional, prompt-aware prediction is $[0.20, -0.60]$. With $s = 3$:

$\hat{\epsilon} = [0.50, -0.30] + 3\big([0.20, -0.60] - [0.50, -0.30]\big) = [-0.40, -1.20]$

Subtracting this guided estimate instead of the raw one pulls the latent much more decisively in the direction the prompt implies, which is exactly why higher guidance yields more prompt-faithful but potentially over-saturated images.

Latent diffusion and samplers

Running diffusion directly on megapixel images is expensive. Latent diffusion first uses a VAE encoder to compress the image into a small latent grid, runs the entire noising and denoising process there, then decodes back to pixels. This compression is why Stable Diffusion is fast enough for consumer GPUs. The sampler (DDPM, DDIM, DPM-Solver, and others) decides how many steps to take: fewer steps are faster but coarser, more steps refine detail with diminishing returns.

Why removing noise creates new images

Further reading

• Ho, Jain, and Abbeel, Denoising Diffusion Probabilistic Models (DDPM), https://arxiv.org/abs/2006.11239
• Ho and Salimans, Classifier-Free Diffusion Guidance, https://arxiv.org/abs/2207.12598
• Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, https://arxiv.org/abs/2112.10752
• Song et al., Score-Based Generative Modeling through Stochastic Differential Equations, https://arxiv.org/abs/2011.13456