Loading lesson…
Generative AI / Modalities & Control
Latent diffusion, and the extra axis of time.
Reviewed by Yuvaraj
Modern text-to-image and text-to-video systems feel like magic, but under the hood they are a disciplined pipeline: a language model reads your prompt, a diffusion model sculpts random noise into structure while continuously consulting that prompt, and a decoder turns the result into pixels. Understanding each stage is what separates prompt-guessing from deliberate control over composition, fidelity, and reproducibility.
A text-to-image model is three cooperating parts, not one monolith.
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Running the diffusion in a compressed latent space, rather than directly on pixels, is the key efficiency trick behind latent diffusion models: the VAE encoder shrinks images roughly eightfold per side, so the expensive denoising loop operates on far fewer values.
For a curious beginner
The prompt is like a running commentary the painter keeps glancing at. As each region of the canvas takes shape, it checks which words are relevant to it, so "red" pulls color into the bicycle and "blue" into the wall.
How it is actually used
Cross-attention layers derive queries from the evolving image latent and keys and values from the text embeddings. The attention mixes prompt information into each spatial location, and this repeats at every denoising step and every attention block, so conditioning is applied continuously rather than once.
The underlying mechanism
With conditioning , the network predicts noise inside attention as . Classifier-free guidance then sharpens adherence to the prompt: where is the guidance scale and is the unconditional (or negative-prompt) branch.
Take the prompt "a red bicycle leaning on a blue wall, golden hour". The encoder produces embeddings for each token. The latent starts as pure noise. In early denoising steps the coarse layout settles: one large object left of center against a flat background. Cross-attention binds "red" to the bicycle-shaped region and "blue" to the wall, while "golden hour" biases global lighting toward warm, low-angle tones. The VAE then decodes the cleaned latent into the final image.
Now change one knob at a time:
| Lever | Controls | Practical effect |
|---|---|---|
| Guidance scale | Prompt adherence strength | Higher is more literal but risks artifacts |
| Seed | Initial noise | Fix it for reproducibility; vary it for options |
| Steps | Denoising iterations | More steps refine detail with diminishing returns |
| Negative prompt | What to avoid | Suppresses unwanted content and defects |
| Resolution / aspect | Output shape | Larger costs more and can alter composition |
Video is not a stack of independent images. The hard extra problem is temporal consistency: frames must agree over time so objects move smoothly instead of flickering or drifting.
Knowledge stability
Fundamentals here (diffusion, cross-attention conditioning, VAE decoding, temporal modeling) are stable. Specific commercial models and their exact limits on clip length, resolution, and quality are FAST-MOVING and EVOLVING; verify current capabilities against vendor docs.
Common mistakes