Controllability & Editing · AI Learning Universe

Generative AI / Modalities & Control

Controllability & Editing

Steering and refining generated output.

L3 · AdvancedFast-movingKnown~13 min

Reviewed by YuvarajPublished 25 Aug 2026Updated 27 Aug 2026

A raw diffusion sample is a roll of the dice: you give a prompt, the model starts from random noise, and you get something in the neighborhood of what you asked for. Controllability is the set of levers that move you from "in the neighborhood" to "exactly this," and editing reuses the very same machinery to change part of an image you already have instead of generating from scratch. Both come down to one idea, feeding the denoiser extra information that constrains what it is allowed to produce.

The core lever: conditioning

Recall that a diffusion model removes noise one step at a time by predicting the noise in the current image. That prediction is a function not just of the noisy image but of a condition, the extra signal you supply. Change the condition and you change every step of the denoising trajectory. Different modalities of condition give you different kinds of control:

Condition	Mechanism	What it controls
Text prompt	Cross-attention to text embeddings	Subject, style, mood
Init image (img2img)	Start from a partially-noised image	Overall composition and color
Mask (inpainting)	Freeze pixels outside the mask	Which region gets regenerated
Structure (ControlNet)	Extra network reads edges/depth/pose	Layout, geometry, pose

Classifier-free guidance

The single most important knob is classifier-free guidance (CFG). It amplifies how strongly the prompt pulls on the result.

Ask the tutor

Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.

Finished this lesson?

Mark it complete to earn XP, keep your streak, and schedule a review.

Intuition

For a curious beginner

The model makes two guesses at each step: one ignoring your prompt, one paying attention to it. Guidance is a dial that says "exaggerate the difference between those two guesses." Turn it up and the image clings harder to your words; turn it too far and it over-saturates and looks fried.

Engineering

How it is actually used

At every denoising step you run the model twice, once with the prompt, once with an empty prompt, then extrapolate away from the unconditioned prediction. The guidance scale is a single float, typically 5 to 9. Below about 3 the prompt is a loose suggestion; above about 15 you get contrast artifacts and mangled details. It is the first parameter to tune when output ignores your prompt.

Mathematical

The underlying mechanism

Let

\epsilon_\varnothing

be the noise the model predicts with no condition and

\epsilon_c

the prediction given condition

c

. The guided prediction is

\hat\epsilon = \epsilon_\varnothing + s\,(\epsilon_c - \epsilon_\varnothing)

, where

s

is the guidance scale. At

s=1

you recover ordinary conditional sampling;

s>1

extrapolates past the conditional estimate along the direction the prompt points.