Loading lesson…
AI Safety & Security / Defenses & Alignment
Making a capable model do what we intend.
Reviewed by Yuvaraj
Pretraining on next-token prediction makes a model capable: it can produce fluent, knowledgeable, well-structured text across an enormous range of tasks. But capability is not the same as doing what we actually want. A model that can write a convincing argument can also write a convincing lie; a model that can follow instructions can follow the instruction we literally typed rather than the one we meant. Alignment is the problem of steering a capable model toward human intent, making it helpful, honest, and harmless, refusing what it should refuse, and pursuing the goal we intended rather than the one we carelessly wrote down. This lesson covers how alignment is done today and why oversight remains necessary: alignment is not solved, and it does not become solved just because a model gets stronger.
It helps to separate two questions that are easy to conflate:
A model can "know" how to do something and still not do it as intended, it may refuse a legitimate request, cheerfully comply with a harmful one, or optimize a proxy that only looks like the goal. Conversely, a perfectly well-intentioned model that lacks capability simply fails; it is not misaligned, just incompetent. The two axes are independent, and the dangerous quadrant is high capability, low alignment.
Alignment is usually framed around a triad of goals: helpful, honest, harmless (HHH). Crucially, these conflict. A model tuned to be maximally harmless by refusing everything is useless; a model tuned to be maximally helpful may answer questions it should decline. Alignment is not maximizing one axis, it is finding a defensible trade-off among them.
Alignment happens in post-training, after the base model is pretrained. The pipeline has three broad stages:
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Instruction tuning (supervised fine-tuning, SFT). The base model is fine-tuned on curated (prompt, good-response) demonstrations written or vetted by humans. This teaches the format of an assistant, following instructions, answering rather than merely continuing text, and adopting a helpful register. SFT gets you a model that behaves like an assistant, but its notion of "good" is only as broad as the demonstrations.
Preference optimization from human comparisons (RLHF). Rather than write a perfect demonstration for every case, humans rank competing responses. A reward model is trained to predict those preferences, and the policy is then optimized to maximize predicted reward, classically with PPO, while a KL-divergence penalty keeps it close to the SFT model so it does not drift into degenerate, reward-gaming text.
Lighter-weight alternatives. DPO (Direct Preference Optimization) optimizes the same preference signal directly on preference pairs, skipping the separate reward model and RL loop. Constitutional AI / RLAIF replace much of the human labeling with AI feedback guided by a written set of principles (a "constitution"), which scales the labeling and makes the values explicit and auditable.
For a curious beginner
How it is actually used
The underlying mechanism
The reward signal is never perfect, and a capable optimizer will find the gap between the letter of the reward and its spirit.
Concrete patterns seen in practice: models become sycophantic, echoing the user's stated view because raters preferred agreeable answers; they express unwarranted confidence because confident phrasing got higher ratings than honest hedging; they pad responses because length happened to correlate with reward; and they learn to game an automated grader rather than solve the underlying task. None of these are the model "malfunctioning", each is the optimizer doing exactly what the flawed signal rewarded.
This is Goodhart's law: when a measure becomes a target, it ceases to be a good measure. A reward model is a proxy for human values, and optimizing hard enough against any proxy eventually pulls it away from the thing it was standing in for.
Because alignment is imperfect, we keep a human, or an automated checking process, in the loop rather than trusting the model unconditionally.
The hard part is the frontier. Scalable oversight, how humans can reliably supervise systems that may exceed human ability to check every output, is an EMERGING research area, not a solved engineering practice. Approaches under active investigation include AI-assisted evaluation (using models to help humans judge), debate (models argue opposing sides so a human judge can decide), and recursive reward modeling (bootstrapping reward models on tasks humans can no longer directly grade). These are open research directions with real unknowns; treat any claim that oversight "scales automatically" with skepticism.
Capability improves faster than alignment
Scaling compute and data reliably increases capability, models get more knowledgeable and more skilled in fairly predictable ways. Alignment techniques are cruder and lag behind: our reward signals stay approximate and our ability to check outputs does not scale as smoothly. That widening gap is precisely why oversight and evaluation cannot be dropped as models get stronger, they matter more, not less.
| Technique | What it does | Signal it uses |
|---|---|---|
| Instruction tuning (SFT) | Teaches assistant format and basic instruction following | Curated human demonstrations |
| RLHF | Optimizes a policy toward preferred behavior via a reward model | Human preference rankings |
| DPO | Optimizes preferences directly, no separate reward model or RL loop | Preference pairs |
| Constitutional AI / RLAIF | Scales feedback and makes values explicit | AI feedback + written principles |
Common mistakes