A pretrained language model predicts plausible next tokens, but "plausible" is not the same as "helpful, honest, and safe." Reinforcement Learning from Human Feedback (RLHF) is the technique that closed much of that gap and turned raw language models into usable assistants. It reframes alignment as an RL problem: the policy is the language model, an action is generating a response, and the reward comes from a model trained to predict human preferences. This lesson walks the three-stage RLHF pipeline, explains the KL penalty that keeps it from going off the rails, examines how reward models get gamed, and introduces RLAIF, which swaps human labels for AI-generated ones.
Why not just fine-tune on good answers?
Supervised fine-tuning (SFT) on curated examples helps, but it has a ceiling. Humans find it far easier to compare two responses ("A is better than B") than to write the ideal response from scratch, and preference signals capture nuances, tone, safety, helpfulness, that are hard to demonstrate exhaustively. RLHF exploits this: it learns a reward function from cheap comparisons, then optimizes the model against it. The result is the classic three-stage pipeline.
1Stage 1, Supervised fine-tuning (SFT)Fine-tune the pretrained base model on high-quality human-written demonstrations of the desired behavior. This produces a competent starting policy and, crucially, a reference model to anchor later stages.
2
Ask the tutor
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Stage 2, Reward model (RM) training
Collect human comparisons: for a prompt, show two or more model responses and have annotators rank them. Train a reward model to output a scalar score that agrees with these preferences.
3Stage 3, RL optimization (PPO)Treat the language model as an RL policy. Sample responses, score them with the reward model, and use PPO to update the policy so it produces higher-reward responses, while a KL penalty keeps it close to the SFT reference.
Stage 2 in detail: learning a reward from preferences
Annotators rarely assign absolute scores; they pick a winner between two candidate responses yw (preferred) and yl (dispreferred) for the same prompt x. The reward model rϕ is trained so the preferred response scores higher, using the Bradley–Terry preference model, which turns a score difference into a probability of preference:
L(ϕ)=−E(x,yw,yl)[logσ(rϕ(x,yw)−rϕ(x,yl,
where σ is the logistic sigmoid. Intuitively, the loss is small when the preferred response already scores well above the dispreferred one, and large when they are close or reversed. The trained rϕ then acts as an automatic, scalable stand-in for a human rater during Stage 3.
The reward model is only a proxy
The reward model does not know "the truth", it knows how to imitate the
average annotator's preferences on the distribution of responses it was
trained on. Everything downstream inherits that limitation, which is the root
of the overoptimization problem discussed below.
Stage 3 in detail: PPO with a KL leash
In Stage 3 the policy πθ (the language model) generates a response, the reward model scores it, and PPO (from the previous lesson) pushes the policy toward higher scores. But there is a trap: if you optimize the reward model's score alone, the policy will chase whatever quirks maximize that score, drifting far from fluent, sensible language. The fix is a per-token KL-divergence penalty against the frozen SFT reference model πref:
R(x,y)=rϕ(x,y)−βKL(πθ(⋅∣x)∥πref(⋅∣x)).
The coefficient β controls how tightly the policy is tethered to its starting point. Two forces are now in tension:
Reward term r_phi(x, y)
Pulls the policy toward responses humans prefer.
Left unchecked, it exploits reward-model blind spots.
Larger updates chase higher scores.
This is the signal we actually want to follow, cautiously.
KL penalty -beta * KL(pi || pi_ref)
Pulls the policy back toward the fluent SFT model.
Prevents mode collapse and gibberish that games the RM.
Preserves general capabilities learned in pretraining/SFT.
Acts as a trust region on the language distribution itself.
Intuition
For a curious beginner
The KL penalty is a tether to base camp. The model is encouraged to explore toward higher-reward answers, but every step away from the sensible reference model costs something, so it wanders only as far as the reward genuinely justifies, instead of sprinting off a cliff.
Engineering
How it is actually used
In practice the KL term is folded into the per-token reward during PPO rollouts, and beta is often adjusted adaptively to hold the measured KL near a target. Too small a beta and the model degenerates into reward-hacking gibberish; too large and it barely moves from the SFT model, wasting the RL stage. Monitoring the running KL is one of the most important RLHF training diagnostics.
Mathematical
The underlying mechanism
KL(πθ∥πref)=Ey∼πθ[logπref(y∣x) measures how much the new policy's output distribution has diverged from the reference. Subtracting β times this from the reward is equivalent to a KL-regularized RL objective whose optimum is a Boltzmann-weighted tilt of the reference distribution toward high reward, the model closest to the reference that still improves the reward.
For a curious beginner
The KL penalty is a tether to base camp. The model is encouraged to explore toward higher-reward answers, but every step away from the sensible reference model costs something, so it wanders only as far as the reward genuinely justifies, instead of sprinting off a cliff.
How it is actually used
In practice the KL term is folded into the per-token reward during PPO rollouts, and beta is often adjusted adaptively to hold the measured KL near a target. Too small a beta and the model degenerates into reward-hacking gibberish; too large and it barely moves from the SFT model, wasting the RL stage. Monitoring the running KL is one of the most important RLHF training diagnostics.
The underlying mechanism
KL(πθ∥πref)=Ey∼πθ[logπref(y∣x) measures how much the new policy's output distribution has diverged from the reference. Subtracting β times this from the reward is equivalent to a KL-regularized RL objective whose optimum is a Boltzmann-weighted tilt of the reference distribution toward high reward, the model closest to the reference that still improves the reward.
Reward-model overoptimization (Goodhart in action)
Because the reward model is a proxy, optimizing it too hard eventually decreases true quality even as the measured reward keeps climbing. This is Goodhart's law: once the reward score becomes the target, it stops being a faithful measure of quality. The policy discovers adversarial inputs the reward model scores highly but humans dislike, verbose padding, sycophantic agreement, confident-sounding but empty phrasing, or formatting tricks the RM happened to correlate with good answers.
Empirically, plotting true quality against the KL distance from the reference shows a rise-then-fall curve: quality improves for a while, peaks, and then degrades as the policy exploits the reward model's errors. Practitioners fight this with the KL penalty, early stopping based on held-out human evals, periodically refreshing the reward model with new comparisons on the policy's current outputs, and ensembling reward models to reduce exploitable blind spots.
Why the reference model does double duty
The SFT model appears twice, as the initialization for the RL policy and as
the fixed anchor in the KL penalty. This is deliberate: starting near a
capable model and staying near it is what lets RLHF sharpen behavior without
destroying the broad competence acquired during pretraining.
RLAIF: replacing the human labeler
Human preference data is slow and expensive to collect. RLAIF (RL from AI Feedback) replaces the human annotator in Stage 2 with an LLM that judges responses according to a written set of principles (a "constitution"). The pipeline is otherwise the same: AI-generated preferences train a reward model (or directly score responses), which then drives PPO.
RLHF (human feedback)
Preferences come from human annotators.
High-quality, aligned with real human values.
Expensive, slow, and hard to scale to millions of comparisons.
Subject to annotator inconsistency, fatigue, and demographic bias.
RLAIF (AI feedback)
Preferences come from an LLM judging against explicit principles.
Cheap, fast, and scalable; principles are transparent and editable.
Quality is capped by the judge model's own biases and errors.
Risks amplifying the judge's blind spots and can compound model errors.
RLAIF is not strictly better or worse, it trades human cost and inconsistency for scalability and the judge model's limitations. In practice the two are often combined: humans supply feedback where judgment is subtle or safety-critical, while AI feedback covers high-volume, well-specified criteria. Direct-optimization methods like DPO also sidestep the separate reward-model-plus-PPO machinery by optimizing the policy directly on preference pairs, but the underlying signal, human or AI comparisons, is the same idea explored here.
Common mistakes
Treating the reward model as ground truth. It is a learned proxy for
average annotator preference; optimizing it without limit degrades true
quality (Goodhart). - Dropping or mis-tuning the KL penalty. With no KL
term the policy collapses into reward-hacking gibberish; with too large a
β it never improves over SFT. - Reusing a stale reward model. As the
policy shifts, it drifts off the RM's training distribution; the RM must be
refreshed on new on-policy comparisons. - Skipping SFT. RL from a raw base
model is unstable and sample-hungry; SFT provides both a competent start and
the reference for the KL anchor. - Assuming RLAIF removes bias. It
relocates bias from human annotators to the judge model, and can amplify the
judge's systematic errors at scale.
Further reading
Ouyang et al., "Training language models to follow instructions with human feedback" (InstructGPT, 2022), the canonical three-stage RLHF pipeline: https://arxiv.org/abs/2203.02155
Christiano et al., "Deep Reinforcement Learning from Human Preferences" (2017), the foundational preference-based reward-learning paper: https://arxiv.org/abs/1706.03741
Bai et al., "Constitutional AI: Harmlessness from AI Feedback" (2022), the reference work on RLAIF: https://arxiv.org/abs/2212.08073
Gao et al., "Scaling Laws for Reward Model Overoptimization" (2022), quantifies the Goodhart / overoptimization curve: https://arxiv.org/abs/2210.10760
Schulman et al., "Proximal Policy Optimization Algorithms" (2017), the RL optimizer used in Stage 3: https://arxiv.org/abs/1707.06347