Preference Optimization & DPO

Path 1, RLHF with a reward model and PPO

The original recipe (InstructGPT) turns preferences into a learned reward, then optimizes against it.

1. 1Collect comparisonsHumans rank responses to the same prompt, producing chosen/rejected pairs.
2. 2Train a reward modelA model learns to score responses so that chosen outscores rejected (Bradley-Terry objective).
3. 3Optimize the policy with PPOReinforcement learning nudges the policy to generate high-reward responses.
4. 4Constrain with a KL penaltyA penalty keeps the policy near the frozen SFT reference so it does not drift or hack the reward.

The KL term is load-bearing: without it, PPO discovers degenerate text that scores high on the reward model but reads as gibberish, classic reward hacking.

Path 2, DPO skips the reward model

Direct Preference Optimization removes the separate reward model and the RL loop entirely. It optimizes the policy directly on preference pairs with one loss:

$\mathcal{L}_{\text{DPO}} = -\log \sigma\!\big(\beta \log \tfrac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} - \beta \log \tfrac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)}\big)$

Here $\pi_\theta$ is the model being trained, $\pi_{\text{ref}}$ is a frozen copy of the SFT model, and the temperature $\beta$ controls how far the policy may deviate from that reference. Minimizing this loss increases the log-probability margin of chosen over rejected, measured relative to what the reference already assigns.

A worked preference triple

Take one training example:

During a DPO step the model computes log-probabilities for both responses under $\pi_\theta$ and under the frozen $\pi_{\text{ref}}$. The gradient raises $\pi_\theta(y_w\mid x)$ and lowers $\pi_\theta(y_l\mid x)$, but always relative to the reference ratios. That reference anchor is what prevents collapse: the model cannot just crank the chosen answer's probability toward 1 and everything else toward gibberish, because both terms are divided by $\pi_{\text{ref}}$, so moving far from the reference distribution is implicitly penalized by $\beta$. The result is a sharpened preference, not a broken language model.

RLHF vs DPO

Further reading

• Rafailov et al., Direct Preference Optimization (2023), the DPO paper and its derivation.
• Ouyang et al., Training language models to follow instructions with human feedback (InstructGPT, 2022), the reference RLHF pipeline.
• Hugging Face TRL, DPOTrainer documentation, practical implementation and hyperparameters.