Loading lesson…
Fine-Tuning / Efficient Fine-Tuning
Why fine-tuning succeeds or fails on the data.
Reviewed by Yuvaraj
When a fine-tune disappoints, the instinct is to reach for the learning rate, the number of epochs, or the LoRA rank. Almost always, that is the wrong knob. Fine-tuning teaches a model to imitate the examples you give it, so the ceiling on quality is set by the dataset, its coverage, its consistency, and its honesty, not by the optimizer. This lesson is about building a dataset that deserves a good model, and recognizing the failure modes that quietly ruin one.
Supervised fine-tuning minimizes next-token cross-entropy on your target completions. The model is literally rewarded for reproducing your outputs, pattern for pattern. That has two consequences. First, whatever formatting, tone, and label vocabulary appear in your targets become the behavior you get back: inconsistency in equals inconsistency out. Second, when two near-identical inputs carry conflicting targets, their gradients pull against each other. The model cannot satisfy both, so it hedges toward an average and grows less confident exactly where you wanted precision.
Hyperparameters only control how fast and how hard you fit the data you already have. A well-tuned learning rate on contradictory data just fits the contradictions faster. No optimizer setting can manufacture signal that is not in the examples.
For a curious beginner
A handful of clear, consistent examples from a good teacher beats a mountain of contradictory notes. The model copies patterns: a consistent pattern teaches it a rule, while conflicting ones teach it to guess. More pages do not help if half of them disagree.
How it is actually used
Every duplicate, mislabeled, or off-distribution row is gradient signal pointing the wrong way. Five hundred consistent examples give a sharp, low-variance target; fifty thousand scraped rows with 15% label noise cap achievable accuracy near the noise ceiling and add format drift. Curate, dedupe, and validate formats before you ever touch the learning rate.
The underlying mechanism
SFT minimizes expected cross-entropy under your data distribution. Label noise raises the irreducible Bayes error, so the loss floor sits at the noise rate and no optimizer setting beats it. Contradictory targets for one input push the conditional toward the average of the conflicting labels, maximizing entropy right where you wanted a confident answer. Clean data lowers the floor and sharpens the conditional.
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Consider an intent classifier. These three rows describe the same kind of billing problem but disagree on how to answer:
{"messages":[{"role":"user","content":"My card was declined at checkout"},{"role":"assistant","content":"Category: Billing"}]}
{"messages":[{"role":"user","content":"Payment failed when I tried to pay"},{"role":"assistant","content":"This looks like a billing issue, let me help!"}]}
{"messages":[{"role":"user","content":"I was charged twice for one order"},{"role":"assistant","content":"payments"}]}
The targets use three different formats and two different label words for one concept. Trained on this, the model cannot infer a rule: it learns that billing inputs sometimes yield Category: Billing, sometimes payments, sometimes a whole sentence, and so it emits an unpredictable mix. Normalizing to a single label vocabulary removes the contradiction:
{"messages":[{"role":"user","content":"My card was declined at checkout"},{"role":"assistant","content":"billing"}]}
{"messages":[{"role":"user","content":"Payment failed when I tried to pay"},{"role":"assistant","content":"billing"}]}
{"messages":[{"role":"user","content":"I was charged twice for one order"},{"role":"assistant","content":"billing"}]}
Now every billing input maps to one token sequence and the gradient points in one direction. Deduplicate exact and near-duplicate rows, resolve remaining contradictions, and correct mislabeled targets before training.
Training loss always falls; it is not evidence that the model learned your task. Hold out a validation set drawn from the production distribution before you train, and never let it leak into training. Evaluate with task-relevant metrics, exact-match or F1 for classification, a format-validity rate for structured output, human or LLM-judge scores for open-ended generation, and watch validation loss to catch the moment fitting turns into memorizing.
| Failure mode | What you see | Root cause | Fix |
|---|---|---|---|
| Overfitting | Validation loss rises while training loss keeps falling; model repeats training examples verbatim | Too many epochs or too few examples | Fewer epochs, more varied data, early stop on validation loss |
| Catastrophic forgetting | Better at your task but worse at general reasoning, formatting, or refusals | Aggressive updates overwrite pretrained ability | Lower learning rate, LoRA or adapters, mix in general examples |
| Distribution shift | Strong offline scores, weak in production | Training data does not match real inputs | Sample from real traffic; cover the edge cases |
| Leakage / contamination | Eval looks great but does not hold up | Test rows or near-duplicates appear in training | Deduplicate across splits; freeze the validation set first |
Common mistakes