SFT & Instruction Tuning

The crucial twist is that the sum runs only over the response tokens. The prompt is fed in as context $x$ but is excluded from the loss, so the model is graded solely on producing the target completion.

From base model to instruction follower

Instruction tuning is simply SFT applied at scale across many diverse tasks (summarize, translate, classify, extract, reason), each phrased as an (instruction, response) pair. Training on this variety teaches a general behavior: read an instruction, then satisfy it. This is what separates a base model from an instruct (or chat) model.

The training example and its mask

Data is typically stored as JSONL, one conversation per line, using role-tagged messages:

{
  "messages": [
    { "role": "system", "content": "You are a concise assistant." },
    { "role": "user", "content": "Convert 3 miles to kilometers." },
    { "role": "assistant", "content": "3 miles is about 4.83 km." }
  ]
}

A tokenizer applies the model's chat template, flattening this into one token sequence with special role markers. During training each token gets a label; prompt tokens are set to an ignore index (-100 in most frameworks) so they are skipped by the loss:

Only the assistant span contributes to $\mathcal{L}$. The model still reads the prompt (it conditions on it) but is never rewarded for reproducing it.

Running an SFT job

1. 1Curate demonstrationsAssemble high-quality (instruction, response) pairs; quality and diversity matter far more than raw count.
2. 2Apply the chat templateRender every example into the exact role format the model was released with; mismatches degrade results.
3. 3Mask the promptAssign the ignore label to system/user/template tokens so the loss covers only assistant tokens.
4. 4Train 1-3 epochsUse a small learning rate (often 1e-5 to 2e-5 for full FT, higher for LoRA) and watch the training loss.
5. 5Evaluate held-outTrack loss and inspect generations on unseen prompts to catch overfitting before it hurts.

Keep epochs low, usually 1-3: demonstration sets are tiny relative to pretraining data, and extra passes quickly memorize phrasing, producing repetition and brittle answers. SFT is only the first alignment stage; preference optimization (DPO) or RLHF usually follows to refine which good response the model prefers.

Further reading

• Ouyang et al., "Training language models to follow instructions with human feedback" (InstructGPT)
• Wei et al., "Finetuned Language Models Are Zero-Shot Learners" (FLAN)
• Hugging Face TRL, SFTTrainer documentation