Loading lesson…
LLM Foundations / How LLMs Work
Controlling the output.
Reviewed by Yuvaraj
At each step, a language model produces a probability for every possible next token. Turning that distribution into an actual choice is called sampling, and a few well-known knobs control it. They are the difference between output that is robotic, output that is creative, and output that is nonsense.
The model's raw scores are logits. A softmax converts them into probabilities that sum to 1. Temperature () rescales the logits before the softmax:
Answer from memory before revealing, retrieval practice is what builds durable recall.
With temperature 0 (greedy decoding), the model:
Raising the sampling temperature generally makes the output:
Top-p (nucleus) sampling keeps the smallest set of tokens whose probabilities sum to at least p, then samples from that set.
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
T < 1) sharpens the distribution, the top tokens get even more likely. Output is focused and repetitive.T > 1) flattens the distribution, unlikely tokens get a real chance. Output is diverse and, pushed too far, incoherent.T → 0 approaches always taking the single most likely token.Take three candidate tokens with logits A = 2, B = 1, C = 0, and watch temperature reshape them:
| Temperature | A | B | C | Effect |
|---|---|---|---|---|
T = 0.5 | 0.87 | 0.12 | 0.02 | Sharper, safer |
T = 1.0 | 0.67 | 0.24 | 0.09 | The raw softmax |
T = 2.0 | 0.51 | 0.31 | 0.19 | Flatter, riskier |
Nothing about the model changed, same logits every time. Temperature only redistributes how much probability the long-shot tokens receive. At T = 0.5, C is almost impossible; at T = 2.0, it has a real 19% shot.
| Method | How it chooses | Character |
|---|---|---|
| Greedy | Always the single highest-probability token | Deterministic, safe, can be dull or looping |
| Top-k | Sample only from the k most likely tokens | Caps how wild it can get |
| Top-p (nucleus) | Sample from the smallest set of tokens whose probability adds up to p | Adapts: few choices when the model is confident, more when it is not |
Top-k and top-p are often combined with temperature: first trim the candidates, then sample among what remains.
No single 'best' setting
There is no universally correct temperature. Factual extraction or code often wants low, focused settings; brainstorming or fiction wants higher, more diverse ones. Sampling is a dial you match to the task.
Low temperature is not 'more truthful'
Turning the temperature down makes output more predictable, not more correct. The model is still predicting likely tokens, not verifying facts. A confident, low-temperature answer can be just as wrong, it will simply be wrong the same way every time.
Greedy decoding is deterministic in principle: same input, same logits, same choice. Sampling introduces randomness, usually controlled by a seed so a run can be repeated. (In practice, hardware and batching details can still cause tiny variations, worth knowing before you rely on exact reproducibility.)