Lab

Sampling & Temperature

Explore temperature, top-k and top-p on token probabilities.

A language model outputs a logit (a raw score) for every token it might say next. To choose one it turns those scores into probabilities with softmax, then narrows the field with temperature, top-k and top-p. Turn the knobs and watch the distribution change.

sunny
37.2%
cloudy
22.6%
warm
15.1%
cold
10.1%
rainy
7.5%
nice
4.6%
quite
2.5%
banana
0.3%

Softmax converts logits into probabilities that sum to 1:

$P(x_i) = \dfrac{e^{z_i / T}}{\sum_j e^{z_j / T}}$

Temperature (T) divides the logits. Low T (toward 0) sharpens toward the single most likely token (greedy); high T flattens the distribution, making rarer tokens competitive. Top-k keeps only the k highest tokens; top-p keeps the smallest set whose probabilities add up to p. Both then renormalize what survives.

Note: the tokens and logits here are illustrative, chosen to make the effect visible, they are not produced by a real model.

Challenge

Set temperature to 0.1, the model almost always says “sunny”. Now raise it to 1.8: which unlikely tokens become reachable? Then keep the high temperature but drop top-p to 0.5, how does nucleus sampling rein the randomness back in?

From the lesson: Sampling & Temperature