Mathematics for AI / Calculus & Probability

Information & Entropy

Surprise, entropy, and the loss LLMs minimize.

L3 · AdvancedEvolvingKnown~14 min

Reviewed by YuvarajPublished 25 Aug 2026Updated 27 Aug 2026

Every time a model makes a prediction, it is implicitly answering one question: "How surprised should I be by what actually happened?" Information theory turns that intuition into arithmetic. It gives us a single unit, the bit, for measuring surprise, uncertainty, and the cost of a wrong belief, and from it falls out the exact loss function that trains almost every classifier you will build. This lesson connects four quantities developers use constantly, usually without naming them: self-information, entropy, cross-entropy, and KL divergence.

Self-information: the surprise of one outcome

The self-information (or "surprisal") of an outcome with probability $p(x)$ is

$I(x) = -\log p(x).$

Rare events carry more information; a certain event carries none. Using measures surprise in , and the shape of explains the unit: halving a probability adds exactly one bit of surprise.

Ask the tutor

Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.

Finished this lesson?

Mark it complete to earn XP, keep your streak, and schedule a review.

\log_2

lo g_{2}

bits

-\log p

Probability $p$	Surprise $-\log_2 p$ (bits)
1.00	0.00
0.50	1.00
0.25	2.00
0.10	3.32
0.01	6.64

Entropy: expected surprise

Entropy is the average self-information of a random variable, how uncertain you are before you see the outcome:

$H(X) = -\sum_i p_i \log p_i.$

It is always non-negative and is maximal when every outcome is equally likely.

Fair coin with $p = 0.5$ per side:

$H = -(0.5\log_2 0.5 + 0.5\log_2 0.5) = -(0.5\cdot(-1) + 0.5\cdot(-1)) = 1 \text{ bit.}$

One yes/no question fully resolves the outcome.

Biased coin with $p(\text{heads}) = 0.9$ :

$H = -(0.9\log_2 0.9 + 0.1\log_2 0.1) \approx -(0.9\cdot(-0.152) + 0.1\cdot(-3.322)) \approx 0.469 \text{ bits.}$

The bias makes the coin more predictable, so it carries less than half the uncertainty of a fair one.

Cross-entropy: scoring a model against reality

Cross-entropy measures the average surprise you experience when the world follows distribution $p$ but you believe $q$ :

$H(p, q) = -\sum_i p_i \log q_i.$

This is exactly the classification loss. For a single labelled example the true distribution $p$ is one-hot, all mass on the correct class, so the sum collapses to $-\log q_{\text{true}}$ : the surprise your model assigned to the right answer. Minimizing it drives $q_{\text{true}} \to 1$ .

1Model emits logitsOne raw score per class, unnormalized.
2Softmax to a distribution qLogits become probabilities that sum to 1.
3Compare against one-hot pThe true label puts all probability on the correct class.
4Cross-entropy lossTake the negative log of the probability given to the true class.

Worked loss. Three classes, true label is class 1, model predicts $q = [0.1, 0.7, 0.2]$ :

$H(p,q) = -\ln(0.7) \approx 0.357 \text{ nats.}$

A confident, correct prediction $q = [0.02, 0.95, 0.03]$ gives $-\ln(0.95) \approx 0.051$ , far lower. A confident but wrong prediction is punished savagely, because $-\log q \to \infty$ as $q \to 0$ .

KL divergence: the avoidable part

KL divergence is the extra surprise you pay for using $q$ instead of the true $p$ :

$D_{KL}(p \parallel q) = \sum_i p_i \log \frac{p_i}{q_i}.$

The three quantities lock together in one identity:

$H(p, q) = H(p) + D_{KL}(p \parallel q).$

Entropy $H(p)$ is the irreducible uncertainty of the data; KL is the penalty for a bad model. Because $H(p)$ does not depend on your parameters, minimizing cross-entropy is identical to minimizing KL divergence. And since one-hot labels have $H(p) = 0$ , in standard classification cross-entropy and KL are numerically equal.

Entropy is a compression bound

Entropy is the minimum average number of bits needed to encode outcomes drawn from $p$ . Cross-entropy is what you actually spend when you build your code around a wrong distribution $q$ , and KL divergence is the wasted overhead. "A good model" and "an efficient code" are the same idea.

Common mistakes

KL divergence is not symmetric: $D_{KL}(p \parallel q) \neq D_{KL}(q \parallel p)$ in general, so it is not a distance metric, the order of the arguments changes the answer and the behavior. Watch the log base: bits use $\log_2$ , but ML frameworks (PyTorch, TensorFlow) use the natural log, so their reported loss is in nats, not bits. Finally, cross-entropy needs $q$ to be a valid probability distribution, apply softmax first; feeding raw logits gives garbage. Use the convention $0\log 0 = 0$ for zero-probability terms.