What to Measure

Most evaluation failures are not arithmetic errors. They are a correct computation of an invalid proxy.

The classification family

When the task is a yes/no decision, four counts describe every prediction: true positives ($TP$), false positives ($FP$), false negatives ($FN$), and true negatives ($TN$). Every classification metric is a ratio of these four numbers.

• Accuracy, $\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}$, the fraction of all predictions that are correct.
• Precision, $\text{Precision} = \frac{TP}{TP + FP}$, of everything you flagged as positive, how much was right? It punishes false alarms.
• Recall, $\text{Recall} = \frac{TP}{TP + FN}$, of everything that was truly positive, how much did you catch? It punishes misses.

Precision and recall pull against each other, so a single summary is convenient. The F1 score is their harmonic mean, which stays low unless both are reasonable:

$F_1 = 2 \cdot \frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$

Accuracy weights every example equally, so it is dominated by whichever class is most common. On imbalanced data, where the interesting class is rare, the majority class alone produces a high score. The base rate, not the model's skill, is doing the work. That is the single most common way a metric lies, and it is worth watching happen in numbers.

Worked example: a "99% accurate" model that catches nothing

A platform flags abusive messages. In a labelled sample of 10,000 messages, 1% are truly abusive, 100 positives and 9,900 negatives. Consider the laziest possible model: it always predicts safe.

1. 1Accuracy(0 + 9,900) / 10,000 = 0.990, a 99% accurate model that has learned nothing.
2. 2Recall0 / (0 + 100) = 0, it catches none of the 100 abusive messages.
3. 3Precision0 / (0 + 0) is undefined: the model made no positive predictions, so there is nothing to be right about. scikit-learn reports 0 here and raises a divide-by-zero warning.
4. 4F1The harmonic mean is 0 whenever either component is 0, so F1 = 0 no matter how high accuracy climbs.

Now compare a genuine detector that flags 150 messages, 70 of them correctly: TP = 70, FP = 80, FN = 30, TN = 9,820. Its precision is $70/150 \approx 0.47$, its recall is $70/100 = 0.70$, and its F1 is $2 \cdot (0.47 \cdot 0.70)/(0.47 + 0.70) \approx 0.56$.

The useful model has lower accuracy than the useless one, 98.9% against 99.0%, because catching abuse means accepting some false alarms, and every false alarm costs an accuracy point. Accuracy cannot tell these two models apart; it even prefers the wrong one. Precision, recall, and F1 separate them instantly, moving the F1 from $0$ to $0.56$.

Quality is only one axis: guardrail metrics

A single quality number, however well chosen, describes only whether the output is good. It says nothing about whether the system is shippable. Production systems optimize a primary quality metric subject to guardrails, secondary metrics that must not regress, even when quality improves.

• Latency, report p95 or p99, not the mean; the tail is what users feel.
• Cost per request, computed from real token counts. A two-point quality gain at 15x the cost is often economically unshippable.
• Safety / policy-violation rate, a rare harmful output can outweigh strong average quality.
• Reliability, how often the system times out, refuses, or returns malformed output.

The pattern is to pick one primary metric to maximize and a small set of guardrails to hold. "Best F1 subject to p95 latency under 800 ms and cost under $2 per 1,000 requests" is a shippable target; "best F1" alone is not.

Validity and Goodhart's law

The deepest hazard in choosing a metric is that measuring changes behavior. Goodhart's law, in Marilyn Strathern's crisp phrasing, warns that when a measure becomes a target, it ceases to be a good measure. A proxy is chosen because it correlates with the goal on today's data. The moment you optimize the proxy hard, you push the system into regimes where that correlation no longer holds, you get more of the number and less of the thing it stood in for.

The failure is concrete in AI evaluation:

• Optimize a thumbs-up rate and models learn to be agreeable and flattering, sycophancy, rather than correct.
• Optimize n-gram overlap with a reference (BLEU, ROUGE) and you reward verbose, keyword-stuffed answers that game the overlap.
• Optimize average latency and answers get truncated or shallow so they finish fast.
• Optimize an LLM-judge score and outputs drift toward the judge's biases, length, confidence, formatting, not toward what users need.

Common mistakes

Further reading

• scikit-learn, Metrics and scoring, precise definitions of precision, recall, F1, and the zero_division behavior for the undefined case.
• Google ML Crash Course, Classification, accuracy, precision, and recall on imbalanced data.
• Hugging Face, Evaluation Guidebook, a practical guide to choosing and running evaluations for modern models.
• Manheim and Garrabrant, Categorizing Variants of Goodhart's Law (arXiv:1803.04585), a rigorous taxonomy of how optimizing a proxy diverges from the goal.