Loading lesson…
AI Systems Engineering: Run, Operate & Evaluate / Operating & Evaluating
Precision, recall, F1, and judging generation.
Reviewed by Yuvaraj
"It seems better" is not evaluation. If you cannot measure whether a change helped, you are not engineering a system, you are decorating it. This lesson covers how to measure AI systems, from the classic classification metrics to the harder problem of judging open-ended text, and it pairs with a lab where you can feel the core trade-off directly.
For any yes/no decision, is this email spam, is this ticket urgent, is this transaction fraud, every prediction lands in one of four boxes:
| Actually positive | Actually negative | |
|---|---|---|
| Predicted positive | True positive (TP) | False positive (FP) |
| Predicted negative | False negative (FN) | True negative (TN) |
Every metric below is just a ratio of these four counts.
Answer from memory before revealing, retrieval practice is what builds durable recall.
On a dataset where only 1% of items are positive, why is accuracy a poor metric?
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
Precision and recall pull against each other. You trade between them by moving the decision threshold, the confidence level at which a score becomes a "yes."
Feel it in the Evaluation Playground
This trade-off is much easier to feel than to read. Open the Evaluation Playground, drag the threshold, and watch precision rise as recall falls (and vice versa) on a real dataset. Then ask which error your product can least afford, that answer chooses your threshold.
F1, the harmonic mean of precision and recall, is a common single-number summary when you care about both:
The harmonic mean (not the ordinary average) punishes imbalance: a model with 100% precision and 1% recall has an F1 near 2%, not 50%. That is the point, it refuses to let one good number hide one terrible one.
Take 1,000 transactions where only 10 are actually fraud. A model flags 12; 8 of those are real fraud (TP = 8, FP = 4) and it misses 2 (FN = 2), leaving TN = 986. Then:
Its accuracy is . But a model that flags nothing scores , nearly identical, while catching zero fraud. Accuracy barely separates a useful model from a useless one here; precision and recall tell you what actually happened.
Which error costs more?
There is no universally "best" threshold. A cancer screen should tolerate false alarms to avoid misses (favor recall); a spam filter should tolerate a little spam to avoid trashing real mail (favor precision). The metric follows the cost of the mistake, not the other way around.
Where you evaluate matters as much as what you measure:
Classification has a right answer; a summary or a chatbot reply does not. Techniques for open-ended output:
Evaluate the judge
LLM-as-judge is a tool, not an oracle. Before trusting its scores, check them against a sample of human judgments on your own task. An unvalidated automatic judge can make a worse system look better with total confidence.
Pick your metric before you make the change, tie it to the cost of the mistake you most want to avoid, and gate releases on it. Evaluation is not the paperwork after the work, it is how you know there was any work.