Every change to an AI system faces two different questions. Offline evaluation asks, on a frozen dataset with known answers, "is the new model measurably better than the old one?", a question you can answer in minutes, before a single user is exposed. Online evaluation asks the question that actually pays the bills: "when real people use this in production, do the outcomes we care about improve without anything important getting worse?" The two are not competitors; they are stages of the same pipeline, and a change that looks like a clear win offline can shrink to nothing, or backfire, once it meets live traffic.
Offline evaluation: judging on a frozen dataset
A held-out test set with reference labels or gold outputs is fixed in advance. You run the candidate model over every example, compute a metric (accuracy, F1, exact match, ROUGE, an LLM-as-judge score, retrieval recall@k, and so on), and compare it against the incumbent's score on the identical set. Because nothing about the data changes between runs, the comparison is deterministic and reproducible: same inputs, same metric, same number. That is what makes offline evaluation cheap, fast, parallelizable, and safe, no user ever sees the candidate, which is exactly what you want for model selection and for a regression gate in CI that blocks a merge when a core score drops.
Its ceiling is set by a single assumption: that the frozen dataset resembles the traffic the model will actually serve. When it does not, the number is precise but misleading. Two failure modes recur:
Distribution shift and staleness. The test set was collected months ago, or from one user segment, and no longer matches live queries.
Test-set overfitting and contamination. Tune hundreds of variants against the same held-out set and information leaks: the score inflates without real improvement (a Goodhart's-law effect). Contamination, test examples that also appear in training data, inflates it further. The fix is honest splits, a held-back set you rarely touch, and periodic refreshes.
Online evaluation: judging on live traffic
Here you route a fraction of real traffic to the candidate and measure real outcomes. Three designs are common:
A/B test. Randomly assign users or sessions to control (current system) and (candidate). The randomization is what lets you attribute any difference in outcomes to the change itself rather than to who happened to see it.
Ask the tutor
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
treatment
Canary / staged rollout. Send a small slice (1–5%) to the candidate first, watch the dashboards, and ramp up only if nothing breaks.
Interleaving. For ranking and search, blend both systems' results into one list and see which system's items users prefer, far more sensitive per user than a split test, so it needs less traffic to reach a conclusion.
Online you track two kinds of metric at once:
Primary (success) metrics, the real user impact you are trying to move: conversion, task-success rate, engagement, retention, revenue per session.
Guardrail metrics, things that must not regress even if the primary metric improves: p95/p99 latency, error rate, cost per request, crash rate, and safety/policy-violation rate. A treatment that lifts conversion but doubles latency or raises the harmful-output rate does not ship.
The price is real: online tests need enough traffic and time (often one to several weeks, to cover weekly seasonality and let novelty effects fade), they expose users to a possibly worse experience, and they are hard to reproduce because the world keeps moving underneath them.
1Offline gateScore the candidate on the frozen test set in CI and block anything that regresses a core metric, before it ever reaches traffic.
2Shadow / canaryServe the candidate to 1-5% of traffic (or in shadow, with no user-visible output) and watch guardrails for crashes, latency, and errors.
3A/B testRandomly split traffic between control and treatment and collect primary and guardrail metrics until the sample is large enough to decide.
4Ramp and monitorIf the primary metric wins and no guardrail regresses, ramp from 5% to 50% to 100%, keeping monitoring in place to catch novelty effects wearing off.
Why an offline win can disappear online
Distribution gap. Your eval set is not a random sample of production traffic, so a gain concentrated on easy or over-represented examples may not generalize to what users actually send.
Proxy-metric mismatch. Offline metrics are proxies. A higher ROUGE, exact-match, or benchmark accuracy is a stand-in for "users are better served," and the correlation is imperfect, so moving the proxy need not move conversion or task success. Optimizing hard against a proxy is precisely when it stops tracking the goal.
System effects that do not exist offline. A bigger model that scores better may add latency that costs more conversions than the extra quality earns; caching, feedback loops, and UI only appear in production.
Inflated offline numbers. As above, an offline lead can be partly illusory from test-set overfitting or contamination.
They are complementary, not redundant
Offline evaluation is a cheap, safe filter: it rejects bad candidates fast and
catches regressions before launch. Online evaluation is the ground truth: it
confirms whether a surviving candidate actually helps real users. Offline is
the gate; online is the verdict. Skipping either one is how teams either ship
regressions or spend weeks A/B testing changes that a five-minute offline
check would have rejected.
Worked example: reading an A/B test
A search-ranking team replaces its reranker. Offline, recall@10 on the frozen judgment set rises from 0.71 to 0.74, a clear win, so the change passes the CI gate. They then run a two-week A/B test, splitting users 50/50. The primary metric is checkout conversion rate; latency and error rate are guardrails.
Metric
Control (A)
Treatment (B)
Users
50,000
50,000
Conversions
4,000
4,300
Conversion rate
8.00%
8.60%
p95 latency
480 ms
545 ms
Error rate
0.30%
0.31%
Observed absolute lift = 8.60% − 8.00% = +0.60 percentage points, a relative lift of 0.60/8.00=+7.5%.
Is it real, or noise? Use a two-proportion z-test. With p1=0.080, p2=0.086, and n1=n2=50000, the pooled rate is
A z of 3.44 gives a two-sided p-value ≈0.0006, far below the usual 0.05 threshold, and the 95% confidence interval for the lift is roughly [+0.26 pp, +0.94 pp], it excludes zero. The conversion gain is statistically significant, not noise. On the guardrail side, p95 latency rose 480 → 545 ms (+65 ms); if the team's guardrail is "no more than +100 ms," this passes, and error rate is flat. Decision: ship, then ramp while monitoring.
Now the sample-size caveat. Suppose the same 8.00% vs 8.60% split had come from only 1,000 users per arm instead of 50,000:
a two-sided p-value ≈0.62, indistinguishable from noise. The observed lift is identical; only the evidence differs. Underpowered tests cannot tell a real 0.6-point gain from random fluctuation, which is why you fix the sample size with a power calculation before you look at results, not after.
Offline vs online at a glance
Offline evaluation
Measures a proxy metric on a frozen, labeled dataset (accuracy, F1, recall@k, judge score)
Runs before deploy: model selection and CI regression gates
Cheap and fast (minutes), fully parallel, and reusable
Deterministic and reproducible: same data gives the same number
Zero risk to users, since nobody is exposed
Only as trustworthy as the dataset; prone to staleness, overfitting, and contamination
Online evaluation
Measures real user outcomes on live traffic plus must-not-regress guardrails
Runs after the offline gate: canary, A/B test, interleaving
Expensive: needs live traffic and days-to-weeks of runtime
Hard to reproduce, because the world keeps changing
Real risk to users, since a bad treatment degrades the live experience
Ground truth on the impact you actually care about
Dimension
Offline
Online
What it measures
Proxy metric on fixed labeled data
Real user-impact metrics + guardrails on live traffic
Speed
Minutes
Days to weeks
Cost
Low (compute only)
High (traffic, time, exposure)
Reproducible?
Yes, deterministic
No, non-stationary world
User risk
None
Real exposure to a possibly worse system
Best for
Filtering candidates, regression gates
Confirming true impact before full rollout
Common mistakes
Shipping on an offline win alone. A higher benchmark or judge score is a hypothesis, not a result. Confirm it online before a full rollout.
Peeking and stopping early. Repeatedly checking a running A/B test and stopping the moment p<0.05 badly inflates false positives. Fix the sample size in advance, or use a sequential test designed for continuous monitoring.
Ignoring guardrails. A conversion lift bought with a latency, cost, or safety regression is often a net loss. Decide the guardrail thresholds before the test, not after you see the numbers.
Reusing the test set until it lies. Tuning many variants against one held-out set, or leaking test data into training, inflates offline scores without any real gain.
Confusing statistical with practical significance. With millions of users a trivial +0.02% can be "significant" yet not worth the added complexity. Pre-register the minimum effect size you actually care about.
Ron Kohavi's experimentation papers, including "Controlled experiments on the web: survey and practical guide", https://exp-platform.com/, a foundational reference on trustworthy A/B testing and its pitfalls.