Loading lesson…
AI Research Foundations / Method
What makes a result believable.
Reviewed by Yuvaraj
A single accuracy number means nothing on its own. "Our model reaches 87 percent" is only interpretable relative to what a sensible alternative achieves under the same conditions, the baseline. And when a system with many parts improves on that baseline, the improvement is unattributed until you show which part caused it. This lesson covers the two disciplines that turn a raw number into a defensible claim: fair baselines and ablation studies.
Every performance figure is implicitly a comparison. 87 percent accuracy is impressive against a 70 percent prior state of the art and embarrassing against a 95 percent one. The baseline is the reference point that gives your number meaning, and choosing it well is a core act of honest science.
A baseline must be two things at once: fair and well-tuned. Fair means it is evaluated under identical conditions, same data splits, same metric, same compute budget, same evaluation code. Well-tuned means it was given a genuine chance to perform: its hyperparameters were searched with the same effort you spent on your own method. A baseline you tuned for an afternoon while lavishing a week on your method is not a baseline; it is a strawman.
Always compute the trivial baseline first
Before comparing against a sophisticated competitor, compute the dumbest reasonable predictor: the majority class for classification, the mean for regression, a random-retrieval baseline for a retrieval system. On imbalanced data, "90 percent accuracy" can be below the majority-class baseline of 92 percent. If your method cannot beat the trivial baseline, no amount of architectural elegance rescues the result.
The most common way strong-looking results are manufactured, sometimes deliberately, more often through motivated carelessness, is a weakened baseline. The gap between your method and the baseline can be widened from either side: by improving your method, or by handicapping the comparison. Only the first is real progress.
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
The tell-tale symptom is an asymmetry of effort. If the paper describes an elaborate hyperparameter search for the proposed method and reports baseline numbers copied from an old paper (run on different splits, or with less compute), the comparison is not fair even if no one intended to deceive. When you read a paper, this is where much of your skepticism should be spent; when you write one, this is where your integrity is most tested.
Prefer a re-run baseline to a quoted one
Numbers quoted from another paper were produced under that paper's conditions, its splits, its preprocessing, its evaluation harness. Whenever feasible, re-run the baseline yourself inside your own pipeline so that the only difference between it and your method is the method itself. A quoted number carries hidden differences you cannot control.
Suppose your system beats a strong baseline. You have earned the right to say the system is better, but not yet to say why. Real systems bundle many changes: a new architecture component, a new loss term, a data-augmentation trick, a longer training schedule. Any of them, or their interaction, could be responsible. An ablation study disentangles this by removing one component at a time and re-measuring, holding everything else fixed.
The logic is identical to the controlled-experiment principle from the previous lesson, applied inside your own system: to attribute an effect to component X, compare the full system to the same system with X removed and nothing else changed. The drop in performance when X is removed is your estimate of X's contribution.
Consider a retrieval-augmented QA system that reaches 84.0 exact-match. An ablation table isolates where that performance comes from:
| Configuration | Exact-match | Delta vs. full |
|---|---|---|
| Full system | 84.0 | , |
| − reranker (retrieve top-5 directly) | 80.3 | −3.7 |
| − query rewriting | 82.6 | −1.4 |
| − hard-negative training | 81.1 | −2.9 |
| − reranker − query rewriting | 78.9 | −5.1 |
| No retrieval (baseline) | 71.2 | −12.8 |
This table tells a story the single 84.0 number cannot. Retrieval itself accounts for most of the gain (the 12.8-point gap to the no-retrieval baseline). Among the refinements, the reranker contributes most (3.7 points), hard-negative training is close behind (2.9), and query rewriting is the smallest (1.4). The combined row (−5.1 from removing two components) also shows the effects are not perfectly additive, 3.7 plus 1.4 is 5.1 here, roughly additive, but interactions can make the joint removal larger or smaller than the sum, and reporting it exposes that.
Ablations are how you earn the word 'because'
A results table lets you say "our system is better." An ablation table lets you say "our system is better because of component X." The second claim is the scientific contribution, it is transferable knowledge others can reuse, while the first is merely a leaderboard entry. Reviewers and careful readers weight the ablation section accordingly.
Three rules keep an ablation interpretable. First, remove exactly one component per row (plus a few deliberate combinations to probe interactions); if you change two things, you are back to the confounded comparison of the previous lesson. Second, when you remove a component, decide honestly what replaces it, removing a reranker means "use the raw top-k," and that replacement must be stated, because a removal is really a substitution. Third, hold the compute and training budget fixed across ablation rows where possible, so a drop reflects the missing component and not simply less training.
A subtle case is the component that only helps in the presence of another. Query rewriting might contribute little on its own but a lot once the reranker is present, because rewriting surfaces candidates the reranker can then exploit. This is why a good ablation includes a few combination rows and does not assume contributions add up independently.
Common mistakes