Reproducibility & Significance

A single run is not evidence

Here is the trap at the heart of empirical ML. Suppose your method scores 84.6 and the baseline scores 83.1. Is your method better? You cannot say, because you do not yet know how much either number would move if you simply changed the seed. If re-running the baseline with different seeds produces anything from 82.0 to 85.5, then the 84.6 is comfortably inside the baseline's own range of luck, and your "improvement" is noise.

The remedy is to run each condition multiple times with different seeds and summarize the distribution, not a single draw. Report the mean and the standard deviation across runs, never a lone number. The sample mean over $n$ runs is

$\bar{x} = \frac{1}{n}\sum_{i=1}^{n} x_i$

and the (sample) standard deviation, which measures how much individual runs scatter around that mean, is

$s = \sqrt{\frac{1}{n-1}\sum_{i=1}^{n}(x_i - \bar{x})^2}$

Reporting "84.6" becomes "84.6 plus-or-minus 0.9 over 5 seeds." That single addition transforms an anecdote into data, because it exposes whether the gap between two methods is large or small relative to the noise.

Confidence intervals and what they actually tell you

A standard deviation describes how much individual runs scatter. But we usually care about how precisely we know the mean, and that precision improves as we average more runs. The standard error of the mean captures this:

$\mathrm{SE} = \frac{s}{\sqrt{n}}$

The $\sqrt{n}$ is the crucial and often-misunderstood part: to halve your uncertainty about the mean you must quadruple the number of runs. From the standard error we form a confidence interval, approximately

$\bar{x} \pm t^{*}\cdot \mathrm{SE}$

where $t^{*}$ is a multiplier from the t-distribution that depends on $n$ and the confidence level (near 2 for a 95 percent interval with a handful of runs).

A confidence interval is easy to misread. A 95 percent confidence interval does not say "there is a 95 percent probability the true mean is in this interval." It says: if you repeated the whole experiment many times, about 95 percent of the intervals constructed this way would contain the true mean. Operationally, treat it as a statement about the precision of your estimate: a wide interval means you have not run enough seeds to know the mean well, and two methods whose intervals overlap heavily are not distinguishable on this evidence.

Significance tests, effect size, and the difference between them

A significance test formalizes "is this gap bigger than noise?" You posit a null hypothesis, typically "the two methods have the same true mean", and compute a p-value: the probability of observing a gap at least as large as the one you saw, if the null were true. A small p-value (conventionally below 0.05) means the observed gap would be surprising under the assumption of no real difference, so you have evidence against that assumption.

Two cautions matter more than the mechanics. First, a p-value is not the probability that your hypothesis is true, and it is not a measure of how big the effect is, with enough runs, a trivially small difference becomes "statistically significant." Second, p-values are easily abused: testing many configurations and reporting only the one that crossed 0.05 (p-hacking) manufactures false positives.

This is why you must also report effect size, how large the difference is, in units you care about, independent of sample size. A common standardized measure is Cohen's d, the gap between the means in units of the pooled standard deviation $s_p$:

$d = \frac{\bar{x}_A - \bar{x}_B}{s_p}$

The pairing is the point: the p-value (or confidence interval) tells you whether an effect is distinguishable from noise, while the effect size tells you whether it is large enough to matter. A result can be statistically significant yet practically negligible, or practically large yet not yet significant because you ran too few seeds. Report both, and interpret them together.

1. 1Run each condition over multiple seedsFive is a common minimum for deep learning; more if the variance is high. Vary the seed only, hold data, code, and environment fixed.
2. 2Report mean and standard deviationNever a single number. State how many seeds and exactly what varied between them.
3. 3Quantify the uncertainty in the meanCompute the standard error and a confidence interval so readers can see how precisely the mean is known.
4. 4Test the difference and report effect sizeUse an appropriate significance test for whether the gap exceeds noise, and report an effect size for whether it is large enough to care about.
5. 5Pre-commit and avoid p-hackingDecide the comparison and metric before running; do not search many settings and report only the one that reached significance.

Further reading

• The ML Reproducibility Checklist (Pineau et al.) and the NeurIPS reproducibility program, the standard list of what to pin and report: https://www.cs.mcgill.ca/~jpineau/ReproducibilityChecklist.pdf
• Henderson et al., "Deep Reinforcement Learning that Matters", shows how ignoring seed variance produces irreproducible, non-significant claims: https://arxiv.org/abs/1709.06560
• Dror et al., "The Hitchhiker's Guide to Testing Statistical Significance in Natural Language Processing", a practical guide to significance testing for ML/NLP: https://aclanthology.org/P18-1128/
• Wasserstein and Lazar, "The ASA Statement on p-Values: Context, Process, and Purpose", the American Statistical Association on what p-values do and do not mean: https://doi.org/10.1080/00031305.2016.1154108
• Bouthillier et al., "Accounting for Variance in Machine Learning Benchmarks", on estimating and reporting variance rigorously: https://arxiv.org/abs/2103.03098