Loading lesson…
AI Research Foundations / Method
Turning a vague idea into a testable hypothesis.
Reviewed by Yuvaraj
Research begins with a question, but a question is not yet something you can answer. "Does retrieval help my model?" cannot be confirmed or refuted, it has no defined subject, no measurement, and no failure condition. The work of experimental design is turning such a question into a falsifiable hypothesis and building the smallest experiment that can decide it. This lesson is about that conversion, and about the discipline of isolating the one effect you claim to measure from everything else that could produce the same number.
A good research question is broad enough to matter and specific enough to investigate. But before you touch a GPU, you must sharpen it into a hypothesis: a precise, testable statement that predicts a particular outcome and could be proven wrong.
Falsifiability is the non-negotiable property. If no possible result would make you abandon the claim, it is not a hypothesis, it is a belief. "Retrieval improves my system" survives any outcome, because "improves" is undefined. "Adding BM25 retrieval of the top-5 passages raises exact-match accuracy on NaturalQuestions by at least 3 points over the no-retrieval baseline, at equal decoding budget" makes a sharp prediction: run it, and the number either clears 3 points or it does not.
The test of a real hypothesis
Ask: "What result would prove me wrong?" If you can name a specific, measurable outcome that would force you to retract the claim, you have a hypothesis. If nothing could, if every conceivable result is compatible with the statement, you have only a hope dressed as science.
Every experiment is organized around variables, and clarity here is the difference between a clean result and an uninterpretable one.
Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.
Finished this lesson?
Mark it complete to earn XP, keep your streak, and schedule a review.
The central rule of controlled experimentation: change one thing at a time. If you turn on retrieval and switch to a larger model and lengthen the context window in the same comparison, a gain tells you nothing, you cannot attribute it to retrieval. Isolating an effect means constructing two conditions that are identical in every respect except the single IV.
| Role | Retrieval example | Why it matters |
|---|---|---|
| Independent variable | Retrieval: off vs. on (top-5, BM25) | The cause under test, the only thing that differs |
| Dependent variable | Exact-match accuracy on the test set | The effect you measure to decide the hypothesis |
| Controls | Same model, prompt, decoding budget, eval split, seed | Held fixed so they cannot explain the difference |
| Confound (to eliminate) | Retrieval condition also uses more input tokens | More context, not retrieval, might drive any gain |
That last row is the subtle one. Turning on retrieval increases the number of tokens the model reads. If the retrieval condition simply has more context to work with, a gain might come from context length rather than from the relevance of what was retrieved. A careful design neutralizes this, for example, by giving the no-retrieval baseline the same number of randomly chosen passages, so the only thing that differs is whether the passages are relevant, not how many there are.
The minimal experiment that decides the sharp hypothesis is now almost mechanical to specify:
Notice what the minimal experiment is not: it does not sweep ten prompts, three models, and five datasets. Each of those additions answers a different question. The minimal experiment answers exactly the hypothesis you wrote and nothing more, which is precisely why its result is interpretable.
Pre-register the decision rule
Write down, before running anything, the metric, the parsing rule, the number of seeds, and the threshold that would confirm or refute the hypothesis. This is a scaled-down version of the pre-registration practice in the sciences. It protects you from the most natural form of self-deception: adjusting what counts as success after you have seen which way the numbers fell.
A confound is anything that offers a rival explanation for your result. The habit to cultivate is adversarial: after designing an experiment, argue against yourself. "Suppose retrieval showed a gain, what else, besides retrieval relevance, differed between my two conditions and could have caused it?" Each answer is a confound to control.
Common confounds in AI experiments include: unequal compute or token budgets between conditions; test-set leakage into the training or retrieval corpus; different amounts of hyperparameter tuning spent on the two conditions (the baseline is often under-tuned, see the baselines lesson); and evaluation noise mistaken for a real effect. The remedy is almost always the same principle applied more carefully: make the two conditions identical except for the single variable whose effect you are claiming.
Common mistakes