Machine Learning / Training & Evaluation

Train, Validation & Test

Three splits and why the test set is sacred.

L1 · BeginnerFoundationalKnown~12 min

Reviewed by YuvarajPublished 25 Aug 2026

When you train a machine learning model, the one number you cannot trust is how well it scores on the data it learned from. A model can memorize its training examples and look brilliant, yet fall apart the moment it meets anything new. To measure real performance you have to hold data back and judge the model on examples it has never seen. That is the whole reason we split a dataset into three parts, training, validation, and test, each with a distinct job and a strict rule about how often you are allowed to look at it.

The three splits

Each split answers a different question, and keeping them separate is what keeps your evaluation honest.

Training set, the model fits its parameters here (the weights of a linear model, the split points of a tree). It sees this data over and over.
Validation set, you use this to tune hyperparameters (learning rate, tree depth, regularization strength) and to choose between models. The model never trains on it, but you consult it many times, so it indirectly shapes your decisions.
Test set, a locked box. You touch it once, at the very end, to get an honest estimate of performance on unseen data. Tune against it and that estimate is no longer honest.

1FitTrain candidate models on the training set.

Ask the tutor

Ask about this lesson, or about anything in AI. Answers cite the lessons they draw on.

Finished this lesson?

Mark it complete to earn XP, keep your streak, and schedule a review.

2Tune and selectCompare them on the validation set, adjust hyperparameters, pick the winner.

3Estimate onceScore the single chosen model on the test set, one final read.

4ReportThe test score is what you quote as expected real-world performance.

Why the test set is touched once

Every time a decision reacts to a score, that data leaks into the model through you. Validation data is meant to influence decisions, so its score is optimistic by design. The test set stays pristine precisely so it can act as an unbiased referee. Peek at it repeatedly and it quietly becomes a second validation set, and you lose your only trustworthy estimate.

A worked example: 1,000 examples

Suppose you have 1,000 labeled examples and use a 70/15/15 split. First shuffle the rows (unless the data is a time series, see below), then slice:

Split	Share	Count	Used for
Train	70%	700	Fitting parameters
Validation	15%	150	Tuning and model choice
Test	15%	150	One final estimate

You train on 700, compare configurations on 150, and, after everything is frozen, report accuracy on the last 150.

k-fold cross-validation for small data

With only a few hundred examples, a single 150-row validation set is noisy: your estimate swings depending on which rows happened to land in it. k-fold cross-validation fixes this by rotating the validation role. Split the data (outside the test set) into $k$ equal folds, say $k = 5$ folds of 200 rows each.

One round: hold out fold 1 as the validation set (200 rows), train on the other four folds (800 rows), and record the score. Then repeat with fold 2 held out, and so on for all five rounds. Averaging the five scores gives a far more stable estimate, and you still keep an untouched test set for the final check.

Single holdout

Fast, one training run
Noisy on small data
Locks rows into a fixed validation set

k-fold

Every row validates exactly once
Stable averaged estimate
Costs k training runs

Common mistakes

Leaking through preprocessing. Fit scalers, imputers, and encoders on the training fold only, then apply them to validation and test. Fitting on the whole dataset lets test statistics seep into training.
Tuning on the test set. If you compare models or pick a decision threshold using test scores, the test set has become a validation set and your final number is inflated.
Shuffling time series. For temporal data, random splits leak the future into the past. Split by time, train on earlier data, test on later, using forward-chaining rather than shuffling.