The Bias-Variance Tradeoff

Why it is genuinely a tradeoff

As you turn the complexity knob up:

• Bias falls, the model can represent richer relationships.
• Variance rises, the model becomes more sensitive to the specific training sample it saw.

Written out fully, the expected error at a point decomposes as:

$\mathbb{E}\big[(y-\hat f(x))^2\big]=\underbrace{(\mathbb{E}[\hat f(x)]-f(x))^2}_{\text{bias}^2}+\underbrace{\mathbb{E}\big[(\hat f(x)-\mathbb{E}[\hat f(x)])^2\big]}_{\text{variance}}+\underbrace{\sigma^2}_{\text{irreducible}}$

Because the irreducible noise $\sigma^2$ never changes, minimizing total error means minimizing the sum of the first two terms. One shrinks exactly as the other grows, so the minimum sits at some intermediate complexity, not at either extreme. Plotted against complexity, validation error traces a U-shape.

Worked example: polynomial regression

Suppose we fit polynomials of increasing degree to the same noisy dataset, then measure root-mean-squared error on the training set and on a held-out validation set.

Read the pattern carefully. Training error falls monotonically, more flexibility can always fit the training points more closely. Validation error is U-shaped: it improves as the model escapes underfitting, bottoms out at degree 3, then climbs as the model starts chasing noise. By degree 12 the model nearly interpolates the training data (error 0.05) yet is useless on new data (4.90). That widening gap between the two columns is the visible signature of rising variance.

Diagnosing your own model

1. 1Measure both errorsAlways compare training error against validation error. Training error alone tells you nothing about how the model will generalize.
2. 2Spot high biasIf both errors are high and close together, you are underfitting. Add complexity or features, or reduce regularization.
3. 3Spot high varianceIf training error is low but validation error is much higher, you are overfitting. Simplify the model, add regularization, or gather more training data.
4. 4Tune to the USweep the complexity knob and pick the setting where validation error is lowest, not where training error is lowest.

Further reading

• Hastie, Tibshirani, and Friedman, The Elements of Statistical Learning (Section 7.3, "The Bias–Variance Decomposition"), hastie.su.domains/ElemStatLearn
• Google Machine Learning Crash Course, "Overfitting: Generalization", developers.google.com/machine-learning/crash-course/overfitting/overfitting
• Scott Fortmann-Roe, "Understanding the Bias–Variance Tradeoff", scott.fortmann-roe.com/docs/BiasVariance.html