Overfitting & Regularization

$\mathcal{L}_{\text{reg}} = \mathcal{L} + \lambda \lVert w \rVert^2$

Here $\lVert w \rVert^2 = \sum_i w_i^2$ is the squared L2 norm (this is ridge regularization). The hyperparameter $\lambda$ controls the trade-off:

• $\lambda = 0$ recovers the original loss, no penalty, maximum freedom to overfit.
• Small $\lambda > 0$ gently discourages large weights.
• Large $\lambda$ forces weights toward zero, which can cause underfitting.

So $\lambda$ trades fit against simplicity: it is the dial between low bias / high variance and high bias / low variance. The L1 (lasso) variant swaps the penalty for $\lambda \lVert w \rVert_1 = \lambda \sum_i |w_i|$, which behaves quite differently.

Beyond weight penalties

Penalties are only one lever. For most models, and especially neural networks, combine several remedies.

1. 1More data and augmentationThe most reliable cure. More varied examples make memorization harder; augmentation (flips, crops, noise, paraphrases) synthesizes new samples for free.
2. 2Early stoppingMonitor validation error during training and stop at its minimum, before the model begins fitting noise.
3. 3Dropout (neural nets)Randomly zero a fraction of activations each step so no neuron can depend on any single partner; approximates training an ensemble.
4. 4L2 / L1 penaltiesAdd the weight penalty to the loss so the optimizer keeps weights small unless the data strongly justifies otherwise.

Worked example

Fit the same three-feature regression two ways. The unregularized model minimizes raw MSE and lands on large weights; the ridge model ($\lambda = 0.1$) pays a penalty for size.

import numpy as np

w_plain = np.array([8.0, -6.5, 5.2])   # unregularized fit
w_ridge = np.array([1.9, -1.2, 0.8])   # ridge, lambda = 0.1
lam = 0.1

lam * np.sum(w_plain**2)   # 0.1 * 133.29 = 13.33
lam * np.sum(w_ridge**2)   # 0.1 *   5.69 =  0.57

The unregularized model drives training MSE almost to zero using big weights, but validation MSE is a dismal 0.95, textbook overfitting. Once the penalty is active, those large weights cost 13.33 in the objective, so the optimizer settles for smaller ones. Training MSE rises slightly to 0.15, yet validation MSE falls to 0.30, and the train/val gap shrinks from 0.93 to 0.15. That is regularization working: a little worse on the training set, much better on new data.

Further reading

• Google ML Crash Course, Regularization for Simplicity: L2
• scikit-learn, Linear Models (Ridge, Lasso, and more)
• Deep Learning (Goodfellow, Bengio, Courville), Chapter 7: Regularization