9. Regularization and dropout
A deep network has enough capacity to fit almost any training set, including its noise. Regularization is the set of techniques that trade a little training accuracy for better generalization. This module covers weight decay (\(L_2\)), its \(L_1\) counterpart, dropout with the inverted-dropout rescaling, and the lighter-weight regularizers early stopping and data augmentation.
Objectives
- Recall what overfitting is and why high-capacity networks are prone to it.
- Add an \(L_2\) penalty to the cost and read off its effect on the gradient (weight decay).
- Contrast \(L_2\) with \(L_1\) and their different pressures on the weights.
- Apply inverted dropout as a Bernoulli mask with \(1/p\) rescaling.
- Explain the ensemble view of dropout and why the rescaling leaves activations unbiased.
- Place early stopping and data augmentation in the same generalization toolbox.
9.1 Overfitting recap
A model overfits when it drives its training cost \(J\) toward zero by memorizing the training examples, including their noise, so it generalizes poorly to unseen data. The gap between training performance and test performance is the tell. Deep networks are especially exposed because their parameter count \(\sum_l n_l\, n_{l-1}\) usually exceeds the number of training examples, so they have the capacity to memorize.
Remark: the underlying bias-variance trade-off was introduced in the Machine Learning course, see General concepts. Regularization pushes a high-variance model back toward the sweet spot.
The remedy is to constrain the effective capacity so the network prefers simpler functions. Every technique below is one such constraint.
9.2 L2 regularization (weight decay)
9.2.1 The penalty
\(L_2\) regularization adds a penalty proportional to the squared magnitude of every weight matrix to the cost. With \(\lambda \ge 0\) the regularization strength, the regularized cost is:
\[\boxed{ J_{\text{reg}} = J + \frac{\lambda}{2}\sum_{l=1}^{L}\lVert W^{[l]} \rVert_F^2 }\]where \(\lVert W^{[l]} \rVert_F^2 = \sum_{i,j}\big(W^{[l]}_{ij}\big)^2\) is the squared Frobenius norm. The biases \(b^{[l]}\) are normally left out of the penalty, as they add negligible capacity and penalizing them tends to underfit.
9.2.2 Effect on the gradient
Differentiating the penalty is what gives the technique its second name. The extra term contributes \(\lambda W^{[l]}\) to the gradient with respect to \(W^{[l]}\):
\[\boxed{ \frac{\partial J_{\text{reg}}}{\partial W^{[l]}} = \frac{\partial J}{\partial W^{[l]}} + \lambda\, W^{[l]} }\]Plugging this into a gradient-descent step with learning rate \(\alpha\) shrinks the weight before the data-driven update is applied:
\[\boxed{ W^{[l]} \leftarrow (1 - \alpha\lambda)\, W^{[l]} - \alpha\,\frac{\partial J}{\partial W^{[l]}} }\]Remark: the factor \((1 - \alpha\lambda) < 1\) multiplies every weight each step, which is literally a decay toward zero. That is why \(L_2\) regularization is called weight decay. Smaller weights mean a smoother, lower-variance function.
9.2.3 Contrast with L1
Replacing the squared norm with the absolute-value norm gives \(L_1\) regularization, penalizing \(\lambda\sum_l \lVert W^{[l]} \rVert_1 = \lambda\sum_{l,i,j}\lvert W^{[l]}_{ij}\rvert\). Its gradient contribution is \(\lambda\,\operatorname{sign}(W^{[l]})\), a constant pull toward zero regardless of magnitude.
| Penalty | Added to cost | Gradient term | Pressure on weights |
|---|---|---|---|
| \(L_2\) | \(\tfrac{\lambda}{2}\lVert W \rVert_F^2\) | \(\lambda W\) | shrinks all weights proportionally, rarely exactly zero |
| \(L_1\) | \(\lambda\lVert W \rVert_1\) | \(\lambda\,\operatorname{sign}(W)\) | drives many weights to exactly zero (sparse) |
Remark: \(L_1\) produces sparse weight matrices and so doubles as feature selection. \(L_2\) is the default in deep learning because it is smooth everywhere and pairs cleanly with gradient descent.
9.3 Dropout
9.3.1 The idea
Dropout regularizes by injecting noise into the activations. On each training forward pass, every unit is kept with probability \(p\) and zeroed with probability \(1 - p\), independently. The network therefore cannot rely on any single unit, so it spreads the representation across many units and stops co-adapting them.
9.3.2 Inverted dropout
Let \(m\) be a Bernoulli\((p)\) mask with the same shape as the activation \(a^{[l]}\), drawn fresh every step. Inverted dropout applies the mask and immediately divides by \(p\):
\[\boxed{ \tilde{a}^{[l]} = \frac{m \odot a^{[l]}}{p}, \qquad m_i \sim \text{Bernoulli}(p) }\]The masked, rescaled \(\tilde{a}^{[l]}\) then flows into layer \(l+1\) in place of \(a^{[l]}\). At inference time dropout is switched off and behaves as the identity, \(\tilde{a}^{[l]} = a^{[l]}\), with no mask and no rescaling.
Remark: keeping the \(1/p\) rescaling at training time (hence inverted) is what lets inference stay a plain forward pass. The older, non-inverted form instead multiplied weights by \(p\) at test time, which is easy to forget.
9.3.3 Why the rescaling
Because \(\mathbb{E}[m_i] = p\), the expected value of a kept, rescaled unit equals the original activation:
\[\boxed{ \mathbb{E}\!\left[\tilde{a}^{[l]}_i\right] = \frac{p\cdot a^{[l]}_i + (1-p)\cdot 0}{p} = a^{[l]}_i }\]So the expected input to the next layer is unchanged, and the network sees the same average signal with dropout on or off. This is exactly why no correction is needed at inference.
9.3.4 The ensemble view
A network with \(k\) droppable units defines \(2^k\) possible thinned subnetworks, one per mask. Each training step samples one subnetwork and takes a gradient step on it, and all subnetworks share weights. At test time the full network with rescaled activations approximates the average prediction of this exponentially large ensemble, which is why dropout behaves like cheap model averaging.
Dropout trains a different thinned subnetwork on each step by randomly removing units, and averages them at inference.
Remark: typical keep probabilities are \(p\) around \(0.8\) for input layers and \(0.5\) for hidden layers. A smaller \(p\) means stronger regularization.
9.4 Other regularizers
9.4.1 Early stopping
Track the validation cost during training and stop at the epoch where it starts rising, even though the training cost is still falling. Halting early keeps the weights near their small initial values, so it acts like an implicit \(L_2\) penalty without adding a term to the cost.
Training loss keeps falling while validation loss turns upward, the gap is overfitting and its minimum is where early stopping halts training.
9.4.2 Data augmentation
Expand the training set with label-preserving transformations of the inputs (random crops, flips, small rotations, colour jitter for images, noise for audio). More effective variety in the data lowers variance directly, which is regularization applied to the dataset rather than to the weights.
9.4.3 Summary
| Technique | Where it acts | Effect |
|---|---|---|
| \(L_2\) (weight decay) | cost via \(\lambda W\) | shrinks weights, smoother function |
| \(L_1\) | cost via \(\lambda\,\operatorname{sign}(W)\) | sparse weights, feature selection |
| Dropout | activations at training | ensemble of thinned subnetworks |
| Early stopping | training loop | keeps weights near initialization |
| Data augmentation | training data | more variety, lower variance |
Remark: these techniques compose. A convolutional network commonly uses weight decay, dropout, and heavy data augmentation together.
With overfitting under control, the next module builds an architecture whose weight sharing is itself a form of regularization: the convolutional network.