7. Initialization and vanishing gradients
Deep networks are hard to train because backpropagation multiplies one Jacobian per layer, so a signal can shrink or blow up geometrically with depth. This module explains where that instability comes from, why naive weight initialization makes it worse, and the two fixes that make deep training routine: variance-preserving initialization (Xavier and He) and gradient clipping.
Objectives
- Write backpropagation as a product of per-layer Jacobians and see when it vanishes or explodes.
- Connect the effect to activation saturation from lesson 3.
- Explain why all-zeros and badly scaled initializations fail.
- Derive the variance target that Xavier and He initialization satisfy.
- Apply gradient clipping to tame exploding gradients.
- Pick an initializer from the activation function.
7.1 Why depth is unstable
7.1.1 The Jacobian product
Recall the forward pass of lesson 6: layer \(l\) computes \(z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}\) and \(a^{[l]} = g^{[l]}(z^{[l]})\), with \(a^{[0]} = x\) and \(\hat{y} = a^{[L]}\). Backpropagation sends the loss gradient from the output back to layer \(l\) by the chain rule. The error signal \(\delta^{[l]} = \partial L / \partial z^{[l]}\) obeys the recurrence \(\delta^{[l]} = (W^{[l+1]})^T \delta^{[l+1]} \odot g'^{[l]}(z^{[l]})\), so unrolling it from the top layer \(L\) down to layer \(l\) gives a product:
\[\boxed{ \frac{\partial L}{\partial z^{[l]}} = \left( \prod_{k=l+1}^{L} \operatorname{diag}\!\left(g'^{[k-1]}(z^{[k-1]})\right) (W^{[k]})^T \right) \frac{\partial L}{\partial z^{[L]}} }\]Each factor is a layer Jacobian: a weight matrix \(W^{[k]}\) combined with the diagonal matrix \(\operatorname{diag}(g'(z))\) of activation slopes. The gradient reaching layer \(l\) is this whole product acting on the top-layer error.
7.1.2 Vanishing and exploding
A product of many factors is governed by their typical magnitude. Write \(\rho\) for the typical size (a spectral norm) of one factor \(W^{[k]} \odot \operatorname{diag}(g')\). Across \(L - l\) layers the signal scales roughly as \(\rho^{\,L-l}\):
\[\boxed{ \left\| \frac{\partial L}{\partial z^{[l]}} \right\| \;\approx\; \rho^{\,L-l} \left\| \frac{\partial L}{\partial z^{[L]}} \right\| }\]
If \(\rho < 1\) consistently the gradient shrinks toward zero as it travels back (the vanishing gradient), so early layers barely update and effectively stop learning. If \(\rho > 1\) it grows without bound (the exploding gradient), so updates overshoot and the loss diverges to NaN. Only \(\rho \approx 1\) keeps the signal alive across depth.
Gradient magnitude across depth: poorly scaled weights make it vanish or explode, while variance-preserving initialization keeps it near one.
Remark: the same product runs forward for the activations themselves. If layer outputs shrink or grow geometrically, the network cannot represent anything useful even before a gradient is computed, so we want both the forward signal and the backward gradient near unit scale.
7.1.3 The saturation connection
The factor \(g'(z)\) links this directly to the saturation seen in lesson 3. The sigmoid and \(\tanh\) flatten for large \(|z|\), so their derivatives fall to near zero there.
| Activation | \(g'(z)\) | max slope | slope when saturated |
|---|---|---|---|
| sigmoid | \(g(z)(1-g(z))\) | \(0.25\) | \(\to 0\) |
| \(\tanh\) | \(1 - \tanh^2(z)\) | \(1\) | \(\to 0\) |
| ReLU | \(1\) for \(z>0\), else \(0\) | \(1\) | \(0\) on the dead side |
Remark: the sigmoid slope never exceeds \(0.25\), so each layer multiplies the backward signal by at most a quarter. Stack ten sigmoid layers and the gradient is scaled by at most \(0.25^{10} \approx 10^{-6}\) before any weight is considered. This is why deep stacks of saturating units train poorly, and why ReLU (slope \(1\) on the active side) became the default.
7.2 Bad initializations
7.2.1 All zeros
Setting \(W^{[l]} = 0\) (or any value that makes every unit in a layer identical) breaks learning through symmetry. If two units in a layer start with the same weights and see the same input, they compute the same activation and receive the same gradient, so they update identically and stay identical forever. The layer then behaves like a single unit no matter how wide it is. Random initialization exists precisely to break this symmetry so units can specialize.
7.2.2 Wrong scale
Even with random, symmetry-breaking weights the variance matters. Consider a linear unit \(z = \sum_{j=1}^{n_{\text{in}}} W_j a_j\) with independent zero-mean weights and inputs. Its variance is a sum of \(n_{\text{in}}\) independent terms:
\[\boxed{ \operatorname{Var}(z) = n_{\text{in}} \cdot \operatorname{Var}(W) \cdot \operatorname{Var}(a) }\]If \(n_{\text{in}} \cdot \operatorname{Var}(W) > 1\) the signal variance grows layer after layer and explodes, and if it is \(< 1\) the variance decays and vanishes. To keep \(\operatorname{Var}(z) \approx \operatorname{Var}(a)\) from layer to layer we need \(n_{\text{in}} \cdot \operatorname{Var}(W) \approx 1\), which fixes the weight variance to roughly \(1 / n_{\text{in}}\). That single condition is the seed of both initializers below.
7.3 Variance-preserving initialization
7.3.1 Xavier / Glorot
Glorot and Bengio balance the forward pass (\(\operatorname{Var}(W) = 1/n_{\text{in}}\)) against the backward pass (\(\operatorname{Var}(W) = 1/n_{\text{out}}\)) by averaging the two, giving the Xavier initialization:
\[\boxed{ \operatorname{Var}(W^{[l]}) = \frac{2}{n_{\text{in}} + n_{\text{out}}} }\]Here \(n_{\text{in}} = n_{l-1}\) is the fan-in and \(n_{\text{out}} = n_l\) is the fan-out of the layer. Xavier is derived assuming the activation is roughly linear near the origin, so it suits symmetric, unit-slope activations like \(\tanh\) and the sigmoid.
7.3.2 He
ReLU zeros out half of its inputs on average, so it halves the variance of what passes through. He initialization compensates with a factor of two:
\[\boxed{ \operatorname{Var}(W^{[l]}) = \frac{2}{n_{\text{in}}} }\]This is the right target for ReLU and its variants (leaky ReLU, ELU, GELU). In practice you draw weights from a Gaussian with this variance, or from a uniform distribution with the matching range, and set the bias \(b^{[l]}\) to zero.
Remark: the bias starts at zero, not the weights. A zero bias does not create symmetry (the weights already differ), and it keeps the initial pre-activation centred so the activation starts in its responsive region rather than saturated.
7.3.3 The mechanism
Variance-preserving initialization keeps the signal and the gradient near unit scale through depth.
Choosing the variance is a one-time fix at the start of training. It positions the network so the Jacobian product of section 7.1 has factors near \(1\), but nothing keeps it there as the weights move during training. That is what the next module addresses.
7.4 Exploding gradients and clipping
Good initialization tames vanishing gradients and greatly reduces explosions, but explosions can still appear during training, especially in recurrent networks where the same weight matrix is reused at every time step. The standard remedy is gradient clipping: rescale the whole gradient vector \(g\) so its norm never exceeds a threshold \(\tau\).
\[\boxed{ g \leftarrow g \cdot \min\!\left(1, \frac{\tau}{\lVert g \rVert}\right) }\]When \(\lVert g \rVert \le \tau\) the factor is \(1\) and the gradient is untouched. When \(\lVert g \rVert > \tau\) the gradient is shrunk back to norm exactly \(\tau\) while keeping its direction, so a single huge step cannot blow up the weights.
Remark: clipping by global norm (rescaling the whole vector together) preserves the update direction, whereas clipping each coordinate independently to \([-\tau, \tau]\) can bend the direction. Global-norm clipping is the usual default.
7.5 Choosing an initializer
Match the initializer to the activation of the layer it feeds.
| Activation | Recommended initializer | Weight variance |
|---|---|---|
| ReLU, leaky ReLU, ELU, GELU | He | \(2 / n_{\text{in}}\) |
| \(\tanh\) | Xavier / Glorot | \(2 / (n_{\text{in}} + n_{\text{out}})\) |
| sigmoid | Xavier / Glorot | \(2 / (n_{\text{in}} + n_{\text{out}})\) |
| softmax / linear output | Xavier / Glorot | \(2 / (n_{\text{in}} + n_{\text{out}})\) |
Remark: initialization and clipping only manage the signal at the endpoints of training and during large steps. Two structural remedies keep it controlled throughout: normalization re-centres and rescales activations at every layer, and residual connections add a shortcut that lets the gradient skip past the Jacobian product entirely.
Good initialization keeps the signal well scaled at step zero, but the statistics drift as training proceeds. The next module keeps them in check at every step with normalization.
Next: Normalization · Course overview