3. Activation functions
Each layer computes a pre-activation \(z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}\) and then an activation \(a^{[l]} = g^{[l]}(z^{[l]})\). The choice of the nonlinearity \(g^{[l]}\) is what makes depth worthwhile. This lesson explains why a nonlinear \(g\) is required, surveys the sigmoid, tanh, and ReLU families, introduces the softmax used at the output, and gives practical guidance on which activation to pick.
Objectives
- Show that a stack of purely linear layers collapses to a single linear map.
- Define the sigmoid and tanh, derive their derivatives, and explain saturation.
- Survey the ReLU family (ReLU, leaky ReLU, PReLU, ELU, GELU) and the dead-unit problem.
- Define the softmax and place it at the output rather than in hidden layers.
- Give a short rule of thumb for choosing an activation per layer.
3.1 Why nonlinearity is required
Suppose every activation were the identity, \(g^{[l]}(z) = z\). Then each layer is just \(a^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}\), and composing two of them gives \(W^{[2]}(W^{[1]} x + b^{[1]}) + b^{[2]} = (W^{[2]} W^{[1]}) x + (W^{[2]} b^{[1]} + b^{[2]})\). That is again of the form \(W x + b\). By induction the whole \(L\)-layer network reduces to a single affine map:
\[\boxed{ g^{[l]} = \text{identity} \;\Rightarrow\; \hat{y} = W' x + b' }\]with \(W' = W^{[L]} \cdots W^{[1]}\) and \(b'\) the accumulated bias. No matter how many linear layers are stacked, the model can only fit a linear function, so the extra depth buys nothing. A nonlinear \(g\) between layers is exactly what breaks this collapse and lets the network represent curved decision boundaries and nonlinear regressions.
Remark: the bias is kept explicit here as \(b^{[l]}\), unlike the Machine Learning course where the intercept was folded into \(w^T x\) via the augmented input \(x_0 = 1\). In this Deep Learning course each layer has its own weight matrix \(W^{[l]}\) and its own bias vector \(b^{[l]}\).
3.2 Sigmoid and tanh
Common activation functions: the bounded sigmoid and tanh saturate in their tails, while ReLU and its variants stay linear for positive inputs.
3.2.1 Sigmoid
The sigmoid squashes any real pre-activation into the open interval \((0, 1)\):
\[\boxed{ \sigma(z) = \frac{1}{1 + e^{-z}} \in (0, 1) }\]Its derivative has the convenient closed form below, which reuses the forward value \(\sigma(z)\) already computed:
\[\boxed{ \sigma'(z) = \sigma(z)\left(1 - \sigma(z)\right) }\]3.2.2 Tanh
The hyperbolic tangent is a rescaled sigmoid centred at zero, with output in \((-1, 1)\). Its derivative is likewise expressible from the forward value:
\[\boxed{ \tanh'(z) = 1 - \tanh(z)^2 }\]Remark: \(\tanh\) is zero-centred while \(\sigma\) is not, so \(\tanh\) often trains a little better as a hidden activation. The two are related by \(\tanh(z) = 2\sigma(2z) - 1\).
3.2.3 Saturation
Both curves flatten in their tails. For large \(|z|\) the output is close to a constant (\(0\) or \(1\) for \(\sigma\), \(\pm 1\) for \(\tanh\)), so the derivative is close to zero: \(\sigma'(z) \to 0\) and \(\tanh'(z) \to 0\). A unit sitting in that flat region is said to saturate, and it passes almost no gradient backward. When many such factors multiply through a deep stack the gradient shrinks toward zero, the vanishing-gradient problem revisited in Initialization and vanishing gradients.
Activation derivatives: sigmoid and tanh gradients vanish in the tails, whereas the ReLU gradient is 1 wherever the unit is active.
3.3 The ReLU family
The rectified linear unit keeps the positive part of its input and zeroes the rest:
\[\boxed{ \text{ReLU}(z) = \max(0, z) }\]Its derivative is \(1\) for \(z > 0\) and \(0\) for \(z < 0\) (undefined at \(z = 0\), taken to be \(0\) or \(1\) by convention). ReLU does not saturate on the positive side, so it keeps a healthy gradient flowing there, which is a large part of why it became the default hidden activation. The cost is the dead-unit problem: if a unit's pre-activation is always negative across the data, its gradient is always zero and it stops learning entirely. The variants below trade a little simplicity to soften that failure or to smooth the kink at the origin.
| name | formula | derivative | dies / saturates? |
|---|---|---|---|
| ReLU | \(\max(0, z)\) | \(1\) if \(z>0\) else \(0\) | can die (zero gradient for \(z<0\)) |
| Leaky ReLU | \(\max(\alpha z, z)\), \(\alpha \approx 0.01\) | \(1\) if \(z>0\) else \(\alpha\) | rarely dies (small negative slope) |
| PReLU | \(\max(\alpha z, z)\), \(\alpha\) learned | \(1\) if \(z>0\) else \(\alpha\) | rarely dies (\(\alpha\) trained per channel) |
| ELU | \(z\) if \(z>0\) else \(\alpha(e^z - 1)\) | \(1\) if \(z>0\) else \(\alpha e^z\) | saturates gently for \(z\to-\infty\) |
| GELU | \(z\,\Phi(z)\), \(\Phi\) the normal CDF | smooth, near \(1\) for large \(z\) | smooth, no hard death |
Remark: leaky ReLU and PReLU add a small slope \(\alpha\) on the negative side so a unit is never fully switched off. GELU weights the input by the probability \(\Phi(z)\) that a standard normal is below \(z\), giving a smooth curve that behaves like ReLU for large \(|z|\). It is the standard choice inside Transformers.
3.4 Softmax for multiclass outputs
For a classification with \(K\) classes the final layer outputs a vector \(z \in \mathbb{R}^K\) of scores, and the softmax turns it into a probability distribution over the classes:
\[\boxed{ \text{softmax}(z)_k = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}} }\]Each component lies in \((0, 1)\) and the components sum to \(1\), so \(\text{softmax}(z)_k\) reads as the predicted probability of class \(k\). The largest score becomes the most likely class.
Remark: softmax belongs at the output layer, not in a hidden layer. It couples every unit through the shared denominator (a normalization across the whole vector), which is exactly what a probability output needs but is not a useful per-unit hidden nonlinearity. For a single output (\(K = 1\) vs its complement) softmax reduces to the sigmoid. The pairing of softmax with its loss is the subject of Loss functions and output layers.
3.5 Choosing an activation
A useful default: use ReLU or GELU in the hidden layers, and choose the output activation from the task. The diagram and table below summarize the decision.
Choosing an activation: ReLU or GELU for hidden layers, and an output activation matched to the task.
| layer / task | recommended activation | reason |
|---|---|---|
| hidden (default) | ReLU or GELU | no positive-side saturation, cheap, trains fast |
| hidden (dead units) | leaky ReLU or ELU | keeps a nonzero gradient for \(z < 0\) |
| output, regression | identity (none) | prediction is an unbounded real value |
| output, binary | sigmoid | maps score to a probability in \((0, 1)\) |
| output, multiclass | softmax | maps scores to a distribution over classes |
Remark: sigmoid and tanh are now rarely used as hidden activations in deep feed-forward networks precisely because of the saturation in Section 3.2.3. They survive at the output (sigmoid) and inside gated recurrent units, where their bounded range is the point.
With the per-layer nonlinearities fixed, the next lesson pairs the output activation with a matching loss so the network has something to minimize.