Each layer computes a pre-activation $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$ and then an activation $a^{[l]} = g^{[l]}(z^{[l]})$. The choice of the nonlinearity $g^{[l]}$ is what makes depth worthwhile. This lesson explains why a nonlinear $g$ is required, surveys the sigmoid, tanh, and ReLU families, introduces the softmax used at the output, and gives practical guidance on which activation to pick.

Suppose every activation were the identity, $g^{[l]}(z) = z$. Then each layer is just $a^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$, and composing two of them gives $W^{[2]}(W^{[1]} x + b^{[1]}) + b^{[2]} = (W^{[2]} W^{[1]}) x + (W^{[2]} b^{[1]} + b^{[2]})$. That is again of the form $W x + b$. By induction the whole $L$-layer network reduces to a single affine map:

$$\boxed{ g^{[l]} = \text{identity} \;\Rightarrow\; \hat{y} = W' x + b' }$$

with $W' = W^{[L]} \cdots W^{[1]}$ and $b'$ the accumulated bias. No matter how many linear layers are stacked, the model can only fit a linear function, so the extra depth buys nothing. A nonlinear $g$ between layers is exactly what breaks this collapse and lets the network represent curved decision boundaries and nonlinear regressions.

The sigmoid squashes any real pre-activation into the open interval $(0, 1)$:

$$\boxed{ \sigma(z) = \frac{1}{1 + e^{-z}} \in (0, 1) }$$

Its derivative has the convenient closed form below, which reuses the forward value $\sigma(z)$ already computed:

$$\boxed{ \sigma'(z) = \sigma(z)\left(1 - \sigma(z)\right) }$$

The hyperbolic tangent is a rescaled sigmoid centred at zero, with output in $(-1, 1)$. Its derivative is likewise expressible from the forward value:

$$\boxed{ \tanh'(z) = 1 - \tanh(z)^2 }$$

*Remark:* $\tanh$ is zero-centred while $\sigma$ is not, so $\tanh$ often trains a little better as a hidden activation. The two are related by $\tanh(z) = 2\sigma(2z) - 1$.

The rectified linear unit keeps the positive part of its input and zeroes the rest:

$$\boxed{ \text{ReLU}(z) = \max(0, z) }$$

Its derivative is $1$ for $z > 0$ and $0$ for $z < 0$ (undefined at $z = 0$, taken to be $0$ or $1$ by convention). ReLU does not saturate on the positive side, so it keeps a healthy gradient flowing there, which is a large part of why it became the default hidden activation. The cost is the dead-unit problem: if a unit's pre-activation is always negative across the data, its gradient is always zero and it stops learning entirely. The variants below trade a little simplicity to soften that failure or to smooth the kink at the origin.

| name | formula | derivative | dies / saturates? |

| --- | --- | --- | --- |

| ReLU | $\max(0, z)$ | $1$ if $z>0$ else $0$ | can die (zero gradient for $z<0$) |

| Leaky ReLU | $\max(\alpha z, z)$, $\alpha \approx 0.01$ | $1$ if $z>0$ else $\alpha$ | rarely dies (small negative slope) |

| PReLU | $\max(\alpha z, z)$, $\alpha$ learned | $1$ if $z>0$ else $\alpha$ | rarely dies ($\alpha$ trained per channel) |

| ELU | $z$ if $z>0$ else $\alpha(e^z - 1)$ | $1$ if $z>0$ else $\alpha e^z$ | saturates gently for $z\to-\infty$ |

| GELU | $z\,\Phi(z)$, $\Phi$ the normal CDF | smooth, near $1$ for large $z$ | smooth, no hard death |

*Remark:* leaky ReLU and PReLU add a small slope $\alpha$ on the negative side so a unit is never fully switched off. GELU weights the input by the probability $\Phi(z)$ that a standard normal is below $z$, giving a smooth curve that behaves like ReLU for large $|z|$. It is the standard choice inside Transformers.

For a classification with $K$ classes the final layer outputs a vector $z \in \mathbb{R}^K$ of scores, and the softmax turns it into a probability distribution over the classes:

$$\boxed{ \text{softmax}(z)_k = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}} }$$

Each component lies in $(0, 1)$ and the components sum to $1$, so $\text{softmax}(z)_k$ reads as the predicted probability of class $k$. The largest score becomes the most likely class.

A useful default: use ReLU or GELU in the hidden layers, and choose the output activation from the task. The diagram and table below summarize the decision.

| layer / task | recommended activation | reason |

| --- | --- | --- |

| hidden (default) | ReLU or GELU | no positive-side saturation, cheap, trains fast |

| hidden (dead units) | leaky ReLU or ELU | keeps a nonzero gradient for $z < 0$ |

| output, regression | identity (none) | prediction is an unbounded real value |

| output, binary | sigmoid | maps score to a probability in $(0, 1)$ |

| output, multiclass | softmax | maps scores to a distribution over classes |

*Remark:* sigmoid and tanh are now rarely used as hidden activations in deep feed-forward networks precisely because of the saturation in Section 3.2.3. They survive at the output (sigmoid) and inside gated recurrent units, where their bounded range is the point.

---