# 6. Multilayer neural networks A single linear unit only draws a straight boundary. Stacking many simple units with a nonlinearity between them gives a multilayer neural network, which fits curved boundaries and learns its own features. This module builds that model the gentle way: take the logistic regression of the previous module, draw it as a graph, and make it deep, one step at a time. The recipe is the one every module has used: a model (layers, run by forward propagation), a loss function matched to the task, and gradient descent, now powered by backpropagation. The story then continues in the [Deep Learning](/en/Deep%20Learning) course, which picks up exactly here: why gradients vanish in deep stacks, the activations that revive them, the good practices that make training behave, and the upgrades to gradient descent. ## 6.1 Linear versus nonlinear The linear classifiers of the [linear classification module](/en/Machine%20Learning/05%20Linear%20classification) separate classes with a single straight boundary, so a problem like XOR, which is not linearly separable, is out of reach. Bending the boundary takes something nonlinear, and the whole question is where the nonlinearity comes from. [Linear regression](/en/Machine%20Learning/04%20Linear%20regression) answered it once already, with basis functions $\phi$ fixed by hand before training. Networks answer it differently: they learn the features themselves. The next section builds that machine out of a model already in hand. ## 6.2 Make logistic regression deep ### 6.2.1 Logistic regression as a network [Linear classification](/en/Machine%20Learning/05%20Linear%20classification) ended with logistic regression: a dot product with the weights $w$ and a sigmoid that squashes the score into a probability, $$\boxed{ \hat{y} = \sigma(w^T x) }$$ with the course's usual convention that $x$ is augmented with a constant $x_0 = 1$, so the weight $w_0$ is the bias. Drawn as a graph, this is already a network, the smallest possible one. The input layer holds $x$, its constant included, and computes nothing. A single output neuron does all the work: multiply by the weights, apply the activation. Every neuron in this module is exactly this unit.  *Each edge carries one weight and the neuron applies $\sigma$ to the weighted sum. The neuron fixed at $1$ carries the bias: its weight is $w_0$.* *Remark:* the bias stays folded into the weights throughout this module, drawn as a constant neuron. The [Deep Learning](/en/Deep%20Learning) course instead keeps an explicit bias vector $b^{[l]}$, and flags the change of convention when it introduces it. ### 6.2.2 Insert a hidden layer Nothing forces the output neuron to read the raw input. Insert a few neurons between the input and the output, say three. Each one is the same dot-product unit as always, with its own weights $w_i$ and a nonlinear activation $g$: $$a_i = g(w_i^T x), \qquad i = 1, 2, 3$$ Stack the three weight vectors $w_i^T$ as the rows of a matrix $W^{[1]}$ and the whole layer becomes one line, $a = g(W^{[1]} x)$. The bracketed superscript $[1]$ is new, and it exists for a mundane reason: the model now has two sets of weights, so each needs a name. $[l]$ simply says which layer a symbol belongs to. The output neuron has not changed at all. It is still the logistic regression of section 6.2.1, it just reads the three learned values $a$, augmented with a constant $a_0 = 1$ (every layer gets a bias neuron, exactly like the input), instead of the raw input: $$\boxed{ \hat{y} = \sigma\!\left(w^{[2]T} a\right) = \sigma\!\left(w^{[2]T}\, g(W^{[1]} x)\right) }$$  *The orange output neuron is identical in both drawings. Making the model deep changed what it reads: three learned features $a$ instead of the raw $x$. Each layer carries its own constant neuron $1$, whose outgoing weights are the biases.* Two facts about this insertion carry the whole story. **The hidden activation must be nonlinear.** If $g$ were the identity, the two layers would collapse into a single linear map, $$\boxed{ W^{[2]}\!\left(W^{[1]} x\right) = W' x }$$ and depth would add nothing. The nonlinearity is what makes stacking worthwhile, and it is why XOR is now within reach. **The hidden layer learns the features.** The output neuron is still a linear classifier, so the hidden layer's job is to move the data somewhere the classes become linearly separable. It plays exactly the role of the basis functions $\phi$ of [Linear regression](/en/Machine%20Learning/04%20Linear%20regression), with one upgrade: $\phi$ was fixed by hand before training, while $a$ is learned from the data, end to end. ### 6.2.3 How to make a prediction? One hidden layer worked, so repeat the move: the vector $a^{[1]}$ can feed a second hidden layer, whose output $a^{[2]}$ can feed a third, until an output layer produces $\hat{y}$. Layer $l$ owns its weight matrix $W^{[l]}$ (one row per neuron, so the output layer above had the single row $w^{[2]T}$) and its activation $g^{[l]}$. Width and depth are the capacity dials: more units and more layers mean more parameters and more expressive boundaries, and, as [General concepts](/en/Machine%20Learning/02%20General%20concepts) warned, more room to overfit.  *Each edge carries a weight in $W^{[l]}$. The constant bias neurons are left out of the drawing.* Computing the prediction by reading the network left to right is called forward propagation, and the general formula only restates what the last two sections built, once per layer: $$\boxed{ z^{[l]} = W^{[l]} a^{[l-1]}, \quad a^{[l]} = g^{[l]}\!\left(z^{[l]}\right), \quad a^{[0]} = x, \quad \hat{y} = a^{[L]} }$$ with one convention to remember: every $a^{[l]}$, like the input, is read with its bias neuron $a^{[l]}_0 = 1$ prepended. In vectorized form the whole mini-batch flows at once, one matrix operation per layer with the examples as columns, which is both clearer and far faster: $$\boxed{ Z^{[l]} = W^{[l]} A^{[l-1]} }$$ Forward propagation is the first half of every training step. The second half, backpropagation, runs the same wiring in reverse (section 6.4). ### 6.2.4 The formula on the graph Every symbol in the forward propagation formula lives somewhere on the network drawing. The figure below places each one on the smallest interesting network, two inputs, one hidden layer of two units, and one output:  *Left: the superscript $[l]$ names the layer, each edge carries one weight $w^{[l]}_{ij}$, and the neurons fixed at $1$ carry the biases $w^{[l]}_{i0}$. Right: inside a unit, the weighted sum gives $z^{[l]}_i$, then the activation $g$ turns it into $a^{[l]}_i$.* | Symbol | Name | Where it lives on the graph | | --- | --- | --- | | $l$, $L$ | layer index, number of layers | which column of units ($l = 0$ is the input, here $L = 2$) | | $x = a^{[0]}$ | the input | the leftmost column | | $w^{[l]}_{ij}$ | one weight | the number carried by one edge: into unit $i$ of layer $l$, from unit $j$ of layer $l-1$ | | $W^{[l]}$ | weight matrix of layer $l$ | all the edges arriving into layer $l$, one row per unit | | $x_0$, $a^{[l]}_0$ | bias neuron | a unit fixed at $1$ in each layer, whose outgoing weight $w^{[l]}_{i0}$ is the bias of unit $i$ | | $z^{[l]}_i$ | pre-activation | the weighted sum the unit computes before applying $g$ | | $g^{[l]}$ | activation function | applied inside every unit of layer $l$ | | $a^{[l]}_i$ | activation | the value the unit sends along its outgoing edges | | $\hat{y} = a^{[L]}$ | the prediction | what leaves the last layer | Now run this exact network with numbers. Take $x = (1, 2)$, augmented to $(1, 1, 2)$ by the bias neuron $x_0 = 1$, the sigmoid of the previous module as the activation everywhere, with $$W^{[1]} = \begin{pmatrix} 0 & 2 & -1 \\ -1 & 1 & 1 \end{pmatrix}, \qquad W^{[2]} = \begin{pmatrix} 0 & 1 & 1 \end{pmatrix}$$ Row $i$ of $W^{[1]}$ collects the weights of the edges arriving into hidden unit $i$, bias first. Layer 1, unit by unit: $$z^{[1]}_1 = \underbrace{0}_{w^{[1]}_{10}} \cdot \underbrace{1}_{x_0} + \underbrace{2}_{w^{[1]}_{11}} \cdot \underbrace{1}_{x_1} + \underbrace{(-1)}_{w^{[1]}_{12}} \cdot \underbrace{2}_{x_2} = 0, \qquad a^{[1]}_1 = \sigma(0) = 0.5$$ $$z^{[1]}_2 = -1 \cdot 1 + 1 \cdot 1 + 1 \cdot 2 = 2, \qquad a^{[1]}_2 = \sigma(2) \approx 0.88$$ Hidden unit 1 lands exactly on zero, the midpoint of the sigmoid, so it outputs $0.5$, while unit 2 sits high on the curve. The output layer repeats the same two steps, now reading $a^{[1]} = (0.5, 0.88)$, augmented to $(1, 0.5, 0.88)$ by its own bias neuron, instead of $x$: $$z^{[2]} = 0 \cdot 1 + 1 \cdot 0.5 + 1 \cdot 0.88 = 1.38, \qquad \hat{y} = a^{[2]} = \sigma(1.38) \approx 0.80$$ The network predicts class 1 with probability about $0.80$. That is all forward propagation does: multiply by the edge weights, bias neuron included, apply the activation, at every unit of every layer. ## 6.3 The loss function The network's body is task-agnostic. The task lives in the last layer: its activation shapes $\hat{y}$, and the loss compares $\hat{y}$ to the label, reusing the losses of [General concepts](/en/Machine%20Learning/02%20General%20concepts) and [Linear classification](/en/Machine%20Learning/05%20Linear%20classification): | Task | Output activation | Loss function | | --- | --- | --- | | regression | identity | squared error | | binary classification | sigmoid | binary cross-entropy | | multiclass classification | softmax | categorical cross-entropy | $$\boxed{ \hat{y} = \frac{1}{1 + e^{-z}} \quad\text{(binary)} \qquad \hat{y}_c = \frac{e^{z_c}}{\sum_{j} e^{z_j}} \quad\text{(multiclass)} }$$ *Remark:* these are exactly the neuron heads of [Linear classification](/en/Machine%20Learning/05%20Linear%20classification). A network is that same head with learned features underneath instead of raw inputs. ## 6.4 How to optimize the parameters? Nothing here is new either: training a network follows the same steps as every model of this course, so let us walk them in order. **Step 0: pose the objective.** Find the weights that minimize the loss plus a regularizer that keeps them small, the maximum a posteriori recipe of [Linear regression](/en/Machine%20Learning/04%20Linear%20regression): $$\boxed{ W^\star = \arg\min_W \; L(W) + \lambda\, R(W), \qquad R(W) = \lVert W \rVert_1 \;\text{ or }\; \lVert W \rVert_2^2 }$$ **Step 1: choose the loss.** That is section 6.3, and its table carries the warning that comes with it: the loss and the output activation are picked as a pair. Cross-entropy calls for a softmax (or a sigmoid), squared error for an identity output. **Step 2: descend the gradient.** Update every weight by a small step against its gradient, with learning rate $\alpha$, exactly the gradient descent of [Linear classification](/en/Machine%20Learning/05%20Linear%20classification): $$\boxed{ w^{[l]}_{ij} \leftarrow w^{[l]}_{ij} - \alpha\, \frac{\partial \left(L + \lambda R\right)}{\partial w^{[l]}_{ij}} }$$ **Step 3: get the gradient by backpropagation.** The genuinely new piece is computing that gradient for every weight in a stack of layers. Backpropagation does it with the chain rule, in one forward and one backward sweep: forward propagation caches each $z^{[l]}$ and $a^{[l]}$, then the backward pass propagates the loss gradient layer by layer, from the output back to the first layer, reusing the cache. With the layer error $\delta^{[l]} = \partial L / \partial z^{[l]}$, $$\boxed{ \delta^{[l]} = \left((W^{[l+1]})^T \delta^{[l+1]}\right) \odot g'^{[l]}\!\left(z^{[l]}\right), \qquad \frac{\partial L}{\partial W^{[l]}} = \delta^{[l]} (a^{[l-1]})^T }$$ The bias neurons fit for free: being constant, they receive no error (their row of $(W^{[l+1]})^T \delta^{[l+1]}$ is simply dropped), and since $a^{[l-1]}$ includes the constant $1$, the same outer product delivers the bias gradients along with the rest.  *Forward propagation computes and caches the activations, backpropagation sends the loss gradient back through the same edges.* One practical point completes the recipe: the gradient of step 2 is computed on a mini-batch. [Linear classification](/en/Machine%20Learning/05%20Linear%20classification) offered two extremes, the full batch or a single example per step. Networks train in between, a small batch per step: a gradient accurate enough to make progress, a step cheap enough to take thousands of them, and the vectorized forward propagation of section 6.2.3 processes the whole mini-batch in one matrix product per layer. <details class="proof"> <summary>Full example: one gradient descent step on the tiny network</summary> Pick up the network of section 6.2.4 exactly where forward propagation left it: $\bar{x} = (1, 1, 2)$, $a^{[1]} = (0.5,\ 0.88)$, $\hat{y} = 0.80$, and give the example a label: the true class is $y = 0$. Take the binary cross-entropy of section 6.3 with no regularization ($\lambda = 0$), so the loss is $$L = -\ln(1 - \hat{y}) = -\ln(0.20) \approx 1.61$$ The network is confidently wrong, and the gradient is about to say so. **Backward through the output layer.** For a sigmoid output trained with cross-entropy, the output error collapses to the familiar $\hat{y} - y$ of [Linear classification](/en/Machine%20Learning/05%20Linear%20classification): $$\delta^{[2]} = \hat{y} - y = 0.80$$ Each weight of $W^{[2]}$ receives $\delta^{[2]}$ times the activation it reads (the outer-product formula, bias neuron included): $$\frac{\partial L}{\partial W^{[2]}} = \delta^{[2]} \left(\bar{a}^{[1]}\right)^T = 0.80 \cdot (1,\ 0.5,\ 0.88) = (0.80,\ 0.40,\ 0.70)$$ **Backward through the hidden layer.** Each hidden unit takes its share of the error through its outgoing weight, times its own slope $\sigma'(z) = \sigma(z)(1 - \sigma(z))$: $$\delta^{[1]}_1 = w^{[2]}_{11}\, \delta^{[2]}\, \sigma'(0) = 1 \cdot 0.80 \cdot 0.25 = 0.20, \qquad \delta^{[1]}_2 = 1 \cdot 0.80 \cdot 0.10 = 0.08$$ (the constant bias neuron takes no error, and note the small slope $0.10$ of unit 2: the [Deep Learning](/en/Deep%20Learning) course makes a story of it). Then the same outer product against $\bar{x} = (1, 1, 2)$: $$\frac{\partial L}{\partial W^{[1]}} = \delta^{[1]}\, \bar{x}^T = \begin{pmatrix} 0.20 & 0.20 & 0.40 \\ 0.08 & 0.08 & 0.16 \end{pmatrix}$$ **The update.** Step 2 with a deliberately large $\alpha = 1$, so the movement is visible: $$W^{[2]} \leftarrow (0,\ 1,\ 1) - (0.80,\ 0.40,\ 0.70) = (-0.80,\ 0.60,\ 0.30)$$ $$W^{[1]} \leftarrow \begin{pmatrix} 0 & 2 & -1 \\ -1 & 1 & 1 \end{pmatrix} - \begin{pmatrix} 0.20 & 0.20 & 0.40 \\ 0.08 & 0.08 & 0.16 \end{pmatrix} = \begin{pmatrix} -0.20 & 1.80 & -1.40 \\ -1.08 & 0.92 & 0.84 \end{pmatrix}$$ **Did it help?** Run forward propagation once more with the new weights: $z^{[1]} = (-1.20,\ 1.52)$, $a^{[1]} = (0.23,\ 0.82)$, $z^{[2]} = -0.42$, and $$\hat{y} = \sigma(-0.42) \approx 0.40, \qquad L = -\ln(1 - 0.40) \approx 0.51$$ One step, and the prediction for class 1 fell from $0.80$ to $0.40$, the loss from $1.61$ to $0.51$. Training is this loop, repeated over mini-batches until the loss settles. </details> *This module is the doorway to the [Deep Learning](/en/Deep%20Learning) course, which picks the story up exactly here: the activation functions, the optimizers, initialization, normalization, regularization, and the architectures built on them. The next module changes tools entirely: decision trees, and the ensembles built from them.* --- Next: [Decision trees and ensemble methods](/en/Machine%20Learning/07%20Decision%20trees%20and%20ensemble%20methods) · [Course overview](/en/Machine%20Learning)