A single linear unit only draws a straight boundary. Stacking many simple units with a nonlinearity between them gives a multilayer neural network, which fits curved boundaries and learns its own features. This module builds that model the gentle way: take the logistic regression of the previous module, draw it as a graph, and make it deep, one step at a time. The recipe is the one every module has used: a model (layers, run by forward propagation), a loss function matched to the task, and gradient descent, now powered by backpropagation. The story then continues the way practice forced it to: gradients vanish in deep stacks, better activations revive them, good practices make training behave, and gradient descent itself gets an upgrade. This module is the gateway to the [Deep Learning](/en/Deep%20Learning) course, which develops every topic here in depth.

The linear classifiers of the [linear classification module](/en/Machine%20Learning/05%20Linear%20classification) separate classes with a single straight boundary, so a problem like XOR, which is not linearly separable, is out of reach. Bending the boundary takes something nonlinear, and the whole question is where the nonlinearity comes from. [Linear regression](/en/Machine%20Learning/04%20Linear%20regression) answered it once already, with basis functions $\phi$ fixed by hand before training. Networks answer it differently: they learn the features themselves. The next section builds that machine out of a model already in hand.

Stack the three weight vectors $w_i^T$ as the rows of a matrix $W^{[1]}$ and the whole layer becomes one line, $a = g(W^{[1]} x)$. The bracketed superscript $[1]$ is new, and it exists for a mundane reason: the model now has two sets of weights, so each needs a name. $[l]$ simply says which layer a symbol belongs to.

The output neuron has not changed at all. It is still the logistic regression of section 6.2.1, it just reads the three learned values $a$, augmented with a constant $a_0 = 1$ (every layer gets a bias neuron, exactly like the input), instead of the raw input:

$$\boxed{ \hat{y} = \sigma\!\left(w^{[2]T} a\right) = \sigma\!\left(w^{[2]T}\, g(W^{[1]} x)\right) }$$


Two facts about this insertion carry the whole story.


Every symbol in the forward propagation formula lives somewhere on the network drawing. The figure below places each one on the smallest interesting network, two inputs, one hidden layer of two units, and one output:


*Remark:* these are exactly the neuron heads of [Linear classification](/en/Machine%20Learning/05%20Linear%20classification). A network is that same head with learned features underneath instead of raw inputs.


The sigmoid saturates in both tails, which is exactly what section 6.5 punished, and its outputs are never negative, so a unit's incoming weights all receive gradients of the same sign and the updates zig-zag. The zero-centered $\tanh$ removes that bias but still saturates. ReLU keeps a slope of exactly $1$ on its whole positive side, so the shrinking factors of section 6.5 disappear, and it costs almost nothing to compute. That is why it is the default hidden activation today.


ReLU has one blind spot: a unit whose input stays negative outputs $0$, has slope $0$, and stops learning, a dead unit. Variants such as Leaky ReLU, $\max(0.01 z, z)$, and ELU keep a small slope on the negative side to prevent it. In practice: start with ReLU, try its variants if units die, and keep the sigmoid only where section 6.3 needs it, at the output of a binary classifier. The [Activation functions](/en/Deep%20Learning/03%20Activation%20functions) lesson of the Deep Learning course compares them all.

Five habits make the difference between a network that trains and one that stalls.

**Train on mini-batches.** [Linear classification](/en/Machine%20Learning/05%20Linear%20classification) offered two extremes, the full batch or a single example per step. Networks train on mini-batches, a small batch per step: a gradient accurate enough to make progress, a step cheap enough to take thousands of them, and the vectorized forward propagation of section 6.2.3 processes the whole mini-batch in one matrix product per layer.

**Momentum** treats the update as a velocity with friction: gradients accumulate, persistent directions build up speed, oscillating ones cancel out:

$$\boxed{ v \leftarrow \rho\, v + \nabla_W L, \qquad W \leftarrow W - \alpha\, v }$$

with the friction $\rho$ typically around $0.9$.

**RMSProp** gives each parameter its own step size, dividing by a running average of the gradient's magnitude, so steep directions are tamed and flat ones sped up:

$$\boxed{ m \leftarrow \beta\, m + (1 - \beta) \left(\nabla_W L\right)^2, \qquad W \leftarrow W - \frac{\alpha}{\sqrt{m} + \varepsilon}\, \nabla_W L }$$

**Adam** combines the two ideas, a velocity for the direction and a per-parameter scale for the step (the full version also corrects a startup bias in $v$ and $m$), and is the default optimizer in practice:

$$\boxed{ v \leftarrow \beta_1 v + (1 - \beta_1)\, \nabla_W L, \qquad m \leftarrow \beta_2 m + (1 - \beta_2) \left(\nabla_W L\right)^2, \qquad W \leftarrow W - \alpha\, \frac{v}{\sqrt{m} + \varepsilon} }$$

Two habits complete the picture: decay the learning rate as training advances, and remember that all three methods still consume the mini-batch gradients of section 6.7, they only spend them more wisely. The [Optimization](/en/Deep%20Learning/06%20Optimization) lesson of the Deep Learning course derives each one and adds the learning-rate schedules.

Next: [Support Vector Machines](/en/Machine%20Learning/07%20Support%20Vector%20Machines) · [Course overview](/en/Machine%20Learning)