A single linear unit only draws a straight boundary. Stacking many simple units with a nonlinearity between them gives a multilayer neural network, which fits curved boundaries and learns its own features. This module is a compact tour of neural networks, from architecture to training, and the gateway to the [Deep Learning](/en/Deep%20Learning) course, which develops every topic here in depth.

- Contrast the linear and nonlinear approaches and see why hidden layers are needed.

- Read a network as input, hidden, and output layers, and write its forward pass.

- Choose the output layer and loss for binary and multiclass classification.

- Pick an activation function and see why zero-centered outputs help.

- Train by the chain rule and backpropagation, with mini-batches, good initialization, and dropout.

- Guard the implementation with gradient checking and vectorization.

The linear classifiers of the [previous module](/en/Machine%20Learning/06%20Linear%20classification) separate classes with a single straight boundary, so a problem like XOR, which is not linearly separable, is out of reach. Composing units through a nonlinear activation $g$ bends the boundary. The nonlinearity is essential: without it, a stack of linear layers collapses back to a single linear map,

$$\boxed{ W^{[2]}\!\left(W^{[1]} x + b^{[1]}\right) + b^{[2]} = W' x + b' }$$

so depth would add nothing. The nonlinear activation is what makes stacking worthwhile.

A single neuron computes $a = g(w^T x + b)$. A layer stacks many neurons, and a network stacks layers. Layer $l$ transforms the previous activations into new ones:

$$\boxed{ z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}, \quad a^{[l]} = g^{[l]}\!\left(z^{[l]}\right), \quad a^{[0]} = x, \quad \hat{y} = a^{[L]} }$$

The input layer holds $x$, the hidden layers learn intermediate features, and the output layer produces the prediction $\hat{y}$.


*Remark:* the bias is now written out explicitly and each layer has its own weight matrix $W^{[l]}$, unlike the earlier convention of folding the bias into $\theta^T x$ with $x_0 = 1$. This is the notation the Deep Learning course uses throughout.

The output layer matches the task, reusing the losses from the previous module. For two classes, a sigmoid output with the binary cross-entropy; for $k$ classes, a softmax output with the categorical cross-entropy:

$$\boxed{ \hat{y} = \frac{1}{1 + e^{-z}} \quad\text{(binary)} \qquad \hat{y}_c = \frac{e^{z_c}}{\sum_{j} e^{z_j}} \quad\text{(multiclass)} }$$

The hidden activation is usually the sigmoid, the hyperbolic tangent, or the rectified linear unit:

$$\boxed{ \sigma(z) = \frac{1}{1 + e^{-z}}, \qquad \tanh(z), \qquad \mathrm{ReLU}(z) = \max(0, z) }$$

The sigmoid saturates in its tails, and its outputs are never negative, so a unit's incoming weights all receive gradients of the same sign and the updates zig-zag. The zero-centered $\tanh$ removes that bias, and ReLU avoids positive-side saturation altogether, which is why it is the common default.


Training minimizes the loss by gradient descent, which needs its gradient with respect to every weight. Backpropagation computes all of them in one forward and one backward sweep: the forward pass caches each $z^{[l]}$ and $a^{[l]}$, then the backward pass applies the chain rule from the loss back to the first layer, reusing the cache. With the layer error $\delta^{[l]} = \partial L / \partial z^{[l]}$,

$$\boxed{ \delta^{[l]} = \left((W^{[l+1]})^T \delta^{[l+1]}\right) \odot g'^{[l]}\!\left(z^{[l]}\right), \qquad \frac{\partial L}{\partial W^{[l]}} = \delta^{[l]} (a^{[l-1]})^T }$$


- **Mini-batches.** Estimate the gradient on a small batch of examples at a time, a middle ground between the full batch (accurate but slow) and one example (noisy but cheap).

- **Vanishing gradient.** Through many saturating layers the backpropagated gradient is a product of small factors and shrinks toward zero, so early layers barely learn. ReLU activations and careful initialization keep it alive.

- **Initialization.** Start the weights small and random to break symmetry, scaling the variance by the number of inputs (Xavier or He), so signals neither vanish nor explode through depth.

- **Dropout.** Randomly zero a fraction of units during training. This prevents units from co-adapting and acts as a regularizer, one of the topics of the [next module](/en/Machine%20Learning/08%20Regularization%20and%20high-dimensional%20inference).

Backpropagation is error-prone, so check the analytic gradient against a numerical finite-difference estimate:

$$\boxed{ \frac{\partial L}{\partial \theta} \approx \frac{L(\theta + \varepsilon) - L(\theta - \varepsilon)}{2\varepsilon} }$$

and implement the passes in vectorized form, one matrix operation per layer over the whole mini-batch (columns are examples), which is both clearer and far faster:

$$\boxed{ Z^{[l]} = W^{[l]} A^{[l-1]} + b^{[l]} }$$

---

Next: [Regularization and high-dimensional inference](/en/Machine%20Learning/08%20Regularization%20and%20high-dimensional%20inference) · [Course overview](/en/Machine%20Learning)