8. Multilayer neural networks
A single linear unit only draws a straight boundary. Stacking many simple units with a nonlinearity between them gives a multilayer neural network, which fits curved boundaries and learns its own features. This module is a compact tour of neural networks, from architecture to training, and the gateway to the Deep Learning course, which develops every topic here in depth.
Objectives
- Contrast the linear and nonlinear approaches and see why hidden layers are needed.
- Read a network as input, hidden, and output layers, and write its forward pass.
- Choose the output layer and loss for binary and multiclass classification.
- Pick an activation function and see why zero-centered outputs help.
- Train by the chain rule and backpropagation, with mini-batches, good initialization, and dropout.
- Guard the implementation with gradient checking and vectorization.
8.1 Linear versus nonlinear
The linear classifiers of the previous module separate classes with a single straight boundary, so a problem like XOR, which is not linearly separable, is out of reach. Composing units through a nonlinear activation \(g\) bends the boundary. The nonlinearity is essential: without it, a stack of linear layers collapses back to a single linear map,
\[\boxed{ W^{[2]}\!\left(W^{[1]} x + b^{[1]}\right) + b^{[2]} = W' x + b' }\]so depth would add nothing. The nonlinear activation is what makes stacking worthwhile.
8.2 Layers: input, hidden, output
A single neuron computes \(a = g(w^T x + b)\). A layer stacks many neurons, and a network stacks layers. Layer \(l\) transforms the previous activations into new ones:
\[\boxed{ z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}, \quad a^{[l]} = g^{[l]}\!\left(z^{[l]}\right), \quad a^{[0]} = x, \quad \hat{y} = a^{[L]} }\]The input layer holds \(x\), the hidden layers learn intermediate features, and the output layer produces the prediction \(\hat{y}\).
Each edge carries a weight in \(W^{[l]}\) and each unit adds a bias then applies the activation.
Remark: the bias is now written out explicitly and each layer has its own weight matrix \(W^{[l]}\), unlike the earlier convention of folding the bias into \(\theta^T x\) with \(x_0 = 1\). This is the notation the Deep Learning course uses throughout.
8.3 Output layer: binary and multiclass
The output layer matches the task, reusing the losses from the previous module. For two classes, a sigmoid output with the binary cross-entropy; for \(k\) classes, a softmax output with the categorical cross-entropy:
\[\boxed{ \hat{y} = \frac{1}{1 + e^{-z}} \quad\text{(binary)} \qquad \hat{y}_c = \frac{e^{z_c}}{\sum_{j} e^{z_j}} \quad\text{(multiclass)} }\]8.4 Activation functions and the zero-centered problem
The hidden activation is usually the sigmoid, the hyperbolic tangent, or the rectified linear unit:
\[\boxed{ \sigma(z) = \frac{1}{1 + e^{-z}}, \qquad \tanh(z), \qquad \mathrm{ReLU}(z) = \max(0, z) }\]The sigmoid saturates in its tails, and its outputs are never negative, so a unit's incoming weights all receive gradients of the same sign and the updates zig-zag. The zero-centered \(\tanh\) removes that bias, and ReLU avoids positive-side saturation altogether, which is why it is the common default.
The tanh is zero-centered while the sigmoid is not, and ReLU stays linear for positive inputs.
8.5 Chain rule and backpropagation
Training minimizes the loss by gradient descent, which needs its gradient with respect to every weight. Backpropagation computes all of them in one forward and one backward sweep: the forward pass caches each \(z^{[l]}\) and \(a^{[l]}\), then the backward pass applies the chain rule from the loss back to the first layer, reusing the cache. With the layer error \(\delta^{[l]} = \partial L / \partial z^{[l]}\),
\[\boxed{ \delta^{[l]} = \left((W^{[l+1]})^T \delta^{[l+1]}\right) \odot g'^{[l]}\!\left(z^{[l]}\right), \qquad \frac{\partial L}{\partial W^{[l]}} = \delta^{[l]} (a^{[l-1]})^T }\]
The Backpropagation lesson of the Deep Learning course derives this step by step.
8.6 Training in practice
- Mini-batches. Estimate the gradient on a small batch of examples at a time, a middle ground between the full batch (accurate but slow) and one example (noisy but cheap).
- Vanishing gradient. Through many saturating layers the backpropagated gradient is a product of small factors and shrinks toward zero, so early layers barely learn. ReLU activations and careful initialization keep it alive.
- Initialization. Start the weights small and random to break symmetry, scaling the variance by the number of inputs (Xavier or He), so signals neither vanish nor explode through depth.
- Dropout. Randomly zero a fraction of units during training. This prevents units from co-adapting and acts as a regularizer, in the spirit of the regularization module.
8.7 Sanity checks and vectorization
Backpropagation is error-prone, so check the analytic gradient against a numerical finite-difference estimate:
\[\boxed{ \frac{\partial L}{\partial \theta} \approx \frac{L(\theta + \varepsilon) - L(\theta - \varepsilon)}{2\varepsilon} }\]and implement the passes in vectorized form, one matrix operation per layer over the whole mini-batch (columns are examples), which is both clearer and far faster:
\[\boxed{ Z^{[l]} = W^{[l]} A^{[l-1]} + b^{[l]} }\]This module is the doorway to the Deep Learning course, which develops architectures, optimizers, initialization, normalization, and regularization in full. The next module returns to linear models from a new angle, the maximum-margin classifier.