2. Multilayer perceptron

A perceptron is one unit, \(a = g(w^T x + b)\). Stack many units that read the same input and you get a layer, stack layers and you get a multilayer perceptron (MLP). This module builds the MLP from units, writes forward propagation for one example and for a mini-batch, tracks the shapes and the parameter count, and states the universal approximation theorem.

Objectives

• Build a layer as a stack of perceptron-like units reading a shared input.
• Write forward propagation for one example with explicit per-layer weights and bias.
• Vectorize the forward pass over a mini-batch with broadcast bias.
• Track the shape of every \(W^{[l]}\) and \(b^{[l]}\) and count the parameters.
• State the universal approximation theorem and contrast width against depth.

2.1 From a unit to a layer

2.1.1 A single unit

A unit takes an input vector \(x \in \mathbb{R}^{n_0}\), forms a weighted sum with a weight vector \(w\) and a bias scalar \(b\), then applies a nonlinear activation \(g\):

This is the perceptron of the previous course, except the hard step is now a smooth activation such as the sigmoid or ReLU. The activation is named here and defined fully in the next lesson.

2.1.2 A layer of units

Now place \(n_1\) units side by side, all reading the same input \(x\). Unit \(i\) has its own weight vector \(w_i\) and bias \(b_i\), producing \(a_i = g(w_i^T x + b_i)\). Collect the weight vectors as the rows of a matrix \(W^{[1]}\) and the biases into a vector \(b^{[1]}\):

The whole layer then computes a pre-activation vector and an activation vector in one matrix expression, \(z^{[1]} = W^{[1]} x + b^{[1]}\) and \(a^{[1]} = g^{[1]}(z^{[1]})\), where \(g^{[1]}\) is applied elementwise.

Remark: the rows of \(W^{[1]}\) are exactly the individual unit weight vectors, so a layer is just many units packed into one matrix. Bias stays explicit here: unlike the Machine Learning course, which folded the intercept into \(w\) via the augmented input \(x_0 = 1\), this course keeps \(b^{[l]}\) as its own vector.

2.2 Forward propagation

Stacking \(L\) such layers gives the MLP. Layer \(l\) reads the activation of the layer below, \(a^{[l-1]}\), and produces \(a^{[l]}\). The input is \(a^{[0]} = x\) and the prediction is the output of the last layer.

2.2.1 One example

For \(l = 1, \dots, L\):

Each layer may use its own activation \(g^{[l]}\): hidden layers typically use ReLU, while the output layer uses sigmoid or softmax for classification and the identity for regression.

Remark: the composition \(\hat{y} = g^{[L]}(W^{[L]} g^{[L-1]}(\cdots g^{[1]}(W^{[1]} x + b^{[1]}) \cdots) + b^{[L]})\) is what makes the network expressive. Without the nonlinear \(g^{[l]}\) the whole stack would collapse to a single linear map \(W x + b\).

2.2.2 Vectorized over a mini-batch

Training runs on batches, not single examples. Place \(m\) examples as the columns of a matrix, so \(A^{[0]} = X \in \mathbb{R}^{n_0 \times m}\), and the forward pass becomes a matrix product with the bias broadcast across all columns:

Here \(Z^{[l]}\) and \(A^{[l]}\) have shape \(n_l \times m\), one column per example. The bias \(b^{[l]} \in \mathbb{R}^{n_l}\) is added to every column, an operation known as broadcasting.

Remark: the only change from the single-example form is that the vector \(a^{[l-1]}\) becomes the matrix \(A^{[l-1]}\). Processing a batch as one matrix multiply is what lets a GPU run the pass efficiently.

2.3 Shapes and parameter count

The shapes follow from one rule: to compute \(z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}\), the matrix \(W^{[l]}\) must map an \(n_{l-1}\)-vector to an \(n_l\)-vector.

Layer \(l\) therefore holds \(n_l \, n_{l-1}\) weights plus \(n_l\) biases. Consider a small network with \(n_0 = 4\) inputs, two hidden layers of \(5\) and \(3\) units, and a single output unit.

Remark: the input layer holds no parameters, it is just the data \(a^{[0]} = x\). When counting layers we count the layers that carry weights, so this network has \(L = 3\).

2.4 A multi-layer network

The diagram below shows the same \(4\)-\(5\)-\(3\)-\(1\) network as a flow of activations. Each arrow group is a full weight matrix, and each box applies its activation to the pre-activation.

A multilayer perceptron: each layer computes z = W a + b then a = g(z), composing input a0 into the prediction aL.

Information flows strictly left to right during the forward pass, which is why this is a feedforward network. Nothing loops back. The reverse direction, used to compute gradients, is the subject of a later lesson.

2.5 Universal approximation

How expressive is an MLP? The universal approximation theorem gives a strong answer. Let \(f\) be any continuous function on a compact set \(K \subset \mathbb{R}^{n_0}\), and let \(\varepsilon > 0\). Then there exists a network with a single hidden layer of finite width, using a suitable nonlinear activation, whose output \(F\) satisfies:

\[\boxed{ \sup_{x \in K} \left| F(x) - f(x) \right| < \varepsilon }\]

In words, one hidden layer with enough units can approximate any continuous function on a bounded region to any desired accuracy \(\varepsilon\). This is an existence result, not a recipe: it promises that such weights exist, but says nothing about how many units are needed or how to find them.

Remark: the catch is width. Matching a target to accuracy \(\varepsilon\) with one hidden layer can demand an enormous number of units, growing fast as \(\varepsilon\) shrinks. Depth is usually far more parameter-efficient: stacking several narrow layers can represent functions that a single layer would need exponentially many units to match. This efficiency of depth over width is the practical reason the field is called deep learning.

A network with one hidden layer approximates a target function by summing many simple activated units.

The network is only defined once the activations \(g^{[l]}\) are fixed. The next lesson defines them, sigmoid, tanh, ReLU and its variants, and explains how each shapes learning.

Next: Activation functions · Course overview