A perceptron is one unit, $a = g(w^T x + b)$. Stack many units that read the same input and you get a layer, stack layers and you get a multilayer perceptron (MLP). This module builds the MLP from units, writes forward propagation for one example and for a mini-batch, tracks the shapes and the parameter count, and states the universal approximation theorem.

- Build a layer as a stack of perceptron-like units reading a shared input.

- Write forward propagation for one example with explicit per-layer weights and bias.

- Vectorize the forward pass over a mini-batch with broadcast bias.

- Track the shape of every $W^{[l]}$ and $b^{[l]}$ and count the parameters.

- State the universal approximation theorem and contrast width against depth.

A unit takes an input vector $x \in \mathbb{R}^{n_0}$, forms a weighted sum with a weight vector $w$ and a bias scalar $b$, then applies a nonlinear activation $g$:

$$\boxed{ a = g\left(w^T x + b\right) }$$

This is the perceptron of the previous course, except the hard step is now a smooth activation such as the sigmoid or ReLU. The activation is named here and defined fully in the [next lesson](/en/Deep%20Learning/03%20Activation%20functions).

Now place $n_1$ units side by side, all reading the same input $x$. Unit $i$ has its own weight vector $w_i$ and bias $b_i$, producing $a_i = g(w_i^T x + b_i)$. Collect the weight vectors as the rows of a matrix $W^{[1]}$ and the biases into a vector $b^{[1]}$:

$$\boxed{ W^{[1]} = \begin{bmatrix} w_1^{T} \\ \vdots \\ w_{n_1}^{T} \end{bmatrix}, \quad b^{[1]} = \begin{bmatrix} b_1 \\ \vdots \\ b_{n_1} \end{bmatrix} }$$

The whole layer then computes a pre-activation vector and an activation vector in one matrix expression, $z^{[1]} = W^{[1]} x + b^{[1]}$ and $a^{[1]} = g^{[1]}(z^{[1]})$, where $g^{[1]}$ is applied elementwise.

Stacking $L$ such layers gives the MLP. Layer $l$ reads the activation of the layer below, $a^{[l-1]}$, and produces $a^{[l]}$. The input is $a^{[0]} = x$ and the prediction is the output of the last layer.

For $l = 1, \dots, L$:

$$\boxed{ a^{[0]} = x, \quad z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}, \quad a^{[l]} = g^{[l]}\!\left(z^{[l]}\right), \quad \hat{y} = a^{[L]} }$$

Each layer may use its own activation $g^{[l]}$: hidden layers typically use ReLU, while the output layer uses sigmoid or softmax for classification and the identity for regression.

*Remark:* the composition $\hat{y} = g^{[L]}(W^{[L]} g^{[L-1]}(\cdots g^{[1]}(W^{[1]} x + b^{[1]}) \cdots) + b^{[L]})$ is what makes the network expressive. Without the nonlinear $g^{[l]}$ the whole stack would collapse to a single linear map $W x + b$.

Training runs on batches, not single examples. Place $m$ examples as the columns of a matrix, so $A^{[0]} = X \in \mathbb{R}^{n_0 \times m}$, and the forward pass becomes a matrix product with the bias broadcast across all columns:

$$\boxed{ Z^{[l]} = W^{[l]} A^{[l-1]} + b^{[l]}, \quad A^{[l]} = g^{[l]}\!\left(Z^{[l]}\right) }$$

Here $Z^{[l]}$ and $A^{[l]}$ have shape $n_l \times m$, one column per example. The bias $b^{[l]} \in \mathbb{R}^{n_l}$ is added to every column, an operation known as broadcasting.

*Remark:* the only change from the single-example form is that the vector $a^{[l-1]}$ becomes the matrix $A^{[l-1]}$. Processing a batch as one matrix multiply is what lets a GPU run the pass efficiently.

The shapes follow from one rule: to compute $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$, the matrix $W^{[l]}$ must map an $n_{l-1}$-vector to an $n_l$-vector.

$$\boxed{ W^{[l]} \in \mathbb{R}^{n_l \times n_{l-1}}, \quad b^{[l]} \in \mathbb{R}^{n_l} }$$

Layer $l$ therefore holds $n_l \, n_{l-1}$ weights plus $n_l$ biases. Consider a small network with $n_0 = 4$ inputs, two hidden layers of $5$ and $3$ units, and a single output unit.

| Layer $l$ | $W^{[l]}$ shape | $b^{[l]}$ shape | Parameters |

| --- | --- | --- | --- |

| 1 | $5 \times 4$ | $5$ | $25$ |

| 2 | $3 \times 5$ | $3$ | $18$ |

| 3 | $1 \times 3$ | $1$ | $4$ |

| Total | | | $47$ |

*Remark:* the input layer holds no parameters, it is just the data $a^{[0]} = x$. When counting layers we count the layers that carry weights, so this network has $L = 3$.

The diagram below shows the same $4$-$5$-$3$-$1$ network as a flow of activations. Each arrow group is a full weight matrix, and each box applies its activation to the pre-activation.


Information flows strictly left to right during the forward pass, which is why this is a feedforward network. Nothing loops back. The reverse direction, used to compute gradients, is the subject of a later lesson.

How expressive is an MLP? The universal approximation theorem gives a strong answer. Let $f$ be any continuous function on a compact set $K \subset \mathbb{R}^{n_0}$, and let $\varepsilon > 0$. Then there exists a network with a single hidden layer of finite width, using a suitable nonlinear activation, whose output $F$ satisfies:

$$\boxed{ \sup_{x \in K} \left| F(x) - f(x) \right| < \varepsilon }$$

In words, one hidden layer with enough units can approximate any continuous function on a bounded region to any desired accuracy $\varepsilon$. This is an existence result, not a recipe: it promises that such weights exist, but says nothing about how many units are needed or how to find them.

*Remark:* the catch is width. Matching a target to accuracy $\varepsilon$ with one hidden layer can demand an enormous number of units, growing fast as $\varepsilon$ shrinks. Depth is usually far more parameter-efficient: stacking several narrow layers can represent functions that a single layer would need exponentially many units to match. This efficiency of depth over width is the practical reason the field is called deep learning.


---

Next: [Activation functions](/en/Deep%20Learning/03%20Activation%20functions) · [Course overview](/en/Deep%20Learning)