Backpropagation is the algorithm that computes the gradient of the cost with respect to every parameter of a network. It is nothing more than the chain rule applied in a careful, reverse order over the computational graph, reusing the quantities cached during the forward pass. This module derives it layer by layer using the error signal $\delta^{[l]} = \partial L / \partial z^{[l]}$.

- Read a network as a composition of functions and see why gradients flow backward by the chain rule.

- Define the layer error $\delta^{[l]}$ and compute the output-layer error $\delta^{[L]}$.

- Establish the backward recursion that carries $\delta$ from layer $L$ down to layer $1$.

- Turn each $\delta^{[l]}$ into the parameter gradients for $W^{[l]}$ and $b^{[l]}$.

- Assemble the full forward-and-backward algorithm and connect it to the parameter update.

A feedforward network is a composition of functions. Each layer $l$ takes the previous activation $a^{[l-1]}$ and produces a pre-activation and an activation:

$$\boxed{ z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}, \quad a^{[l]} = g^{[l]}\!\left(z^{[l]}\right) }$$

with $a^{[0]} = x$ and prediction $\hat{y} = a^{[L]}$. The scalar loss $L$ sits at the end of this chain. Because the cost is a composition, its derivative with respect to any intermediate quantity is a product of local derivatives, one per link in the graph. The chain rule tells us to accumulate those products.

The efficient way to do this is to walk the graph backward. A single reverse traversal computes, for every node, the derivative of the final loss with respect to that node, and each step reuses the derivative already computed for the node just downstream. This reuse is what makes backpropagation cost about the same as one forward pass, rather than one pass per parameter.


*Remark:* the solid arrows are the forward pass (data flowing to the loss) and the dashed arrows are the backward pass (gradients flowing from the loss). The two passes traverse the same graph in opposite directions.

The central object is the error of layer $l$, the sensitivity of the loss to the pre-activation $z^{[l]}$:

$$\boxed{ \delta^{[l]} = \frac{\partial L}{\partial z^{[l]}} \in \mathbb{R}^{n_l} }$$

Once we know $\delta^{[l]}$ at every layer, all parameter gradients follow immediately (Section 5.5). The whole algorithm reduces to computing these vectors, first at the output layer, then recursively backward.

*Remark:* placing $\delta$ at the pre-activation $z^{[l]}$ rather than at the activation $a^{[l]}$ is a deliberate choice. It makes the activation derivative $g'^{[l]}$ appear exactly once per layer and keeps the recursion clean.

At the output layer the chain rule has two links: the loss depends on $a^{[L]} = \hat{y}$, and $a^{[L]}$ depends on $z^{[L]}$ through the activation $g^{[L]}$. Multiplying the two local derivatives elementwise gives the output error:

$$\boxed{ \delta^{[L]} = \nabla_{a^{[L]}} L \;\odot\; g'^{[L]}\!\left(z^{[L]}\right) }$$

The Hadamard product $\odot$ appears because $g^{[L]}$ acts elementwise, so component $j$ of $z^{[L]}$ influences only component $j$ of $a^{[L]}$.

For multiclass classification the natural pairing is a softmax output with the cross-entropy loss (introduced in [Loss functions and output layers](/en/Deep%20Learning/04%20Loss%20functions%20and%20output%20layers)). The two derivatives combine and cancel, leaving a strikingly simple result:

$$\boxed{ \delta^{[L]} = \hat{y} - y }$$

*Remark:* the same clean form appears for a sigmoid output with binary cross-entropy, and for a linear output with squared error. In each case the output activation is the matched inverse link of the loss, so the messy factors cancel and the error is just the residual $\hat{y} - y$.

Given the error at layer $l+1$, we obtain the error at layer $l$. The loss depends on $z^{[l]}$ only through $z^{[l+1]} = W^{[l+1]} a^{[l]} + b^{[l]}$, and $a^{[l]} = g^{[l]}(z^{[l]})$. Propagating the sensitivity back through the weight matrix and then through the activation gives:

$$\boxed{ \delta^{[l]} = \left( \left(W^{[l+1]}\right)^{T} \delta^{[l+1]} \right) \odot g'^{[l]}\!\left(z^{[l]}\right) }$$

Two operations happen here. The transpose $\left(W^{[l+1]}\right)^{T}$ sends the downstream error back across the linear map, spreading each downstream component onto the units that fed it. The elementwise product with $g'^{[l]}(z^{[l]})$ then filters it by how sensitive each activation was at its operating point.

| Symbol | Meaning | Shape |

| --- | --- | --- |

| $\delta^{[l]}$ | error at layer $l$ | $(n_l)$ |

| $W^{[l+1]}$ | weights into layer $l+1$ | $(n_{l+1} \times n_l)$ |

| $\left(W^{[l+1]}\right)^{T}\delta^{[l+1]}$ | error pushed back to layer $l$ | $(n_l)$ |

| $g'^{[l]}(z^{[l]})$ | local activation slope | $(n_l)$ |

*Remark:* the forward pass uses $W^{[l+1]}$ and the backward pass uses its transpose. This is the same linear map read in reverse, which is why the backward pass has the same cost as the forward pass.

The error $\delta^{[l]}$ is all we need for the parameters of layer $l$. Since $z^{[l]} = W^{[l]} a^{[l-1]} + b^{[l]}$ is linear in $W^{[l]}$ and $b^{[l]}$, the last chain-rule link is easy. The weight gradient is the outer product of the layer error with the cached input activation:

$$\boxed{ \frac{\partial L}{\partial W^{[l]}} = \delta^{[l]} \left(a^{[l-1]}\right)^{T}, \qquad \frac{\partial L}{\partial b^{[l]}} = \delta^{[l]} }$$

The weight gradient has shape $(n_l \times n_{l-1})$, matching $W^{[l]}$, and the bias gradient has shape $(n_l)$, matching $b^{[l]}$. The bias gradient is exactly $\delta^{[l]}$ because $\partial z^{[l]} / \partial b^{[l]}$ is the identity.

*Remark:* the cached activation $a^{[l-1]}$ from the forward pass is reused verbatim in the weight gradient. This is the concrete payoff of caching: nothing from the forward pass is recomputed.

Backpropagation runs one forward pass to fill a cache, one backward pass to propagate $\delta$, and then a parameter update.

1. **Forward pass.** Set $a^{[0]} = x$. For $l = 1, \dots, L$ compute $z^{[l]}$ and $a^{[l]}$, caching each. Evaluate the loss $L$ at $\hat{y} = a^{[L]}$.

2. **Output error.** Compute $\delta^{[L]}$ from Section 5.3.

3. **Backward pass.** For $l = L-1, \dots, 1$ apply the recursion of Section 5.4 to get $\delta^{[l]}$.

4. **Gradients.** For each layer form $\partial L / \partial W^{[l]}$ and $\partial L / \partial b^{[l]}$ from Section 5.5.

5. **Update.** Over a batch, average the per-example gradients into the cost gradient $\nabla J$ and take one gradient-descent step (detailed in [Optimization](/en/Deep%20Learning/06%20Optimization)).


*Remark:* backpropagation gives the gradient, not the step. It answers which direction lowers the cost, and by how much per unit of each parameter. Turning that gradient into an actual weight change is the job of the optimizer.

In short, backpropagation is an ordered, single-pass application of the chain rule that reuses cached forward quantities to compute every gradient at the price of roughly one extra forward pass. *With the gradient in hand, the next lesson studies how to use it well: learning rates, momentum, and the adaptive methods that make deep networks trainable.*

---

Next: [Optimization](/en/Deep%20Learning/06%20Optimization) · [Course overview](/en/Deep%20Learning)