Before a network can learn it needs a target to descend toward. The output layer turns the last activation $a^{[L]}$ into a prediction, and the loss measures how far that prediction is from the true label. This module fixes both choices per task, because backpropagation in the next module differentiates a concrete loss. The output activation and the loss are not picked independently: matching them to the task shape is what makes the training signal clean.

- Go from a per-example loss $L$ to the cost $J$ averaged over the batch.

- Choose a linear output with mean squared error for regression.

- Choose a sigmoid output with binary cross-entropy for two-class problems.

- Choose a softmax output with categorical cross-entropy for multiclass problems.

- Derive the clean logit gradient of the softmax and cross-entropy pair.

- Map any task to its output activation and loss with a single lookup table.

The network predicts $\hat{y} = a^{[L]}$ from input $a^{[0]} = x$. For a single example the loss $L(\hat{y}, y)$ scores that prediction against the target $y$. Training minimizes the cost $J$, defined as the average of $L$ over the $m$ examples in the batch or dataset:

$$\boxed{ J = \frac{1}{m}\sum_{i=1}^{m} L\!\left(\hat{y}^{(i)}, y^{(i)}\right) }$$

*Remark:* the loss $L$ scores one prediction, the cost $J$ is what the optimizer actually reduces. Averaging (rather than summing) keeps the gradient scale independent of the batch size, so the learning rate does not have to be retuned when $m$ changes.

The three tasks below reuse the losses introduced in the Machine Learning course. The cross-entropy row of the loss table at [General concepts](/en/Machine%20Learning/02%20General%20concepts), labelled "Neural networks", is exactly the objective a classification network minimizes. The novelty here is pairing each loss with the output activation $g^{[L]}$ that produces $\hat{y}$.

For a continuous target $y \in \mathbb{R}^{n_L}$ the output layer uses no activation, so it is linear (the identity) and the prediction can take any real value:

$$\boxed{ \hat{y} = a^{[L]} = z^{[L]} = W^{[L]} a^{[L-1]} + b^{[L]} }$$

The per-example loss is the squared Euclidean distance between prediction and target, scaled by one half:

$$\boxed{ L(\hat{y}, y) = \tfrac{1}{2}\,\lVert \hat{y} - y \rVert^2 }$$

*Remark:* the factor $\tfrac{1}{2}$ cancels the $2$ that appears on differentiating the square, leaving the tidy residual gradient $\partial L / \partial \hat{y} = \hat{y} - y$. This is the same mean-squared-error objective used for linear regression, now sitting on top of a deep network instead of a single linear score.

For a two-class label $y \in \{0, 1\}$ the output layer has a single unit whose activation is the sigmoid, squashing the logit $z^{[L]}$ into a probability:

$$\hat{y} = a^{[L]} = \sigma\!\left(z^{[L]}\right) = \frac{1}{1 + e^{-z^{[L]}}} \in (0, 1)$$

Here $\hat{y}$ is read as $p(y = 1 \mid x)$. The matching loss is the binary cross-entropy, the negative log-likelihood of the Bernoulli label:

$$\boxed{ L(\hat{y}, y) = -\Big[\, y \log \hat{y} + (1 - y)\log(1 - \hat{y}) \,\Big] }$$

*Remark:* only one of the two terms is active for any given label. When $y = 1$ the loss is $-\log \hat{y}$, penalizing a small predicted probability, and when $y = 0$ it is $-\log(1 - \hat{y})$. Cross-entropy is preferred over squared error here because it keeps the gradient large when the prediction is confidently wrong, so learning does not stall.


For a $K$-class label the output layer has $K$ units and the softmax activation turns the logit vector $z^{[L]} \in \mathbb{R}^{K}$ into a probability distribution over the classes:

$$\boxed{ \hat{y}_k = \frac{e^{z^{[L]}_k}}{\sum_{j=1}^{K} e^{z^{[L]}_j}} }$$

The outputs are positive and sum to one, so $\hat{y}$ is a proper distribution and $\hat{y}_k = p(y = k \mid x)$. The target $y$ is one-hot: $y_k = 1$ for the true class and $0$ otherwise. The matching loss is the categorical cross-entropy:

$$\boxed{ L(\hat{y}, y) = -\sum_{k=1}^{K} y_k \log \hat{y}_k }$$

*Remark:* because $y$ is one-hot the sum collapses to a single term, $-\log \hat{y}_{k^\star}$, where $k^\star$ is the true class. The loss therefore rewards putting probability mass on the correct class and ignores how the remaining mass is split. Binary cross-entropy is the special case $K = 2$.

The softmax output and the categorical cross-entropy loss are used together because their composition has a remarkably clean derivative at the logits $z^{[L]}$. Differentiating $L$ with respect to a single logit $z^{[L]}_k$ gives:

$$\boxed{ \frac{\partial L}{\partial z^{[L]}_k} = \hat{y}_k - y_k }$$

The gradient at the output layer is just the prediction minus the target, a plain residual with no awkward sigmoid or softmax factor left over. The same identity holds for the sigmoid and binary cross-entropy pair, which is its $K = 2$ instance. This is precisely why each activation is coupled to its matching loss rather than mixed with, say, squared error.

*Remark:* the elementwise form $\partial L / \partial z^{[L]} = \hat{y} - y$ is what seeds backpropagation. The next module starts the backward pass from this vector and then repeatedly applies the chain rule and the Hadamard product $\odot$ to push it back through the hidden layers.

The three cases collapse into one lookup. Fix the task, and the output activation and loss follow.

| Task | Output activation $g^{[L]}$ | Per-example loss $L$ | Logit gradient $\partial L / \partial z^{[L]}$ |

| --- | --- | --- | --- |

| Regression | linear (identity) | mean squared error | $\hat{y} - y$ |

| Binary classification | sigmoid | binary cross-entropy | $\hat{y} - y$ |

| Multiclass classification | softmax | categorical cross-entropy | $\hat{y} - y$ |

*Remark:* the last column is identical across all three rows. Matching the output activation to its natural loss makes the network start its backward pass from the same simple residual regardless of the task.


---

Next: [Backpropagation](/en/Deep%20Learning/05%20Backpropagation) · [Course overview](/en/Deep%20Learning)