Loss Functions for Deep Networks

A loss function $L(\hat y, y)$ maps a prediction $\hat y$ (the network's output after Forward Propagation) and a ground-truth target $y$ to a nonnegative real number, with $L=0$ only when the prediction is perfect. Averaged over a dataset of $n$ examples, this gives the cost function (sometimes the terms are used interchangeably)

where $\theta$ collects all weights and biases. Training a network is exactly the problem of choosing $\theta$ to minimize $J(\theta)$, typically via gradient-based optimization. The loss function is therefore the objective the entire learning machinery serves — every architectural choice from The Perceptron onward exists to produce a $\hat y$ that this objective can score. Two losses dominate deep learning because they arise naturally from maximum likelihood estimation. If we assume that targets are generated as $y = f(x) + \epsilon$ with Gaussian noise $\epsilon \sim \mathcal N(0,\sigma^2)$, maximizing the likelihood of the data under this model is equivalent to minimizing the sum of squared errors — this gives mean squared error (MSE), the standard loss for regression. If instead we assume targets are drawn from a categorical distribution over $K$ classes with the network producing estimated probabilities $\hat p$, maximizing likelihood is equivalent to minimizing the negative log-likelihood of the true class, which is exactly cross-entropy. Both losses, then, are not arbitrary engineering choices — they are the loss that makes gradient descent equivalent to maximum likelihood estimation under a natural noise model for the task.

Mean Squared Error. For a regression network producing scalar (or vector) output $\hat y$ against target $y \in \mathbb R^d$:

Its gradient with respect to the output is simple and linear:

which is exactly proportional to the error — large errors produce large gradients, small errors produce small gradients. This is a desirable, well-behaved signal as long as the output layer itself has a well-behaved gradient (e.g., a linear output unit).

Cross-entropy. For a $K$-way classifier, the network typically outputs raw scores ("logits") $z_1,\dots,z_K$, passed through a softmax:

Given a one-hot target vector $y$ (with $y_c = 1$ for the true class $c$, else 0), cross-entropy is

For binary classification with a sigmoid output $\hat p \in (0,1)$ and target $y\in\{0,1\}$, this specializes to

The key mechanical fact, derived below, is that when softmax (or sigmoid) is paired with cross-entropy, the gradient with respect to the logits $z$ collapses to the remarkably clean form

This is the gradient that The Chain Rule and Backpropagation receives as its starting point and propagates backward through every earlier layer.

Step-by-step algorithm (one training step)

```text function COMPUTE_LOSS_AND_GRAD(x, y_true, network, task):

# ---- forward pass (see Forward Propagation) ---- z = network.forward_to_logits(x) # raw scores, shape (K,) or scalar if task == "regression": y_hat = z # linear output unit loss = mean((y_hat - y_true)^2) dL_dyhat = (2 / len(y_true)) (y_hat - y_true) return loss, dL_dyhat # feed into backprop at output layer if task == "binary_classification": p_hat = sigmoid(z) eps = 1e-12 # numerical floor, avoids log(0) loss = -( y_truelog(p_hat+eps) + (1-y_true)*log(1-p_hat+eps) ) dL_dz = p_hat - y_true # combined sigmoid+BCE gradient return loss, dL_dz if task == "multiclass_classification": p_hat = softmax(z) # shift-stabilized, see below c = index_of_true_class(y_true) loss = -log(p_hat[c] + eps) dL_dz = p_hat.copy() dL_dz[c] -= 1 # p_hat - one_hot(y_true) return loss, dL_dz ``` The averaged cost over a mini-batch of size $m$ is simply $\tfrac1m\sum_i L_i$, and the gradients above are likewise averaged before being handed to the optimizer.

Numerical stability note. Computing softmax naively, $e^{z_k}/\sum_j e^{z_j}$, can overflow for large logits. The standard fix subtracts the max logit first: ```text function SOFTMAX_STABLE(z): m = max(z) exps = [exp(zk - m) for zk in z] s = sum(exps) return [e / s for e in exps] ``` This is mathematically identical (the max cancels in numerator and denominator) but keeps exponents $\le 0$, avoiding overflow.

Time complexity. For a $K$-class output, computing softmax is $O(K)$ (one exponential and one sum per class), and computing cross-entropy given softmax output is $O(1)$ additional work (a single logarithm lookup for the true class, or $O(K)$ if using the full one-hot dot product). MSE over a $d$-dimensional output is $O(d)$. Over a mini-batch of size $m$, both losses cost $O(mK)$ or $O(md)$ — linear in batch size and output dimension, which is negligible compared to the $O(\text{parameters})$ cost of the forward and backward passes through the network itself. Loss computation is never the bottleneck of training; it is a thin, cheap layer sitting on top of an otherwise expensive computation graph.

Space complexity. Both losses require only $O(K)$ or $O(d)$ auxiliary storage per example (the softmax vector, or the elementwise squared differences), which is dominated by the memory needed to store activations for the backward pass.

Correctness: why MSE is the right loss for Gaussian regression. Assume $y_i = f_\theta(x_i) + \epsilon_i$ with $\epsilon_i \sim \mathcal N(0, \sigma^2)$ i.i.d. The likelihood of the dataset is

Taking the negative log-likelihood and dropping constants independent of $\theta$:

Minimizing this over $\theta$ is identical to minimizing $\sum_i (y_i - \hat y_i)^2$ — i.e., MSE. So MSE is not an ad hoc choice; it is exactly the maximum-likelihood objective under a Gaussian-noise assumption. This also explains a well-known limitation: MSE is sensitive to outliers, because squared error grows quadratically, mirroring the thin tails of the Gaussian it assumes.

Correctness: why cross-entropy is the right loss for classification. With targets drawn categorically ($y$ is one-hot over $K$ classes with true probability 1 on the correct class) and model $\hat p_\theta(x)$, the likelihood of one example is $\prod_k \hat p_k^{\,y_k} = \hat p_c$ (only the true-class term survives). The negative log-likelihood over the dataset is

which is exactly the sum of cross-entropy losses. So cross-entropy minimization is maximum likelihood estimation for a categorical output model — the same logic that justifies MSE for Gaussian regression.

Correctness: the softmax + cross-entropy gradient simplification. This is the crucial computational fact that makes cross-entropy so practical. Let $z$ be logits, $\hat p = \text{softmax}(z)$, $L = -\sum_k y_k \log \hat p_k$. Using $\partial \hat p_k/\partial z_j = \hat p_k(\mathbb 1[k=j] - \hat p_j)$ (the softmax Jacobian) and the chain rule:

using $\sum_k y_k = 1$ since $y$ is one-hot. So despite softmax and log both being nonlinear, their combined gradient is the trivially simple residual $\hat p - y$. This matters enormously in practice: it means the error signal handed to backpropagation is large when the prediction is confidently wrong and vanishes gracefully as $\hat p \to y$, with no extra multiplicative shrinkage from a saturating log or exponential derivative. Contrast this with using MSE on softmax outputs for classification: the chain rule there multiplies by the softmax Jacobian and the derivative of squared error, and because softmax saturates (derivatives near 0 or 1 flatten out), the resulting gradient can become vanishingly small even when the prediction is badly wrong — a classic cause of stalled training. This is the mathematical reason cross-entropy, not MSE, is the standard loss for classifiers.

Convexity caveat. Both losses are convex functions of the model's output ($\hat y$ or $\hat p$) — a genuine, checkable mathematical property. But composed with a deep, nonlinear network $f_\theta$, the loss as a function of the weights $\theta$ is generally non-convex, so gradient descent finds only a local optimum (or saddle), not a certified global minimum. Nothing about choosing the "right" loss removes this fundamental difficulty of deep learning optimization.

Consider a 3-class classifier (cat / dog / bird) that has just produced logits $z = [2.0, 1.0, 0.1]$ for an image whose true label is dog (class index 1).

Step 1 — stabilized softmax. Subtract $\max(z) = 2.0$: shifted logits $= [0, -1.0, -1.9]$.

Sum $= 1.5175$. Normalizing:

(check: $0.6590+0.2424+0.0986 = 1.0000$ ✓)

Step 2 — cross-entropy loss. True class is dog, so $L = -\log(0.2424) = 1.417$ (natural log). Notice the loss is fairly large: the model put most of its probability mass (0.659) on the wrong class (cat).

Step 3 — gradient with respect to logits, using $\partial L/\partial z_k = \hat p_k - y_k$ with one-hot $y=[0,1,0]$:

Interpretation: the gradient pushes the "cat" logit down (positive gradient on a loss we minimize means decrease it) and pushes the "dog" logit up (large negative gradient means increase it strongly), exactly the correction we want. This 3-vector is exactly what gets handed to Backpropagation to update every earlier weight in the network.

Contrast with MSE. If instead we trained by applying MSE directly to $\hat p$ against $y$: $L_{\text{MSE}} = \frac13[(0.659-0)^2+(0.2424-1)^2+(0.0986-0)^2] = \tfrac13[0.4343+0.5738+0.0097]=0.339$. Computing $\partial L/\partial \hat p_{\text{dog}} = \tfrac{2}{3}(0.2424-1) = -0.505$, then multiplying by the softmax Jacobian entry $\hat p_{\text{dog}}(1-\hat p_{\text{dog}}) = 0.2424\times0.7576 = 0.1836$ gives a logit gradient of only about $-0.093$ — nearly eight times smaller than the $-0.7576$ that cross-entropy delivers directly, for an equally wrong prediction. This numerically confirms the vanishing-gradient argument made above.

The probabilistic view of loss functions traces back to the foundations of statistics and information theory, later fused with neural network training in the 1980s–1990s as researchers connected likelihood-based objectives to gradient descent.

MSE and cross-entropy are the default loss functions built into every major deep learning framework, and nearly every production classifier or regressor uses one of them (or a close variant) as its training objective. Image classifiers underlying photo search and medical imaging diagnosis train with cross-entropy over thousands of classes. Speech recognition and machine translation systems use cross-entropy over vocabulary distributions at each decoding step, essentially treating "predict the next token" as a giant classification problem. Regression-style tasks — predicting a house price, a sensor reading, or a continuous control signal in robotics — use MSE or its robust cousins (Huber loss, mean absolute error) directly. Even large language models, at their core, are trained by minimizing cross-entropy between predicted next-token distributions and observed text, making the ideas in this article the literal training signal behind systems used by billions of people daily. Understanding these losses is also the necessary bridge to understanding The Chain Rule and Backpropagation, since the loss's gradient is precisely the starting point that backpropagation pushes through the network.

Ask the tutor: Can you derive, step by step, why the gradient of binary cross-entropy with a sigmoid output also simplifies to $\hat p - y$?