Maximum Likelihood and Cross-Entropy Loss

Maximum likelihood estimation is a general principle for fitting probabilistic models: given data and a family of distributions indexed by parameters $\theta$, choose $\theta$ to maximize the probability (density) of having observed exactly that data. Formally, if the data points $x_1, \dots, x_n$ are independent and identically distributed given $\theta$, the likelihood function is

and MLE solves $\hat{\theta} = \arg\max_\theta L(\theta)$. In binary classification we model each label $y_i \in \{0, 1\}$ as a Bernoulli random variable whose success probability depends on the input $x_i$ through the model. Logistic regression defines this probability using the sigmoid of a linear score (the logit) $z_i = w^\top x_i + b$:

Because $y_i$ is Bernoulli, its probability mass function can be written compactly for both outcomes at once:

Check this: if $y_i = 1$ the exponent on $(1-\hat y_i)$ vanishes and the expression reduces to $\hat y_i$; if $y_i = 0$ it reduces to $1 - \hat y_i$. This single-formula trick is what lets us differentiate one expression instead of branching on cases. Assuming the training examples are independent given the inputs, the likelihood of the entire dataset is the product of these individual Bernoulli probabilities. Cross-entropy loss is defined as the negative logarithm of this likelihood, averaged (or summed) over the dataset — turning the maximization of a product into the minimization of a sum, which is both numerically stable and analytically convenient.

The derivation proceeds in five mechanical but important steps. Each step is a standard MLE manipulation that recurs throughout statistical learning theory.

Step 1 — Write the per-example likelihood. For example $i$ with true label $y_i$ and model prediction $\hat y_i = \sigma(z_i)$:

Step 2 — Form the joint likelihood over the dataset. Assuming conditional independence of labels given inputs and parameters:

Step 3 — Take the logarithm (log-likelihood). The logarithm is strictly increasing, so it preserves the location of the maximum while converting the product into a sum:

Step 4 — Negate and average to get a loss to minimize. Optimization libraries minimize by convention, and averaging (rather than summing) keeps the loss scale independent of dataset size:

This $J$ is exactly binary cross-entropy loss.

Step 5 — Differentiate and optimize. Because $\hat y_i = \sigma(z_i)$ and $\sigma'(z) = \sigma(z)(1-\sigma(z))$, the chain rule yields a remarkably clean gradient with respect to the logit:

The gradient with respect to the weights follows from $z_i = w^\top x_i + b$:

This is passed to an iterative optimizer such as gradient descent or a quasi-Newton method. ``` Algorithm: Fit logistic regression by maximum likelihood (cross-entropy minimization) Input: training pairs {(x_i, y_i)}, i = 1..n, learning rate eta, iterations T Output: weights w, bias b 1. Initialize w <- 0 (or small random values), b <- 0 2. for t = 1 to T: 3. z_i <- w^T x_i + b for all i // compute logits 4. yhat_i <- sigmoid(z_i) for all i // Bernoulli success prob. 5. J <- -(1/n) sum_i [ y_i log(yhat_i) + (1 - y_i) log(1 - yhat_i) ] 6. grad_z_i <- yhat_i - y_i for all i // dJ/dz_i 7. grad_w <- (1/n) sum_i grad_z_i x_i 8. grad_b <- (1/n) sum_i grad_z_i 9. w <- w - eta grad_w 10. b <- b - eta grad_b 11. return w, b ``` Two edge cases the pseudocode must respect in a real implementation: (a) if $\hat y_i$ is exactly $0$ or $1$ due to floating point saturation, $\log \hat y_i$ or $\log(1-\hat y_i)$ overflows to $-\infty$, so production code clips $\hat y_i$ into $[\epsilon, 1-\epsilon]$ or fuses the sigmoid and log into a single numerically stable "log-sum-exp" formula; (b) the loop must handle $n=0$ (no data) by refusing to divide, though this is a degenerate input that should be rejected upstream.

Time complexity. Each forward pass over $n$ examples with $d$-dimensional features costs $O(nd)$: computing all logits $z_i = w^\top x_i + b$ takes $O(d)$ per example. The backward pass (gradient computation) reuses the same $\hat y_i - y_i$ residuals and also costs $O(nd)$ to accumulate $\nabla_w J$. If gradient descent runs for $T$ iterations, total time is $O(Tnd)$. This is the same asymptotic cost as fitting linear regression by least squares via gradient descent — the probabilistic reinterpretation changes the loss function, not the per-step arithmetic cost.

Space complexity. Storing the dataset requires $O(nd)$; the parameters $w$ require $O(d)$; intermediate logits and predictions require $O(n)$. Mini-batch variants reduce the per-step memory footprint to $O(bd)$ for batch size $b \ll n$, trading more iterations for lower peak memory — the standard time/space tradeoff in large-scale learning.

Correctness argument. There are two things to verify: that minimizing $J$ is equivalent to maximizing likelihood, and that this minimization problem is well-behaved (has a unique solution reachable by gradient descent). Equivalence. The logarithm is a strictly monotonically increasing function, so for any two parameter settings $\theta_1, \theta_2$,

Hence $\arg\max_\theta L(\theta) = \arg\min_\theta \big(-\log L(\theta)\big)$, and dividing by the constant $n$ to average does not change the location of the minimizer. This proves $J(w,b)$ as defined is minimized at exactly the MLE. Convexity. Write the loss for one example as a function of the logit $z$:

Its second derivative with respect to $z$ is

for every $z$, regardless of $y$ (this uses $\sigma' = \sigma(1-\sigma)$ and the fact that the $y$-dependence drops out after differentiating twice — a short calculation confirms the cross terms cancel). Since $z = w^\top x + b$ is an affine function of $w$, and $\phi$ is convex in $z$, the composition $J(w,b)$ is convex in $(w,b)$ (convexity is preserved under affine pre-composition). A convex function bounded below and differentiable has the property that every stationary point (where the gradient is zero) is a global minimum — so gradient descent, run with a suitable step size, cannot get stuck in a bad local minimum. This is a much stronger correctness guarantee than is available for the non-convex loss surfaces of deep networks.

Honest tradeoffs. Convexity relies on the model being logistic regression specifically; once $\hat y$ is produced by a deep network with nonlinear hidden layers, the same cross-entropy loss composed with the network is generally non-convex in the network's weights, even though it remains convex in the final logit. Cross-entropy also assumes labels are correctly Bernoulli-distributed given $x$; mislabeled data or heavy class imbalance violates this assumption and can make the MLE solution behave poorly (see pitfalls below). Finally, cross-entropy heavily penalizes confident wrong predictions ($\log$ blows up near $0$), which is a feature for calibrated probability estimates but a liability if a small fraction of labels are simply noise.

Consider a 1-D toy dataset with a single feature $x$ (already centered/scaled) and label $y$, and suppose the model has converged to $w = 2.0$, $b = -0.5$. We evaluate the loss on four training points to see cross-entropy in action.

Step through row 4 by hand, since it is the interesting case (a confident wrong-direction prediction). With $x_4 = -1.0$, $z_4 = 2(-1.0) - 0.5 = -2.5$. Then

The true label is $y_4 = 1$, but the model assigns it only a $7.6\%$ chance of being class 1 — a confidently wrong prediction. Its loss is

roughly 13 times larger than the loss on the correctly-and-confidently-classified point (row 3, loss $0.0297$). This asymmetry is exactly the point of cross-entropy: because it is $-\log$ of a probability, loss grows without bound as $\hat y \to 0$ for a positive example, so the objective aggressively pushes the optimizer to fix confident mistakes, while giving diminishing returns for making already-correct, already-confident predictions even more confident. Averaging the four losses gives the batch cross-entropy:

A gradient step would use $\hat y_i - y_i$ for each row — note row 4 contributes $0.0759 - 1 = -0.9241$, by far the largest-magnitude residual, so it dominates the gradient direction and pulls $w, b$ to correct this point on the next iteration. This is the mechanism by which cross-entropy "focuses" learning on the hardest current mistakes.

Cross-entropy loss (or its multi-class generalization, softmax cross-entropy) is the default training objective for essentially every modern classifier: image classification networks, spam and fraud detectors, medical diagnosis models, and the token-prediction objective at the heart of large language models is itself a categorical cross-entropy over the vocabulary at each position. Search ranking systems and recommendation engines use binary cross-entropy to model click/no-click and like/dislike events as Bernoulli outcomes. Speech recognition systems historically trained acoustic models with cross-entropy before sequence-level objectives were layered on top. The same likelihood-based derivation underlies logistic regression in epidemiology and social science, where interpretable coefficients from MLE fits are reported directly as odds ratios. Any system that needs a calibrated probability — not just a class label — leans on the fact that cross-entropy-trained sigmoid/softmax outputs approximate $P(y \mid x)$ under the MLE framework, which is why it is preferred over margin-based losses (like the hinge loss in SVMs) when probability estimates, not just decisions, are required downstream.

Ask the tutor: Can you show how this same maximum-likelihood recipe derives mean squared error from a Gaussian noise assumption in linear regression?