Maximum Likelihood Estimation

Maximum likelihood estimation is the backbone of parametric statistical modeling across essentially every quantitative discipline. In biostatistics and epidemiology, MLE fits survival models (exponential, Weibull, Cox proportional hazards) to time-to-event data exactly as in the light-bulb example above. In machine learning, training a logistic regression classifier or a Gaussian mixture model (via the EM algorithm) is literally maximum likelihood estimation, and the cross-entropy loss minimized in neural network classifiers is the negative log-likelihood of a categorical model. In genetics, phylogenetic tree reconstruction from DNA sequence data is typically done by maximizing the likelihood of the tree topology and branch lengths under a nucleotide substitution model. In econometrics, MLE underlies probit and logit demand models, ARCH/GARCH volatility models, and structural macro models. In physics and engineering, MLE (often disguised as "least squares" under Gaussian noise, itself a special case) calibrates instruments and estimates decay rates, signal parameters, and channel characteristics in communications. In cryptography and coding theory, maximum-likelihood decoding chooses the transmitted codeword maximizing the probability of the observed noisy channel output, directly generalizing the estimation principle to a decision-theoretic setting. This universality is precisely why the theory built around MLE — its consistency, asymptotic normality, and connection to The Cramer-Rao Lower Bound via Properties of MLEs — is such foundational machinery for the Likelihood Ratio Tests used throughout modern hypothesis testing.

Ask the tutor: How does the Fisher information determine the asymptotic variance of the MLE, and how does that connect to the Cramér–Rao lower bound?

Think of $L(\theta)$ as a landscape whose height at each point $\theta$ tells you how consistent that parameter value is with the data you actually recorded. Different candidate parameters make the observed sample more or less "typical" — a parameter far from the truth would make the observed sample a bizarre, low-probability fluke, while a well-chosen parameter makes the observed sample look ordinary. Maximum likelihood estimation simply climbs to the top of this landscape. Because $L(\theta)$ is a product of $n$ terms, each between $0$ and $1$ (for densities bounded appropriately, or pmf values), it tends to be an extremely small number that underflows numerically and is unpleasant to differentiate via the product rule. Taking logarithms converts the product into a sum without changing where the maximum occurs, since $\log$ is strictly increasing. This is why one almost always maximizes $\ell(\theta) = \sum_i \log f(x_i;\theta)$ instead of $L(\theta)$ directly — same maximizer, vastly friendlier calculus. The score $S(\theta) = \ell'(\theta)$ measures the local slope of the log-likelihood. Where the score is positive, increasing $\theta$ increases plausibility; where it is negative, decreasing $\theta$ helps. A stationary point $S(\hat\theta)=0$ is a candidate summit — but one must check it is a maximum (via a second-derivative or global argument), and one must also check the true maximum is not hiding at a boundary of $\Theta$, where the score need not vanish at all.

Example 1: Exponential lifetimes (rate parameter, interior maximum). Suppose $X_1,\dots,X_n$ are i.i.d. $\mathrm{Exponential}(\lambda)$ with density $f(x;\lambda) = \lambda e^{-\lambda x}$ for $x>0$, $\lambda>0$, modeling the ten light-bulb lifetimes from the overview. Take asymmetric observed data (in hours): $x = (2.1,\ 5.4,\ 1.0,\ 3.3,\ 7.8,\ 0.6,\ 4.5,\ 2.9,\ 6.1,\ 1.3)$, so $n=10$ and $\sum x_i = 35.0$. The likelihood is

The log-likelihood is

The support of $f(\cdot;\lambda)$ is $(0,\infty)$ for every $\lambda>0$, satisfying the support condition; $\log f(x;\lambda) = \log\lambda - \lambda x$ is smooth in $\lambda$ on $(0,\infty)$, satisfying the smoothness condition; and $\Theta = (0,\infty)$ is open. The score is

Setting $S(\lambda)=0$ gives $\hat\lambda = 10/35.0 = 2/7 \approx 0.2857$ per hour. Check the second derivative: $\ell''(\lambda) = -10/\lambda^2 < 0$ for all $\lambda>0$, so $\ell$ is (strictly) concave everywhere on $\Theta$. By the theorem's converse direction, the unique stationary point $\hat\lambda = 2/7$ is the global maximizer. In general, with sample mean $\bar x = n^{-1}\sum x_i$, the same computation gives $\hat\lambda = 1/\bar x$; here $\bar x = 3.5$ and indeed $1/3.5 = 2/7$. The MLE of the rate is the reciprocal of the sample mean lifetime — matching the intuitive method-of-moments estimator, though the two need not always coincide in other models.

Example 2: Uniform$(0,\theta)$ (boundary maximum, calculus does not apply). Suppose $X_1,\dots,X_n$ are i.i.d. $\mathrm{Uniform}(0,\theta)$ with density

This model violates hypothesis 1 of the theorem: the support $\{x: f(x;\theta)>0\} = [0,\theta]$ depends on $\theta$. So we cannot expect the likelihood equation to locate the maximum, and indeed it will not. Take asymmetric data $x = (1.2,\ 4.7,\ 0.3,\ 3.9,\ 2.5)$, so $n=5$ and $m := \max_i x_i = 4.7$. The joint density is

For $\theta < 4.7$, $L(\theta) = 0$ (the data point $x=4.7$ is impossible under that $\theta$). For $\theta \ge 4.7$, $L(\theta) = \theta^{-5}$, which is strictly decreasing in $\theta$ — there is no interior stationary point since $L'(\theta) = -5\theta^{-6} \ne 0$ anywhere on $(4.7,\infty)$. The maximum of $L$ over $\theta \ge 4.7$ therefore occurs at the smallest admissible value, the left endpoint $\theta = 4.7$:

This example shows explicitly why hypothesis 1 (constant support) is essential to the theorem: here the maximizer is found by direct inspection of the shape of $L$ (increasing on $[0,m)$ trivially — it is zero there — then strictly decreasing on $[m,\infty)$), not by solving a likelihood equation, since $\ell$ is not even differentiable at the maximizing point $\hat\theta=m$.