Properties of MLEs

Maximum likelihood asymptotics underlie standard errors and Wald/likelihood-ratio/score tests reported by virtually every statistical software package, from simple regression to complex generalized linear and mixed models. In genetics, MLE consistency and normality justify inference for allele frequencies and evolutionary rates from sequence data. In econometrics, asymptotic normality of the MLE is the basis for standard errors in logistic, Poisson, and duration models, and for the delta method used to obtain confidence intervals for nonlinear functions of parameters (odds ratios, elasticities). In machine learning, the same theory explains why maximum-likelihood-trained parametric models (e.g., exponential family models, some neural network heads under simplifying assumptions) have asymptotically Gaussian parameter uncertainty, which underlies Laplace approximations to Bayesian posteriors. In signal processing and cryptanalysis, Fisher information governs the fundamental limit on how precisely a parameter (e.g., a signal's arrival time, or an unknown key-related statistic) can be estimated from noisy observations, directly invoking the Cramér–Rao-type efficiency bound this article's results feed into.

Ask the tutor: How does the delta method extend asymptotic normality of the MLE to a nonlinear transformation of the parameter?

Picture the log-likelihood $\ell_n(\theta)$ as a random hill whose shape depends on the data. As $n$ grows, the Law of Large Numbers (via the Kullback–Leibler argument below) forces this hill, after dividing by $n$, to converge to a deterministic function that is uniquely maximized at the true parameter $\theta_0$. Consequently the location of the peak — the MLE — is pulled toward $\theta_0$: this is consistency. Zooming in on the peak with a microscope of power $\sqrt{n}$, the hill looks locally like an upside-down parabola (a Taylor expansion of $\ell_n$ around $\theta_0$), and the randomness in where the peak sits becomes Gaussian by the Central Limit Theorem applied to the score. The width of this local parabola is governed by $I(\theta_0)$: a sharply curved log-likelihood (large information) pins down $\theta$ tightly, giving a narrow Gaussian; a flat log-likelihood (small information) gives a wide one. The invariance property is a separate, purely geometric fact: maximizing $L_n(\theta)$ over $\theta$ and then applying $g$ to the maximizer is the same point as maximizing the reparametrized likelihood over $g(\theta)$ directly, because $g$ (if one-to-one) just relabels the same hill without changing where its peak is.

Example 1 — Exponential lifetime data with asymmetric rate. Let $X_1, \dots, X_n$ be i.i.d. $\text{Exp}(\lambda_0)$ with true rate $\lambda_0 = 3$ (so mean lifetime $1/3$), density $f(x;\lambda) = \lambda e^{-\lambda x}$, $x>0$. The log-likelihood is

giving score $s_n(\lambda) = n/\lambda - \sum X_i$; setting it to zero yields the familiar MLE $\hat\lambda_n = 1/\bar X_n$. The per-observation score is $s(x;\lambda) = 1/\lambda - x$, so

using $\operatorname{Var}(X) = 1/\lambda^2$ for an exponential. At $\lambda_0 = 3$, $I(\lambda_0) = 1/9$. Consistency gives $\hat\lambda_n \xrightarrow{P} 3$. Asymptotic normality gives

So for large $n$, $\hat\lambda_n \approx N(3, 9/n)$; e.g. with $n = 400$, the standard error is $\sqrt{9/400} = 0.15$, and an approximate 95% confidence interval is $3 \pm 1.96 \times 0.15 \approx (2.71, 3.29)$ (foreshadowing Confidence Intervals: Concept and Construction). Now apply invariance: suppose we actually care about the mean lifetime $\mu = g(\lambda) = 1/\lambda$, a bijection on $(0,\infty)$. By the invariance theorem, the MLE of $\mu$ is $g(\hat\lambda_n) = 1/\hat\lambda_n = \bar X_n$ — reassuringly, the sample mean, which is indeed the natural estimator of the mean and independently known to be the MLE of $\mu$ if we'd parametrized by $\mu$ from the start. This cross-check illustrates the theorem's content: we did not need to re-derive the MLE in the $\mu$-parametrization; we simply transformed the $\lambda$-MLE. To get the asymptotic variance of $\hat\mu_n = \bar X_n$ directly from the theory (rather than trusting the well-known $\operatorname{Var}(\bar X_n) = 1/(n\lambda_0^2)$), use the delta method (a consequence of asymptotic normality, not proved here but noted since it is the natural extension): if $\sqrt n(\hat\lambda_n - \lambda_0) \to N(0, I(\lambda_0)^{-1})$ and $g$ is differentiable at $\lambda_0$ with $g'(\lambda_0) \ne 0$, then

Here $g(\lambda) = 1/\lambda$, $g'(\lambda) = -1/\lambda^2$, so $g'(3)^2 = 1/81$, and the asymptotic variance of $\sqrt n(\hat\mu_n - 1/3)$ is $(1/81)(9) = 1/9$. Direct computation: $\operatorname{Var}(\bar X_n) = \operatorname{Var}(X)/n = (1/\lambda_0^2)/n = (1/9)/n$, matching $n \cdot \operatorname{Var}(\bar X_n) = 1/9$ exactly. The numbers agree, confirming both the invariance calculation and the delta-method variance.

Example 2 — Uniform on $(0,\theta)$: where the standard theorem fails, and why. Let $X_1,\dots,X_n$ be i.i.d. $\text{Unif}(0,\theta_0)$ with true value $\theta_0 = 5$. The likelihood is $L_n(\theta) = \theta^{-n} \mathbf 1\{\theta \geq \max_i X_i\}$, maximized at $\hat\theta_n = X_{(n)} = \max_i X_i$. This is a case where the support depends on $\theta$, violating regularity condition (R2), so the general asymptotic-normality theorem above does not apply — and indeed the conclusion fails in the expected form. We can still analyze $\hat\theta_n = X_{(n)}$ directly to see how it fails to be asymptotically normal at the $\sqrt n$ rate. For $t \geq 0$,

as $n \to \infty$ (for $0 \le t \le n\theta_0$), which is a legitimate probability. Hence

i.e. the correct convergence rate is $n$, not $\sqrt n$, and the limit is exponential, not Gaussian. At $\theta_0 = 5$: $n(5 - X_{(n)}) \xrightarrow{d} \text{Exp}(1/5)$, so for large $n$ the estimator is downward-biased with $E[X_{(n)}] \approx \theta_0 - \theta_0/n = 5 - 5/n$ (exactly $E[X_{(n)}] = \frac{n}{n+1}\theta_0$ here, computable directly from the order-statistic density $f_{X_{(n)}}(x) = n x^{n-1}/\theta_0^n$), and its fluctuations around $\theta_0$ are of order $1/n$, much smaller (faster concentration!) than the usual $1/\sqrt n$ — but non-Gaussian and skewed. This example is the standard illustration that dropping the fixed-support hypothesis (R2) can change both the rate and the limiting law entirely, not just the variance formula. By contrast, invariance still holds here with no extra hypotheses beyond the definition of MLE: for instance, the MLE of $\theta_0^2$ (say, if we cared about the square of the range) is exactly $X_{(n)}^2$, since invariance (proved above) required no regularity conditions at all — it is a purely algebraic fact about $\arg\max$.