The Cramer-Rao Lower Bound

The Cramér–Rao bound is a cornerstone of statistical signal processing: engineers designing radar, sonar, and communication systems compute the CRLB for parameters like time delay or frequency to benchmark any estimator's achievable precision before building it. In physics, it underlies the theory of quantum metrology, where the quantum Fisher information generalizes the classical bound to set ultimate limits on how precisely a physical parameter (like phase in an interferometer) can be measured given quantum states. In machine learning and computational statistics, the bound motivates the study of asymptotic efficiency of maximum likelihood estimators, since under regularity conditions MLEs asymptotically achieve the Cramér–Rao bound (a fact building on the large-sample theory of the score). Cryptographic parameter estimation and experimental design (choosing sample sizes or measurement configurations to maximize Fisher information) both use the bound directly as an optimization target. The bound also connects tightly to Sufficiency and the Factorization Theorem: equality in the CRLB forces the estimator to be an affine function of the score, which happens precisely in exponential families where the natural sufficient statistic plays that role — and to The Rao-Blackwell Theorem, which shows how conditioning any unbiased estimator on a sufficient statistic can only reduce variance, providing a constructive path toward (though not a guarantee of) attaining the Cramér–Rao bound.

Ask the tutor: Can you show me how the Cramér–Rao bound connects to the asymptotic variance of the MLE via the second-derivative (observed information) form of Fisher information?

Think of the log-likelihood $\ell(\theta) = \log f(x;\theta)$ as a hill whose peak sits at the maximum likelihood estimate. Fisher information $I(\theta)$ measures the expected curvature of this hill at the true parameter: a sharply peaked hill (high curvature, high information) pins down $\theta$ tightly, while a broad, flat hill (low curvature, low information) leaves $\theta$ poorly determined by the data. The score function $S(X;\theta) = \partial_\theta \log f(X;\theta)$ is the slope of this hill, and it has mean zero and variance $I(\theta)$ under the regularity conditions. The proof idea is that any unbiased estimator $T$ must be correlated with the score in a very specific way — differentiating the unbiasedness identity $E_\theta[T] = \tau(\theta)$ forces $\operatorname{Cov}_\theta(T, S) = \tau'(\theta)$. Cauchy–Schwarz then converts this fixed covariance into a lower bound on $\operatorname{Var}(T)$, since covariance cannot exceed the product of standard deviations. The sharper the score's own variance (the higher $I(\theta)$), the less variance $T$ needs to produce the required covariance — hence the bound shrinks as information grows.

Technique used: this proof is the classical Cauchy–Schwarz argument applied to the pair (estimator, score), preceded by a differentiation-under-the-integral-sign computation that produces the exact covariance $\tau'(\theta)$. No circularity: the lemma about the score is proved from first principles (differentiating the normalization identity $\int f = 1$), and the main theorem uses only that lemma and Cauchy–Schwarz, neither of which presupposes the CRLB.

Example 1: Exponential distribution, estimating the mean. Let $X_1,\dots,X_n$ be i.i.d. $\mathrm{Exponential}(\lambda)$ with density $f(x;\lambda) = \lambda e^{-\lambda x}$, $x>0$, so $E[X_i] = 1/\lambda$. We parametrize by $\theta = 1/\lambda$ (the mean) directly for a cleaner target, writing the density as $f(x;\theta) = \theta^{-1} e^{-x/\theta}$. Log-likelihood for one observation: $\ell(\theta) = -\log\theta - x/\theta$. Score:

Second derivative:

Using the standard identity $I(\theta) = -E_\theta\left[\partial^2\ell/\partial\theta^2\right]$ (valid under the same regularity conditions, since $E[S]=0$ implies $E[\partial_\theta S] = -\operatorname{Var}$ ... let us instead confirm directly via the second-derivative form, which is a standard equivalent computation):

For $n$ i.i.d. observations, information adds: $I_n(\theta) = n/\theta^2$. Sample mean as estimator. Take $T = \bar X_n = \frac{1}{n}\sum X_i$. This is unbiased for $\theta$ since $E_\theta[\bar X_n] = \theta$, so $\tau(\theta)=\theta$, $\tau'(\theta)=1$. Its true variance is

since $\operatorname{Var}(X_1) = \theta^2$ for $\mathrm{Exponential}(1/\theta)$ (a standard fact: variance of an exponential equals the square of its mean). The Cramér–Rao bound is

These match exactly: $\operatorname{Var}_\theta(\bar X_n) = \theta^2/n = \text{CRLB}$. So $\bar X_n$ is an efficient estimator of the mean $\theta$ for the exponential model, for every finite $n$ — not just asymptotically. This is consistent with the equality condition: here $T - \tau(\theta) = \bar X_n - \theta$ and the score is $S_n(\theta) = -n/\theta + \left(\sum X_i\right)/\theta^2$, and indeed

which is exactly the required proportionality with $c(\theta) = \theta^2/n$ — confirming equality holds.

Example 2: Poisson distribution, estimating $\theta^2$ (a nonlinear target with a numerical mismatch). Let $X_1,\dots,X_n$ be i.i.d. $\mathrm{Poisson}(\theta)$, $\theta>0$, pmf $f(x;\theta) = e^{-\theta}\theta^x/x!$. Log-likelihood for one observation: $\ell(\theta) = -\theta + x\log\theta - \log x!$. Score:

Then $\operatorname{Var}(S) = \operatorname{Var}(X)/\theta^2 = \theta/\theta^2 = 1/\theta$ (using $\operatorname{Var}(X)=\theta$ for Poisson), so $I_1(\theta) = 1/\theta$, and for $n$ observations $I_n(\theta) = n/\theta$. Suppose we want to estimate $\tau(\theta) = \theta^2$ using the natural unbiased-corrected estimator built from $\bar X_n$. First note $E[\bar X_n^2] = \operatorname{Var}(\bar X_n) + (E[\bar X_n])^2 = \theta/n + \theta^2$. So $\bar X_n^2$ is biased upward for $\theta^2$ by $\theta/n$. The unbiased estimator is

since $E_\theta[T] = \left(\theta/n + \theta^2\right) - \theta/n = \theta^2 = \tau(\theta)$. Here $\tau'(\theta) = 2\theta$, so the CRLB is

To see the bound is generally not attained here, compute $\operatorname{Var}_\theta(T)$ for concrete numbers: let $n=1$ (single Poisson observation $X$), so $T = X^2 - X$. We need $E[T^2]$ and $E[T]$. We already have $E[T] = \theta^2$. For $E[T^2] = E[(X^2-X)^2] = E[X^4 - 2X^3 + X^2]$, use Poisson factorial moments: $E[X(X-1)] = \theta^2$ so $E[X^2]=\theta^2+\theta$; the factorial moment $E[X(X-1)(X-2)(X-3)] = \theta^4$ gives, after expansion, the fourth central moments of Poisson. It is a standard fact that for $X\sim\mathrm{Poisson}(\theta)$,

Write $X^2 - X = X(X-1)$, so in fact $T = X(X-1)$ exactly (a cleaner form: $\bar X_n^2 - \bar X_n/n$ with $n=1$ is $X^2 - X = X(X-1)$). Then

Expand $X^2(X-1)^2 = X(X-1)\cdot X(X-1)$; write $X(X-1) = X^2-X$, and use the factorial-moment expansion:

It is cleaner to expand directly in factorial moments using the known identity

which one checks by substituting $y = X(X-1)$ so $X^2(X-1)^2 = y^2$ and comparing polynomial coefficients in $X$ on both sides (both sides are degree-4 polynomials in $X$ agreeing at $X=0,1,2,3,4$ by direct substitution, hence identical as polynomials). Taking expectations:

Hence

Compare to the CRLB at $n=1$: $4\theta^3/1 = 4\theta^3$. For, say, $\theta = 2$: $\operatorname{Var}(T) = 6(8) + 6(4) = 48+24=72$, while CRLB $=4(8)=32$. Indeed $72 > 32$, confirming the inequality holds strictly — $T$ is unbiased but not efficient for $\tau(\theta)=\theta^2$, and the gap ($72$ vs. $32$) is substantial, illustrating that nonlinear functionals of $\theta$ typically fail to admit efficient unbiased estimators even when $\theta$ itself does.