Fisher Information and the Score

Fisher information is the load-bearing quantity of classical statistical inference. It appears directly in the Cramer-Rao Lower Bound, which states that no unbiased estimator can have variance smaller than $1/I_n(\theta)$ — turning the information computed here into a hard performance guarantee. It governs the asymptotic variance of the maximum likelihood estimator: under regularity, $\sqrt n(\hat\theta_n - \theta) \to N(0, I(\theta)^{-1})$, so Fisher information directly predicts the width of confidence intervals built from likelihood theory (see Properties of MLEs). Likelihood-ratio, Wald, and score (Rao) tests in applied statistics all use estimates of $I(\theta)$ to build test statistics and standard errors reported by every generalized linear model fitting routine. Beyond classical statistics, Fisher information equips the parameter space of a statistical model with a Riemannian metric — the Fisher information metric — the starting point of information geometry, connecting statistics to differential geometry. In physics, the same quantity appears in quantum estimation theory (quantum Fisher information bounds the precision of parameter estimation from quantum measurements) and in statistical mechanics via its close relationship to relative entropy. In signal processing and communications engineering, Fisher information underlies the Cramér–Rao bound used to assess the fundamental accuracy limits of radar, sonar, and localization systems. In cryptography and information theory, Fisher information's local, "small perturbation" character complements Shannon entropy's global characterization of uncertainty, and both concepts intersect in the study of the relationship between statistical estimation and channel capacity.

:::quiz[Under the regularity conditions of the theorem, what is $\mathbb{E}_\theta[U(\theta;X)]$?]

• $I(\theta)$
• $1$
• $-I(\theta)$
• $0$ ✓

:::faq[Why does the sign convention in $I(\theta) = -\mathbb{E}_\theta[\partial_\theta^2 \log f]$ make sense?] Because the log-likelihood attains a local maximum at the true parameter value (in a well-behaved model, on average), its second derivative there is typically negative — the log-likelihood curves downward from its peak. Negating this negative curvature yields a positive number, matching the fact that $I(\theta)$ is a variance and therefore must be nonnegative. Large negative curvature (a sharply peaked log-likelihood) corresponds to large Fisher information and precise estimation; near-zero curvature (a flat log-likelihood) corresponds to small Fisher information and imprecise estimation, exactly matching the intuition built in the overview.

Ask the tutor: How does Fisher information generalize to a vector parameter, and what does the resulting Fisher information matrix look like for the normal distribution with both mean and variance unknown?

Think of the log-likelihood $\ell(\theta) = \log f(x;\theta)$ as a hill whose shape depends on the observed data $x$. The score $U(\theta;x) = \ell'(\theta)$ is the slope of this hill at a given $\theta$. The maximum likelihood estimate is the point where the slope is zero — the summit. Fisher information asks: on average, how sharply peaked is this hill near the true parameter? A large Fisher information means the log-likelihood is sharply curved: nearby values of $\theta$ produce very different likelihoods, so the data can distinguish them easily, and the MLE will be precise. A small Fisher information means the log-likelihood is nearly flat: many values of $\theta$ explain the data almost equally well, and the MLE will be imprecise. This is why $I(\theta)$ enters as the variance of the score (a measure of typical fluctuation of the slope around its zero mean) and, equivalently, as the negative expected curvature (a direct measure of peakedness). The picture below shows two log-likelihood shapes from the same sample size but different underlying models: a peaked one (high information, precise inference) and a flat one (low information, imprecise inference).

Example 1: Exponential distribution (asymmetric rate parameterization). Let $X \sim \mathrm{Exp}(\theta)$ with density $f(x;\theta) = \theta e^{-\theta x}$ for $x > 0$, $\theta > 0$. We verify the regularity conditions hold (the support $(0,\infty)$ does not depend on $\theta$, and $f$ is smooth in $\theta$ with integrable derivatives — a routine dominated-convergence check we take as given here), then compute $I(\theta)$ two ways. Log-likelihood and score.

Check mean-zero score. Since $\mathbb{E}_\theta[X] = 1/\theta$,

confirming part (a) of the theorem directly. Fisher information via variance of the score. Since $U(\theta;X) = \tfrac1\theta - X$, $\operatorname{Var}_\theta(U(\theta;X)) = \operatorname{Var}_\theta(X)$. For $\mathrm{Exp}(\theta)$, $\operatorname{Var}_\theta(X) = 1/\theta^2$. Hence

Fisher information via curvature (cross-check). Differentiate the score:

This does not depend on $x$, so

matching the direct computation exactly, as guaranteed by part (b) of the theorem. Sample of size $n$. For $n$ i.i.d. exponential observations, $I_n(\theta) = n/\theta^2$, consistent with additivity (part (c)). This number will reappear as the denominator in the Cramér–Rao bound for estimating $\theta$: any unbiased estimator of $\theta$ has variance at least $\theta^2/n$.

Example 2: Poisson distribution with a nonstandard target (information about $\lambda^2$, illustrating reparametrization pitfalls). Let $X \sim \mathrm{Poisson}(\lambda)$, $\lambda > 0$, with p.m.f. $f(x;\lambda) = \dfrac{\lambda^x e^{-\lambda}}{x!}$, $x = 0,1,2,\dots$. Score and information in $\lambda$.

Mean-zero check: $\mathbb{E}_\lambda[X] = \lambda$, so $\mathbb{E}_\lambda[U(\lambda;X)] = \lambda/\lambda - 1 = 0$. Good. Variance of the score: $\operatorname{Var}_\lambda(U(\lambda;X)) = \operatorname{Var}_\lambda(X)/\lambda^2 = \lambda/\lambda^2 = 1/\lambda$. Curvature check: $\partial_\lambda U = -x/\lambda^2$, so $I(\lambda) = -\mathbb{E}_\lambda[-X/\lambda^2] = \lambda/\lambda^2 = 1/\lambda$. Both agree: $I(\lambda) = 1/\lambda$. Now suppose we instead parameterize by $\eta = \lambda^2$ and ask for the information about $\eta$, illustrating that Fisher information is <strong>not</strong> parameterization-invariant in absolute terms — it transforms. Write $\lambda = \sqrt{\eta}$, so the density as a function of $\eta$ is $g(x;\eta) = f(x;\sqrt\eta)$, and

The score in $\eta$ is

Its variance is $\operatorname{Var}_\eta(X)/(4\eta^2) = \lambda/(4\eta^2) = \sqrt\eta/(4\eta^2) = 1/(4\eta^{3/2})$. Substituting $\eta = \lambda^2$ gives $I(\eta) = 1/(4\lambda^3)$. Sanity check via the change-of-variables rule $I(\eta) = I(\lambda)\left(\dfrac{d\lambda}{d\eta}\right)^2$ (a consequence of the chain rule applied to the score, provable directly from the definition): with $\lambda = \sqrt\eta$, $d\lambda/d\eta = \tfrac{1}{2\sqrt\eta} = \tfrac{1}{2\lambda}$, so

matching. This example shows numerically that Fisher information depends on the chosen parameterization — a subtlety often glossed over, and one reason the Cramér–Rao bound must always be quoted with the parameter it is measuring information about.