Bias of an Estimator

Bias correction and awareness of systematic estimator error appear throughout applied quantitative work. In survey statistics and econometrics, sample weights and finite-population corrections are designed explicitly to remove bias from estimators of totals and means under complex sampling designs. In machine learning, the bias–variance decomposition of prediction error is a central diagnostic tool for choosing model complexity, regularization strength, and ensemble methods, generalizing the mean-squared-error decomposition seen here. In signal processing and control theory, biased estimators (like shrinkage or ridge estimators) are deliberately introduced to trade a small bias for a large reduction in variance, improving overall estimation accuracy — a strategy formalized later via the The Rao-Blackwell Theorem and related results on efficient estimation. In physics and metrology, calibration procedures for instruments are essentially bias-correction algorithms, ensuring that repeated measurements average to the true physical quantity. And in finance, unbiasedness of volatility or risk estimators (like sample variance of returns) directly affects the correctness of pricing and risk models, making the $n-1$ correction from this article a matter of real economic consequence.

:::quiz[If $E_\theta[\hat\theta] = \theta + 3$ for all $\theta$, what is the bias of $\hat\theta$?]

• 0, since the estimator still moves correctly with θ
• 3, a constant bias for every θ ✓
• θ, because the bias depends on the parameter
• Undefined, because bias only applies to unbiased estimators

Ask the tutor: How does the bias of an estimator interact with its variance to determine overall estimator quality via mean squared error?

Think of an estimator as an archer shooting at a target repeatedly, where the bullseye is the true parameter $\theta$. Each sample gives one arrow; the sampling distribution of $\hat\theta$ describes where arrows tend to land across many repetitions of "draw a fresh sample, compute $\hat\theta$." Bias asks only about the center of mass of all those arrow positions: is it exactly on the bullseye, or offset to one side? An unbiased archer might still scatter arrows widely (high variance) but their average landing point is dead-center. A biased archer might group arrows tightly (low variance) but consistently to the upper-left of the bullseye. This is a subtle but crucial point: bias is a statement about $E[\hat\theta]$, computed by averaging over the hypothetical repeated sampling, not a statement about how far a particular computed estimate is from $\theta$. You never observe $\theta$, so you can never directly check whether one estimate was "biased" — bias is a property of the estimation procedure, verified mathematically or via simulation, not of a single number.

Example 1 (Estimating the maximum of a uniform distribution — an asymmetric, non-obvious case). Let $X_1, \dots, X_n$ be i.i.d. $\mathrm{Uniform}(0,\theta)$ with unknown $\theta > 0$, and consider the natural estimator $\hat\theta = X_{(n)} = \max(X_1,\dots,X_n)$, the sample maximum. First find the density of $X_{(n)}$. The CDF of $X_i$ is $F(x) = x/\theta$ for $0 \le x \le \theta$. By independence, the CDF of the maximum is

Differentiating, the density is $f_{(n)}(x) = \frac{n x^{n-1}}{\theta^n}$ on $(0,\theta)$. Then

So

and $X_{(n)}$ systematically underestimates $\theta$ — it can never exceed $\theta$, so this is intuitively unsurprising, but the theorem quantifies exactly how much on average. To correct this, define

Then $E[\hat\theta^*] = \frac{n+1}{n}\cdot\frac{n}{n+1}\theta = \theta$, so $\hat\theta^*$ is unbiased. Concretely, with $n = 4$ observations and observed maximum $x_{(4)} = 7.2$, the biased estimate is $7.2$ while the bias-corrected estimate is $\frac{5}{4}(7.2) = 9.0$ — a substantial correction reflecting that with only 4 draws, the maximum tends to fall well short of $\theta$.

Example 2 (Estimating $p^2$ for a Bernoulli proportion — bias of a nonlinear function of an unbiased estimator). Let $X_1,\dots,X_n$ be i.i.d. $\mathrm{Bernoulli}(p)$, and suppose we want to estimate $\tau(p) = p^2$ (e.g. the probability two independent components both work, if $p$ is the reliability of one component). The sample proportion $\hat p = \bar X$ is unbiased for $p$: $E[\hat p] = p$. A tempting estimator for $p^2$ is $\hat p^2$. Is it unbiased for $p^2$? Recall the identity $\operatorname{Var}(\hat p) = E[\hat p^2] - (E[\hat p])^2$, so

using $\operatorname{Var}(\hat p) = \operatorname{Var}(\bar X) = \sigma^2/n = p(1-p)/n$ for Bernoulli$(p)$ (since $\sigma^2 = p(1-p)$). Thus

So $\hat p^2$ overestimates $p^2$ on average — a concrete instance of the general fact that a nonlinear (here, convex) transformation of an unbiased estimator is typically biased (a consequence of Jensen's inequality, since $g(x)=x^2$ is strictly convex). For example, with $n = 10$ and $p = 0.3$: the true value is $p^2 = 0.09$, while $E[\hat p^2] = \frac{0.3 \times 0.7}{10} + 0.09 = 0.021 + 0.09 = 0.111$, a relative overestimation of about 23%. An unbiased estimator of $p^2$ instead is $\hat p^2 - \frac{\hat p(1-\hat p)}{n-1}$ (the standard bias-corrected estimator), which can be verified to have expectation exactly $p^2$.