Variance and Mean Squared Error

The bias-variance tradeoff encoded in the MSE decomposition is one of the most widely reused ideas in quantitative science. In statistics itself, it directly motivates the Rao-Blackwell Theorem, which shows how to reduce variance of an unbiased estimator without increasing bias by conditioning on a sufficient statistic, and the Cramer-Rao Lower Bound, which gives a hard floor on the variance (hence the MSE) achievable by any unbiased estimator. In regression, the Gauss-Markov Theorem identifies ordinary least squares as the minimum-variance unbiased linear estimator, but modern ridge regression and LASSO deliberately introduce bias via a penalty term to shrink coefficient variance and reduce total prediction MSE, especially when predictors are correlated or numerous relative to sample size. In machine learning, the identical decomposition (renamed "bias-variance tradeoff" for prediction error rather than parameter estimation) explains why simple models underfit (high bias) while overly flexible models overfit (high variance), guiding the choice of model complexity, regularization strength, and ensembling strategies such as bagging (which primarily reduces variance) and boosting (which primarily reduces bias). In signal processing and control theory, the Kalman filter's optimality criterion is minimum mean squared error, and the entire field of "MMSE estimation" is built on exactly this decomposition. In physics and engineering measurement, calibration procedures explicitly manage the tradeoff between systematic error (bias) and random error (variance) when combining or filtering noisy instrument readings. Empirical Bayes and James–Stein shrinkage estimators, which provably beat the sample mean in mean squared error for three or more simultaneously estimated parameters, are a striking and famous demonstration that unbiasedness can be strictly suboptimal even in the simplest Gaussian models.

:::quiz[Why does the proof of the bias-variance decomposition require $\mathbb{E}_\theta[\hat\theta^2] < \infty$?]

• It is a stylistic convenience with no mathematical necessity
• It guarantees the mean and variance of $\hat\theta$ exist as finite numbers so the algebraic expansion and linearity-of-expectation steps are valid ✓
• It is needed only to compute the bias, not the variance
• It ensures $\hat\theta$ is unbiased

Ask the tutor: Can you show me how the bias-variance decomposition extends to derive the Cramér-Rao lower bound?

Picture an archer shooting arrows at a target, where the bullseye is the true parameter $\theta$ and each arrow is one realization of the estimator $\hat\theta$ computed from a fresh sample. Two failure modes are possible. The shots can be scattered widely around the bullseye (high variance, low bias), or they can be tightly clustered but consistently off-center (low variance, high bias). MSE is exactly the average of the squared distances from the bullseye over many shots, and the bias-variance decomposition says this average splits cleanly into "how far is the center of the cluster from the bullseye" (bias squared) plus "how spread out is the cluster around its own center" (variance). This reframes the entire question of estimator quality. An estimator is not disqualified merely because its shots are off-center on average; if the cluster is tight enough, a slightly off-center but tight cluster beats a centered but wildly scattered one. The art of point estimation, and much of modern statistics (ridge regression, James-Stein shrinkage, regularized machine learning), is precisely about deliberately accepting a controlled amount of bias in exchange for a larger reduction in variance, so that the total MSE goes down.

Example 1: Estimating a normal variance — biased MLE versus unbiased sample variance. Let $X_1,\dots,X_n$ be i.i.d. $N(\mu,\sigma^2)$ with both $\mu$ and $\sigma^2$ unknown, and let $\bar X = \frac1n\sum_{i=1}^n X_i$. Consider two estimators of $\sigma^2$:

It is a standard fact (proved via the Bias of an Estimator framework) that $S^2$ is unbiased, $\mathbb{E}[S^2] = \sigma^2$, while $\hat\sigma^2_{\text{MLE}} = \frac{n-1}{n}S^2$ is biased, with

For normal samples, the exact sampling distribution gives $\frac{(n-1)S^2}{\sigma^2} \sim \chi^2_{n-1}$, and since a $\chi^2_{k}$ random variable has variance $2k$,

Since $S^2$ is unbiased, $\text{MSE}(S^2) = \text{Var}(S^2) = \dfrac{2\sigma^4}{n-1}$. Now compute $\text{MSE}(\hat\sigma^2_{\text{MLE}})$. Since $\hat\sigma^2_{\text{MLE}} = \frac{n-1}{n}S^2$,

By the Bias–Variance Decomposition of MSE,

Compare the two MSEs by cross-multiplying (both denominators positive):

We ask whether $\dfrac{2n-1}{n^2} < \dfrac{2}{n-1}$, i.e. whether $(2n-1)(n-1) < 2n^2$. Expand: $(2n-1)(n-1) = 2n^2 - 3n + 1$, so the inequality becomes $2n^2 - 3n + 1 < 2n^2$, i.e. $-3n+1 < 0$, i.e. $n > 1/3$, which holds for every $n \ge 2$ (the smallest sample size for which $S^2$ is defined).

Conclusion: for every $n \ge 2$, the biased MLE $\hat\sigma^2_{\text{MLE}}$ has strictly smaller MSE than the unbiased $S^2$. For a concrete number take $n = 5$, $\sigma^2 = 1$: $\text{MSE}(S^2) = 2/4 = 0.5$, while $\text{MSE}(\hat\sigma^2_{\text{MLE}}) = 9/25 = 0.36$. Unbiasedness alone did not deliver the smaller total error; the small sacrifice in bias ($-\sigma^2/5$) was more than repaid by the reduction in variance.

Example 2: A shrinkage estimator of a Bernoulli mean. Let $X_1,\dots,X_n$ be i.i.d. $\text{Bernoulli}(p)$, $p \in (0,1)$ unknown, and let $\hat p = \bar X = \frac1n\sum X_i$ be the usual sample proportion, which is unbiased with $\text{Var}(\hat p) = p(1-p)/n$, so $\text{MSE}(\hat p) = p(1-p)/n$. Consider the shrinkage estimator that pulls $\hat p$ toward $1/2$:

Its bias is

so $\text{Bias}(\tilde p) = (1-\lambda)(\tfrac12 - p)$, nonzero unless $\lambda = 1$ or $p = 1/2$. Its variance is

By the decomposition,

Take a concrete asymmetric case: $n = 20$, true $p = 0.3$, and compare $\lambda = 1$ (the plain unbiased $\hat p$) against $\lambda = 0.8$ (shrunk toward $1/2$). For $\lambda = 1$: $\text{MSE}(\hat p) = \dfrac{0.3 \times 0.7}{20} = \dfrac{0.21}{20} = 0.0105.$ For $\lambda = 0.8$: variance term $= (0.8)^2 \cdot \dfrac{0.21}{20} = 0.64 \times 0.0105 = 0.00672$; bias term $= (0.2)^2(0.5-0.3)^2 = 0.04 \times 0.04 = 0.0016$. Total:

Since $0.00832 < 0.0105$, the shrunk, biased estimator has smaller MSE at $p=0.3$ for this $\lambda$. (Had the true $p$ been very far from $1/2$, e.g. $p = 0.05$, the bias term would dominate and shrinkage toward $1/2$ would instead hurt: at $p=0.05$, $\lambda=0.8$ gives variance term $0.64\times \frac{0.05\times0.95}{20} = 0.00152$ and bias term $0.04\times(0.45)^2 = 0.0081$, total $0.00962$, versus $\text{MSE}(\hat p) = 0.05\times0.95/20 = 0.002375$ for $\lambda=1$ — here shrinkage is worse.) This illustrates that the optimal tradeoff depends on the unknown truth, which is precisely why shrinkage constants are typically tuned by cross-validation or Bayesian priors rather than fixed blindly.