The Gauss-Markov Theorem

The Gauss–Markov theorem underlies the standard justification for ordinary least squares in econometrics, where it licenses OLS as the default estimator whenever errors are plausibly homoscedastic and uncorrelated, and motivates diagnostic testing (for heteroscedasticity, autocorrelation) precisely because violations break the BLUE guarantee. In experimental design and analysis of variance, the same linear-model framework and theorem justify treatment-effect estimators in randomized experiments. In signal processing and control theory, the theorem is the static-in-time special case of the Kalman filter's optimality proof, where the recursive linear estimator is shown to be the minimum-variance linear unbiased estimator of a dynamically evolving state given noisy measurements. In geodesy and surveying — Gauss's own original application — least-squares adjustment of over-determined measurement networks (leveling networks, GPS trilateration) still invokes this exact variance-optimality argument. In numerical linear algebra and machine learning, the theorem is the benchmark against which regularized estimators (ridge regression, LASSO) are compared: those methods deliberately introduce bias to reduce variance beyond what Gauss–Markov permits within the unbiased class, and understanding why they can beat OLS in mean squared error requires first understanding exactly what Gauss–Markov does and does not promise.

Ask the tutor: How does the Gauss–Markov theorem change if we use Generalized Least Squares instead of OLS?

Think of $\hat\beta$ as one particular linear rule for reading off coefficients from data, built from the matrix $A_0 = (X^\top X)^{-1}X^\top$. There are infinitely many other linear rules $\tilde\beta = AY$; the unbiasedness constraint $\mathbb{E}[AY] = A X\beta = \beta$ for all $\beta$ forces $AX = I_p$, which is a strong linear constraint on $A$ but still leaves an entire affine family of valid matrices (any $A$ of the form $A_0 + D$ where $DX = 0$). Each such $A$ defines a legitimate unbiased linear estimator, and the theorem says that padding $A_0$ with any nonzero "$D$ satisfying $DX=0$" can only inflate variance, never shrink it. Geometrically, $D X = 0$ means the rows of $D$ live in the orthogonal complement (with respect to the ordinary inner product on $\mathbb{R}^n$) of the column space of $X$ — the same subspace that contains the OLS residuals. Adding such a $D$ to $A_0$ injects "extra noise" that is uncorrelated with the useful signal captured by $A_0$, and uncorrelated terms add variance rather than cancel it. This is exactly the Pythagorean-like decomposition the proof makes precise.

Example 1 — Simple linear regression with an asymmetric design, comparing OLS to the sample-mean-of-slopes competitor. Consider $n = 4$ observations of a simple regression $Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$ with design points

deliberately unequally spaced (asymmetric) so that no cancellation simplifies the algebra. Assume $\mathrm{Cov}(\varepsilon) = \sigma^2 I_4$ with $\sigma^2 = 2$. The design matrix is

Compute

So

The OLS estimator of $\beta_1$ (the slope) has row vector

Compute entrywise: $-12(1,1,1,1) = (-12,-12,-12,-12)$ and $4(0,1,3,8) = (0,4,12,32)$. Sum:

Check unbiasedness-consistency: $b^\top \mathbf{1} = \frac{1}{152}(-12-8+0+20) = 0$ ✓ (intercept-orthogonality), and $b^\top x = \frac{1}{152}(-12\cdot 0 -8\cdot1+0\cdot3+20\cdot8) = \frac{1}{152}(-8+160)=\frac{152}{152}=1$ ✓, confirming $b^\top X = (0,1)$ as required for unbiasedly estimating $\beta_1$. Its variance (using $\mathrm{Cov}(\varepsilon)=\sigma^2 I$, so $\mathrm{Var}(b^\top Y) = \sigma^2 \|b\|_2^2$):

Simplify: $608/23104 = 19/722$ (dividing numerator and denominator by 32). So

Indeed $361 = 19^2$, so $\mathrm{Var}(\hat\beta_1) = \dfrac{1}{19} \approx 0.0526$.

A genuinely different linear unbiased competitor. Consider the two-point-slope estimator using only the two extreme observations:

This is linear in $Y$ with weight vector $t^\top = (-\tfrac18, 0, 0, \tfrac18)$. Check unbiasedness: $t^\top X = (-\tfrac18+\tfrac18,\ -\tfrac18\cdot 0 + \tfrac18\cdot 8) = (0, 1)$ ✓, so $\mathbb{E}[\tilde\beta_1] = \beta_1$ for all $\beta$: it is a legitimate linear unbiased competitor. Its variance: $\mathrm{Var}(\tilde\beta_1) = \sigma^2\|t\|_2^2 = 2\left(\tfrac{1}{64}+\tfrac{1}{64}\right) = 2\cdot\tfrac{2}{64} = \tfrac{4}{64} = \tfrac{1}{16} = 0.0625$. Compare: $\mathrm{Var}(\hat\beta_1) = 1/19 \approx 0.05263 < 1/16 = 0.0625 = \mathrm{Var}(\tilde\beta_1)$, exactly as the theorem guarantees — OLS beats the two-point estimator, using information from all four points rather than only the extremes.

Example 2 — Verifying the exact decomposition $\mathrm{Cov}(\tilde\beta)=\mathrm{Cov}(\hat\beta)+\sigma^2DD^\top$ numerically. Reuse the design above but now estimate the full vector $\beta=(\beta_0,\beta_1)$. Take the competitor to be OLS applied to only the last three points padded appropriately is awkward in vector form, so instead build an explicit $D$ satisfying $DX=0$ directly and confirm the variance inflation formula, which is the mechanism underlying the whole theorem. Since $DX = 0$ means each row of $D$ is orthogonal (dot product zero) to both columns of $X$ — i.e. orthogonal to $\mathbf{1} = (1,1,1,1)^\top$ and to $x = (0,1,3,8)^\top$ — we need vectors in $\mathbb{R}^4$ orthogonal to both. The orthogonal complement of $\mathrm{span}\{\mathbf 1, x\}$ in $\mathbb{R}^4$ is 2-dimensional. Find one such vector $d$ by solving $d\cdot\mathbf1=0$, $d\cdot x=0$. Try $d = (a,b,c,e)$ with

Pick $e=0$, $c=1$: then $b=-3$, and $a = -b-c-e = 3-1=2$. So $d=(2,-3,1,0)$. Check: sum $=2-3+1+0=0$ ✓; weighted sum $=0(2)+1(-3)+3(1)+8(0) = -3+3=0$ ✓. Let $D$ be the $2\times4$ matrix with first row $d^\top=(2,-3,1,0)$ scaled by a constant $\lambda$ and second row zero (we perturb only the intercept-estimating row, to keep the slope estimator equal to OLS's and isolate the effect on the intercept):

Then $A = (X^\top X)^{-1}X^\top + D$ still satisfies $AX = I_2$ (since $DX=0$), so $\tilde\beta = AY$ is a valid linear unbiased estimator of $(\beta_0,\beta_1)$ for any $\lambda\in\mathbb{R}$. The OLS row for the intercept is (first row of $(X^\top X)^{-1}X^\top$):

Check: $a_0^\top \mathbf1 = (74+62+38-22)/152 = 152/152=1$ ✓; $a_0^\top x = (0+62+114-176)/152 = 0/152=0$ ✓, confirming this row correctly estimates $\beta_0$ unbiasedly. $\|a_0\|_2^2 = (74^2+62^2+38^2+22^2)/152^2 = (5476+3844+1444+484)/23104 = 11248/23104$. Simplify: $\gcd(11248,23104)$. $11248 = 16\cdot703$, $23104=16\cdot1444$, so ratio $=703/1444$. Since $1444=38^2$ and $703 = 19\cdot37$, this doesn't reduce further nicely; numerically $703/1444 \approx 0.48685$. So $\mathrm{Var}(\hat\beta_0) = \sigma^2\cdot 0.48685 = 2(0.48685)= 0.9737$ (using $\sigma^2=2$). Now the perturbed row is $\lambda d^\top + a_0^\top$ (rows add since $D$'s first row is $\lambda d^\top$). Its squared norm:

We already know from the general proof that the cross term $d^\top a_0$ must vanish (since $DX=0$ forces $Ba_0$-type cross terms to vanish row-wise too — let's verify directly): $d^\top a_0 = \frac{1}{152}\big(2\cdot74 + (-3)(62) + 1\cdot 38 + 0\cdot(-22)\big) = \frac{1}{152}(148 -186+38+0) = \frac{0}{152}=0.$ ✓ Confirms the cross-term cancellation proved abstractly in Step 2 of the theorem's proof. $\|d\|_2^2 = 4+9+1+0=14$. So

for every $\lambda$, with equality iff $\lambda = 0$, i.e. iff $\tilde\beta_0=\hat\beta_0$ — exactly reproducing the theorem's conclusion and equality condition in a fully explicit numerical instance. For example at $\lambda = 0.1$: $\mathrm{Var}(\tilde\beta_0) = 0.9737 + 14(2)(0.01) = 0.9737+0.28=1.2537 > 0.9737$.