Regression Inference

Regression inference is the body of statistical methods used to draw conclusions about population regression parameters (slopes, intercepts, and predicted means) from a sample of data. Its key result: under the standard linear model assumptions, the least-squares estimators are normally distributed (or approximately so), enabling confidence intervals, hypothesis tests, and prediction intervals for the unknown true relationship. It applies wherever one models how a response variable depends on one or more predictors. Developed by Galton, Pearson, Fisher, Gauss, and others from the 1800s through the 20th century.

Regression inference takes the descriptive machinery of fitting a line (or hyperplane) to data and adds a probabilistic layer that quantifies uncertainty. The fitted slope $\hat{β}_{1}$ is a random variable whose value would differ if we drew a new sample; inference asks what the data tell us about the true slope $β_{1}$ governing the population. This matters because almost every empirical claim of the form "X affects Y by a certain amount" is, at bottom, a statement about a regression coefficient and the confidence we have in it. A surprising fact: even though the response $Y$ is noisy and never lies exactly on the line, the least-squares estimators are unbiased and—under normal errors—have exactly known distributions, so we can make precise probabilistic statements from finite, messy data.

The Core Insight

The estimated coefficients are statistics computed from random data, so they have sampling distributions. Once we know the mean and variance (and shape) of those distributions, every inferential question—Is the slope nonzero? What range plausibly contains it? What range will a future observation fall in?—becomes an exercise in standardizing to a $t$ or $z$ variable. Inference is the bridge from one observed line to a statement about the whole family of lines that could have produced the data.

Formal Statement

The Simple Linear Regression Model

We assume the data $(x_{i}, y_{i})$ , $i = 1, \dots, n$ , are generated by

Y_{i} = β_{0} + β_{1} x_{i} + ε_{i}, ε_{i} \sim iid N (0, σ^{2}) .

⟳this is an assumption (the model), not a derivation; the

x_{i}

are treated as fixed/known constants and all randomness enters through

ε_{i}

Symbols:

$Y_{i}$ — the random response for observation $i$ .
$x_{i}$ — the (fixed) predictor value.
$β_{0}$ — true intercept (population mean of $Y$ when $x = 0$ ).
$β_{1}$ — true slope (change in mean $Y$ per unit $x$ ).
$ε_{i}$ — random error; assumed independent, mean zero, constant variance $σ^{2}$ , normally distributed.
$σ^{2}$ — common error variance (usually unknown, estimated).

The assumptions (mnemonic LINE): Linearity of the mean, Independence of errors, Normality of errors, Equal variance (homoscedasticity).

Least-Squares Estimators

The estimators minimizing $\sum_{i} (y_{i} - b_{0} - b_{1} x_{i})^{2}$ are

\hat{β}_{1} = \frac{\sum _{i} ( x _{i} - x ˉ ) ( y _{i} - y ˉ )}{\sum _{i} ( x _{i} - x ˉ ) ^{2}} = \frac{S _{x y}}{S _{xx}}, \hat{β}_{0} = \overset{y}{ˉ} - \hat{β}_{1} \overset{x}{ˉ} .

⟳set the two partial derivatives of the sum of squared residuals with respect to

b_{0}

and

b_{1}

to zero (the normal equations) and solve the resulting

2 \times 2

linear system.

Here $S_{x y} = \sum (x_{i} - \overset{x}{ˉ}) (y_{i} - \overset{y}{ˉ})$ and $S_{xx} = \sum (x_{i} - \overset{x}{ˉ})^{2}$ .

Sampling Distribution of the Slope

Under model [1], the slope estimator satisfies

\hat{β}_{1} \sim N (β_{1}, \frac{σ ^{2}}{S _{xx}}) .

⟳write

\hat{β}_{1} = \sum_{i} c_{i} Y_{i}

with weights

c_{i} = (x_{i} - \overset{x}{ˉ}) / S_{xx}

; then

\hat{β}_{1}

is a linear combination of independent normals, so it is normal with mean

\sum c_{i} (β_{0} + β_{1} x_{i}) = β_{1}

and variance

σ^{2} \sum c_{i}^{2} = σ^{2} / S_{xx}

Because $σ^{2}$ is unknown, we estimate it by the mean squared error

s^{2} = \frac{\sum _{i} ( y _{i} - y ^ _{i} ) ^{2}}{n - 2} = \frac{SSE}{n - 2},

⟳dividing by

n - 2

(degrees of freedom = sample size minus the two estimated parameters) makes

s^{2}

unbiased for

σ^{2}

; the loss of two df comes from estimating

β_{0}

and

β_{1}

and the resulting standardized slope follows a $t$ distribution:

T = \frac{β ^ _{1} - β _{1}}{s / S _{xx}} = \frac{β ^ _{1} - β _{1}}{SE ( β ^ _{1} )} \sim t_{n - 2} .

⟳a standard normal (from [3]) divided by

(independent χ_{n - 2}^{2}) / (n - 2)

yields a

t

; here

SSE / σ^{2} \sim χ_{n - 2}^{2}

and is independent of

\hat{β}_{1}

(Cochran's theorem).

Gauss–Markov

Among all linear unbiased estimators of $β_{0}$ and $β_{1}$ , the least-squares estimators have the smallest variance. They are the Best Linear Unbiased Estimators (BLUE). This requires only mean-zero, uncorrelated, equal-variance errors—normality is not needed for Gauss–Markov, only for the exact $t$ distribution.

Why It Holds / Derivation

Confidence interval for the slope

Step 1. From [3],

\frac{β ^ _{1} - β _{1}}{σ / S _{xx}} \sim N (0, 1)

— exact normality of the standardized slope. Step 2. From [4],

\frac{( n - 2 ) s ^{2}}{σ ^{2}} \sim χ_{n - 2}^{2}

, independent of

\hat{β}_{1}

. Step 3. The ratio of Step 1's normal to

Step 2 / (n - 2)

cancels the unknown

σ

and gives the pivot [5]:

T \sim t_{n - 2}

. Step 4. A pivot has a distribution free of unknown parameters, so for

t^{*} = t_{n - 2, 1 - α /2}

P (- t^{*} \leq \frac{β ^ _{1} - β _{1}}{SE ( β ^ _{1} )} \leq t^{*}) = 1 - α .

Step 5. Solve the inequality for

β_{1}

to obtain the interval

\hat{β}_{1} \pm t_{n - 2, 1 - α /2} \cdot \frac{s}{S _{xx}} .

⟳algebraically isolate

β_{1}

inside the probability statement; the random endpoints define a

(1 - α)

confidence interval.

Step 6. To test

H_{0} : β_{1} = 0

(no linear relationship), set

β_{1} = 0

in [5] and compute the test statistic

t = \frac{β ^ _{1}}{SE ( β ^ _{1} )},

⟳under

H_{0}

the numerator of the pivot becomes

\hat{β}_{1} - 0

; compare

t

to the

t_{n - 2}

distribution to obtain a p-value.

rejecting $H_{0}$ when $∣ t ∣$ exceeds the critical value. $■$

Inference for the mean response and prediction

For a target predictor value $x_{0}$ , the fitted value is $\overset{y}{^}_{0} = \hat{β}_{0} + \hat{β}_{1} x_{0}$ , with

Var (\overset{y}{^}_{0}) = σ^{2} [\frac{1}{n} + \frac{( x _{0} - x ˉ ) ^{2}}{S _{xx}}] .

⟳write

\overset{y}{^}_{0}

as a linear combination of the

Y_{i}

, or propagate variances of

\hat{β}_{0}

and

\hat{β}_{1}

accounting for their covariance; the term grows quadratically as

x_{0}

departs from

\overset{x}{ˉ}

A confidence interval for the mean response $E [Y ∣ x_{0}]$ uses [8]. A prediction interval for a new individual observation adds the irreducible error $σ^{2}$ :

Var (pred error) = σ^{2} [1 + \frac{1}{n} + \frac{( x _{0} - x ˉ ) ^{2}}{S _{xx}}] .

⟳the new observation

Y_{0} = β_{0} + β_{1} x_{0} + ε_{0}

contributes its own independent error

ε_{0}

with variance

σ^{2}

, added to the estimation variance from [8].

Replacing $σ$ by $s$ and using $t_{n - 2}$ gives the intervals

\overset{y}{^}_{0} \pm t_{n - 2, 1 - α /2} s \frac{1}{n} + \frac{( x _{0} - x ˉ ) ^{2}}{S _{xx}} (mean), \overset{y}{^}_{0} \pm t_{n - 2, 1 - α /2} s 1 + \frac{1}{n} + \frac{( x _{0} - x ˉ ) ^{2}}{S _{xx}} (prediction) .

⟳substitute

s

for

σ

in [8] and [9], take square roots for standard errors, and form the usual

t

-based interval.

Ronald A. Fisher | 1890–1962

Fisher gave regression inference its modern rigor. He clarified degrees of freedom, developed the analysis of variance that underlies the regression $F$ -test, derived exact small-sample distributions building on Gosset's $t$ , and championed maximum likelihood and significance testing. The decomposition of total variation into explained and residual sums of squares—and its interpretation through $F$ and $t$ statistics—is largely his legacy, refined at Rothamsted Experimental Station in the 1920s.

Building Intuition

Picture drawing your sample over and over. Each sample yields a slightly different best-fit line, pivoting around the center of mass $(\overset{x}{ˉ}, \overset{y}{ˉ})$ . The spread of those slopes is the standard error. Equation [3] says this spread shrinks when (a) the noise $σ$ is small, (b) the sample is large, or (c) the $x$ -values are widely spread out (large $S_{xx}$ )—a long lever arm makes the slope easy to pin down. The confidence and prediction bands flare outward at the extremes of $x$ (the $(x_{0} - \overset{x}{ˉ})^{2}$ term in [8]–[10]) because a small wobble in the slope swings the line most where it is farthest from the pivot. Finally, the prediction band is always wider than the confidence band because predicting one noisy individual is harder than estimating an average.

Regression line with confidence (inner) and prediction (outer) bands. Bands are narrowest at the mean of x and flare outward at the extremes.

Worked Examples

Slope test and confidence interval

Given:

n = 12

observations. Computed quantities:

S_{xx} = 200

S_{x y} = 160

\overset{x}{ˉ} = 5

\overset{y}{ˉ} = 14

, and residual sum of squares

SSE = 72

. Step 1 — Slope:

\hat{β}_{1} = S_{x y} / S_{xx} = 160/200 = 0.80

. Step 2 — Intercept:

\hat{β}_{0} = \overset{y}{ˉ} - \hat{β}_{1} \overset{x}{ˉ} = 14 - 0.80 (5) = 10.0

. Step 3 — Error variance:

s^{2} = SSE / (n - 2) = 72/10 = 7.2

, so

s = 2.683

. Step 4 — Standard error of slope:

SE (\hat{β}_{1}) = s / S_{xx} = 2.683/ 200 = 2.683/14.14 = 0.1897

. Step 5 — Test $H_{0} : β_{1} = 0$ :

t = 0.80/0.1897 = 4.22

n - 2 = 10

df. Since

t_{10, 0.975} = 2.228

, we reject

H_{0}

\approx 0.0018

). Step 6 — 95% CI for slope:

0.80 \pm 2.228 (0.1897) = 0.80 \pm 0.423

\hat{β}_{1} = 0.80, t = 4.22, 95% CI for β_{1} = (0.377, 1.223) [units: Δ y /Δ x]

Confidence vs. prediction interval at a point

Given: Same fit as above (

\hat{β}_{0} = 10

\hat{β}_{1} = 0.80

s = 2.683

n = 12

\overset{x}{ˉ} = 5

S_{xx} = 200

). Predict at

x_{0} = 8

. Step 1 — Point estimate:

\overset{y}{^}_{0} = 10 + 0.80 (8) = 16.4

. Step 2 — Leverage term:

\frac{1}{n} + \frac{( x _{0} - x ˉ ) ^{2}}{S _{xx}} = \frac{1}{12} + \frac{( 8 - 5 ) ^{2}}{200} = 0.0833 + 0.045 = 0.1283

. Step 3 — Mean-response SE:

s 0.1283 = 2.683 (0.3582) = 0.961

. With

t_{10, 0.975} = 2.228

: margin

= 2.141

. Step 4 — 95% CI for mean response:

16.4 \pm 2.141 = (14.26, 18.54)

. Step 5 — Prediction SE:

s 1 + 0.1283 = 2.683 (1.0623) = 2.850

; margin

= 2.228 (2.850) = 6.351

. Step 6 — 95% prediction interval:

16.4 \pm 6.351 = (10.05, 22.75)

Mean E [Y ∣ 8] \in (14.26, 18.54); New Y \in (10.05, 22.75)

The prediction interval is far wider, reflecting the extra

σ^{2}

of an individual outcome.

Common Misconceptions

A confidence interval is not a prediction interval. The CI in [10] (left) covers the average

Y

x_{0}

; the PI (right) covers a single future

Y

. Reporting the narrow CI when you mean to predict an individual outcome dramatically overstates precision. A high $R^{2}$ does not validate inference.

R^{2}

measures variance explained, not whether the LINE assumptions hold or whether

β_{1}

is significant. You can have a tiny

R^{2}

with a highly significant slope (large

n

) or a large

R^{2}

with badly nonlinear residuals. "Fail to reject" $\neq =$ "the slope is zero." A non-significant

t

means insufficient evidence, not proof of no effect—especially with small

S_{xx}

or small

n

, where the SE is large. Extrapolation invalidates the intervals. Equations [8]–[10] are valid only over the observed range of

x

; outside it, the linearity assumption is untested and the flaring bands become meaninglessly wide and unreliable.

Significance does not imply causation or correct model

Suppose $y$ truly depends on $x$ quadratically, $y = x^{2} + ε$ , but we fit a straight line over $x \in [1, 10]$ . The fitted slope is highly significant ( $p < 0.001$ ) and the $t$ -test "confirms a relationship." Yet the inference about $β_{1}$ is about a misspecified model: the residuals show a clear U-shape, violating linearity and independence-of-trend. The valid-looking p-value is computed under assumptions that are false, so the standard errors—and hence the entire inference—are not trustworthy. Always inspect residual plots before believing a regression p-value.

Connections

Regression inference sits at the meeting point of several pillars. The $t$ pivot in [5] is the same machinery as Student's t-Distribution inference for a mean; in fact testing a single slope is a special case of the general Hypothesis Testing framework. The variance decomposition behind the overall fit ( $SST = SSR + SSE$ ) connects directly to Analysis of Variance and the $F$ -test, which for one predictor satisfies $F = t^{2}$ . Multiple predictors generalize everything through Ordinary Least Squares in matrix form, $\hat{β} = (X^{T} X)^{- 1} X^{T} Y$ , with covariance $σ^{2} (X^{T} X)^{- 1}$ . The estimator-optimality result is the Gauss-Markov Theorem, while the route to exact distributions when errors are normal is Maximum Likelihood Estimation (least squares = MLE under normal errors). The correlation coefficient and $R^{2}$ link to Pearson Correlation.

Real-World Applications

Economics — wage equations. Regress log-wage on years of education; the slope estimates the percentage return per year of schooling, with a CI quantifying uncertainty (Mincer earnings function).
Medicine — dose-response. Estimate how a biomarker changes per unit drug dose; prediction intervals bound an individual patient's expected response in clinical trials.
Engineering — calibration curves. Instrument readings vs. known concentrations; inverse prediction with [10] gives the uncertainty of an inferred concentration in analytical chemistry.
Climate science — trend detection. Regress global temperature on year; the slope's confidence interval determines whether a warming trend is statistically distinguishable from zero.
Finance — the CAPM beta. Regress a stock's excess return on the market's; $\hat{β}_{1}$ is the asset's market beta, and its SE indicates how precisely systematic risk is measured.
Public health — air quality and mortality. Slope of mortality rate on pollutant concentration, with inference guiding regulatory thresholds.
Agriculture — yield response. Crop yield vs. fertilizer rate at Rothamsted-style field trials, the original setting Fisher developed these methods for.

Historical Development

1805–1809 — A.-M. Legendre (1805) publishes the method of least squares; C. F. Gauss (1809) claims prior use and links it to the normal error law.
1823 — Gauss proves the Gauss–Markov theorem (optimality of least squares among linear unbiased estimators) in his Theoria Combinationis.
1885–1889 — Francis Galton studies heredity, coins "regression toward mediocrity" (the regression effect) while comparing parents' and children's heights.
1896 — Karl Pearson formalizes the correlation coefficient and the mathematics of regression, founding biometry.
1908 — W. S. Gosset ("Student") derives the $t$ -distribution, enabling exact small-sample inference for means and, later, slopes.
1918–1925 — R. A. Fisher introduces analysis of variance, degrees of freedom, and the $F$ -distribution; Statistical Methods for Research Workers (1925) systematizes regression inference.
1933–1938 — Neyman and Pearson develop the hypothesis-testing framework (Type I/II errors, power); Neyman formalizes confidence intervals (1937).
Mid-20th century — Matrix formulation of multiple regression and the general linear model become standard; computing makes routine inference feasible.
1960s–1980s — Diagnostics (residual analysis, leverage, Cook's distance), robust and heteroscedasticity-consistent standard errors extend inference beyond the classical LINE assumptions.

Subtleties & Limits

Classical regression inference rests on assumptions that real data routinely break, and handling these robustly remains an active concern:

Heteroscedasticity and autocorrelation. When $Var (ε_{i})$ varies or errors are correlated (time series), the SEs in [3]–[10] are wrong. White's heteroscedasticity-consistent ("sandwich") estimators and Newey–West corrections patch this, but choosing among them and ensuring valid coverage in finite samples is non-trivial.
Errors-in-variables. If $x$ is itself measured with noise, $\hat{β}_{1}$ is biased toward zero (attenuation); the standard inference is invalid and identification requires instrumental variables or structural assumptions.
Selective inference / data snooping. Building a model from the data and then testing on the same data invalidates p-values. Post-selection inference (e.g., the work of Berk, Lockhart, Taylor, Tibshirani on selective inference and the LASSO) seeks valid intervals after model selection—an area still developing.
Non-normal, heavy-tailed errors. The $t$ pivot is exact only under normality; the central limit theorem rescues large-sample inference, but small samples with outliers can mislead, motivating robust/quantile regression.
Causal interpretation. A precise CI for $β_{1}$ says nothing about causation absent randomization or a credible identification strategy—confounding remains the deepest practical limit.