Inference for Regression

Suppose we fit the simple linear regression model

where $\beta_0$ and $\beta_1$ are fixed but unknown population parameters, and $\varepsilon_i$ are random errors assumed independent, normally distributed, with mean 0 and constant variance $\sigma^2$. The observed data give least-squares estimates $b_0$ and $b_1$, computed exactly as in The Least-Squares Line and Residuals by minimizing the sum of squared residuals. Crucially, $b_1$ is a statistic: if you drew a fresh sample of $n$ pairs $(x_i, y_i)$ from the same population, you would get a slightly different $b_1$. Its sampling distribution, under the model assumptions, is approximately normal (exactly normal if the errors themselves are normal), centered at the true $\beta_1$, with standard error

Here $s$ is the residual standard deviation (estimating $\sigma$), and $s_x$ is the sample standard deviation of the $x$-values. Two things this formula makes precise, and both are biologically intuitive: more scatter around the line ($s$ large) inflates the standard error, while a wider spread of $x$-values ($s_x$ large) shrinks it — spreading your predictor levels out gives more leverage to pin down the slope, which is why well-designed experiments deliberately space treatment levels widely rather than clustering them. Because $\sigma^2$ is estimated (not known), inference uses the t-distribution with $n-2$ degrees of freedom — one degree of freedom is spent estimating $\beta_0$, another estimating $\beta_1$, mirroring exactly how the correlation coefficient's significance test in Inference for Correlation also runs on $n-2$ df. This shared machinery is not a coincidence: testing $H_0: \rho = 0$ and testing $H_0: \beta_1 = 0$ are algebraically equivalent for simple linear regression, producing the identical $t$-statistic and $p$-value, because a slope of zero and a correlation of zero are the same statement about independence between $X$ and $Y$.

Step 1 — Partition the variability. Total variation in $y$ splits into variation explained by the line and leftover residual variation:

SSE, divided by $n-2$, gives the residual variance $s^2$, the estimate of the "noise floor" $\sigma^2$ around the true line.

Step 2 — Compute the standard error of the slope. Using $s$ from Step 1 and the spread of $x$, $SE_{b_1} = s/(s_x\sqrt{n-1})$. This quantity answers: if I replicated this study many times with the same $x$-design, how much would my estimated slope typically vary from the truth?

Step 3 — Standardize into a t-statistic. For the hypothesis test of no linear relationship,

on $n - 2$ degrees of freedom. Large $|t|$ means the observed slope is many standard errors from zero — implausible under the null of no relationship.

Step 4 — Build the confidence interval.

This interval is the set of hypothesized slopes that the data would not reject at level $\alpha$ — the duality between confidence intervals and two-sided tests that runs throughout inferential statistics.

Step 5 — Distinguish two different predictive questions. Once the line is fit, people often want to predict a $y$-value at some new $x_0$. There are two distinct targets:

• The mean response $\mu_{Y|x_0} = \beta_0 + \beta_1 x_0$ — "what is the average $y$ for all individuals with this $x$?"
• A single future observation $y_0$ at $x_0$ — "what will one particular new individual's $y$-value be?"

Both are estimated by the same point $\hat y_0 = b_0 + b_1 x_0$, but their uncertainties differ. The confidence interval for the mean only has to account for uncertainty in estimating the line:

The prediction interval for one new observation must additionally account for that individual's own scatter around the (also uncertain) line:

The extra "$1$" under the square root is the signature of the prediction interval, and it is always wider — sometimes dramatically so — than the confidence interval for the mean at the same $x_0$. Both intervals flare outward as $x_0$ moves away from $\bar x$, forming the characteristic "bowtie" shape around the fitted line.

Step 6 — Verify assumptions before trusting any of it. All the standard errors above are derived under LINE:

• Linearity — the true relationship between $E[Y]$ and $x$ is a straight line.
• Independence — errors $\varepsilon_i$ are independent across observations (violated by repeated measures, spatial or temporal autocorrelation, litter/clutch clustering).
• Normality — errors are normally distributed (matters most for small samples; large-$n$ inference is robust by the Central Limit Theorem).
• Equal variance (homoscedasticity) — $\sigma^2$ does not change with $x$.

These are checked with residual plots: residuals ($y_i - \hat y_i$) plotted against fitted values or against $x$. A random, structureless scatter around zero supports the assumptions; a curved pattern flags nonlinearity; a "funnel" (fanning) shape flags unequal variance; and a normal quantile plot of residuals checks normality directly.

Regression inference is not a single dramatic experiment like Meselson–Stahl but a body of statistical theory, developed and stress-tested across decades, whose validity rests on mathematical derivation confirmed by simulation and by consistent replication across real datasets. Two historical threads are worth knowing.

Gauss and Legendre, and the Gauss–Markov theorem (early 1800s–1821). Carl Friedrich Gauss and Adrien-Marie Legendre independently developed least squares to fit orbits of celestial bodies from noisy astronomical observations; Gauss later proved (in what is now called the Gauss–Markov theorem) that among all linear unbiased estimators of the regression coefficients, the least-squares estimator has the smallest variance — the "Best Linear Unbiased Estimator" (BLUE) property. This theorem is the mathematical bedrock that justifies using $b_1$ as our estimate of $\beta_1$ at all, and it requires only that errors have mean zero and constant variance — normality is needed specifically for the t-based confidence intervals and p-values, not for the point estimate itself.

William Sealy Gosset and the t-distribution ("Student," 1908). Working as a chemist at the Guinness brewery in Dublin, Gosset needed a way to draw conclusions from very small samples — batches of barley, tiny fermentation trials — where the normal-distribution-based methods of the day, calibrated for large samples, gave misleadingly narrow intervals. He derived the exact sampling distribution of the standardized sample mean when the population variance is estimated rather than known, publishing under the pseudonym "Student" because Guinness barred employees from publishing under their own names (to protect trade secrets). Gosset's t-distribution — heavier-tailed than the normal, converging to it as degrees of freedom grow — is precisely the distribution that governs $t = b_1/SE_{b_1}$ in regression inference. Ronald Fisher later generalized and popularized the t-test's use across experimental design, including regression and ANOVA, cementing it as the standard tool for small-sample inference in biology.

Modern validation by simulation. The claim that $b_1$'s sampling distribution is (approximately) $t_{n-2}$-shaped is now routinely confirmed by Monte Carlo simulation: generate thousands of synthetic datasets from a known linear model with specified $\beta_1$ and error distribution, compute $b_1$ and $SE_{b_1}$ for each, and check that roughly 95% of the resulting 95%-confidence intervals actually contain the true $\beta_1$. Such simulations consistently confirm the theory when LINE assumptions hold, and just as reliably show coverage failure — intervals that are too narrow or badly centered — when independence or equal-variance assumptions are violated, which is exactly why residual diagnostics are taught as a mandatory companion to the arithmetic, not an optional afterthought.

Example 1 — Genetics-adjacent physiological calculation: metabolic rate and body mass in shrews. A physiologist measures resting metabolic rate ($y$, in mL O₂/hour) against body mass ($x$, in grams) for $n = 14$ shrews. Least-squares fitting gives:

Standard error of the slope:

Test $H_0: \beta_1 = 0$ vs. $H_a: \beta_1 \neq 0$:

The critical value $t^*_{12, 0.025} \approx 2.179$. Since $4.61 > 2.179$, we reject $H_0$ at $\alpha = 0.05$ (in fact $p < 0.001$). A 95% CI for $\beta_1$:

Interpretation: for every extra gram of body mass, resting metabolic rate increases by somewhere between 0.43 and 1.21 mL O₂/hr, with 95% confidence — and the interval excludes zero, consistent with the well-established (though sublinear across broader mass ranges) mass–metabolism relationship in mammals.

Example 2 — Prediction vs. confidence interval in an ecological/clinical scenario. An agricultural scientist regresses crop yield ($y$, tons/hectare) on nitrogen fertilizer applied ($x$, kg/hectare) across $n = 22$ plots, obtaining $b_0 = 2.10$, $b_1 = 0.034$, $s = 0.51$, $\bar x = 120$ kg/ha, $s_x = 38$ kg/ha. She wants to know two different things at $x_0 = 180$ kg/ha (a relatively high, less-densely-sampled dose): Point estimate:

(a) Mean yield across all plots that would receive 180 kg/ha — a confidence interval for $\mu_{Y|180}$:

With $t^*_{20,0.025} \approx 2.086$:

(b) Yield on one particular new plot given 180 kg/ha — a prediction interval:

The prediction interval (2.30 tons wide) is more than double the confidence interval (0.86 tons wide) — the extra width is the price of predicting one noisy individual plot rather than an infinitely-replicated average, and both intervals would flare out further still if the scientist pushed $x_0$ to 300 kg/ha, well outside the sampled range of nitrogen doses — a warning sign for extrapolation discussed below.

Gosset trained in chemistry and mathematics at Oxford before joining Guinness in 1899, where he faced a distinctly practical problem: brewing experiments — comparing barley varieties, hop treatments, yeast strains — could only ever be run on small numbers of batches, yet the statistical theory of the era assumed sample sizes large enough that the sample standard deviation could be treated as if it were the true population standard deviation. Gosset noticed that this assumption badly understated uncertainty when samples were small, giving false confidence in brewing decisions. Through both theoretical derivation and, notably, empirical simulation — he generated thousands of samples from measured height and finger-length data of criminals (a dataset then available) and tabulated the resulting distribution of the standardized mean by hand — he worked out the exact small-sample sampling distribution, published in 1908 in Biometrika under the pseudonym "Student." Guinness's confidentiality policy forbade employees from publishing, fearing competitors might infer trade secrets from statistical papers, so his identity remained something of an open secret in statistical circles for years. Ronald Fisher, recognizing the importance of Gosset's work, refined the notation and proofs and built the t-test into the foundation of experimental design and analysis of variance, from which it propagated into essentially every corner of quantitative biology — including, directly, the standard error and confidence interval formulas for a regression slope used throughout this article.

Medicine and epidemiology. Dose–response regressions (e.g., drug concentration vs. blood pressure reduction) rely on slope confidence intervals to establish whether a treatment effect is distinguishable from zero, and prediction intervals inform individualized risk estimates — e.g., predicting a single patient's expected lab value given age and weight, which requires the wider prediction interval, not the confidence band for the population mean.

Agriculture and ecology. Fertilizer-response curves, growth-rate-vs-temperature relationships, and species-abundance-vs-habitat-variable regressions all use slope tests to decide whether a management variable causally tracks with an outcome, and prediction intervals to set realistic expectations (e.g., "how much yield variation should one field show") rather than overstating precision with mean-response intervals alone.

Conservation biology. Regressions of population size against habitat area, or extinction risk against fragment size, use slope confidence intervals to argue for policy thresholds; residual diagnostics are essential here because ecological data often violate independence (spatially clustered sites) or equal variance (small populations are noisier than large ones), demanding corrected standard errors.

Biotechnology and calibration curves. Standard curves in assays (e.g., ELISA absorbance vs. known concentration) are regressions whose slope and intercept convert instrument readings into concentrations; prediction intervals around the calibration line quantify the uncertainty in back-calculated concentrations for unknown samples — a routine, high-stakes application in diagnostic labs.

Public health surveillance. Trend lines fit to disease incidence over time use slope tests to determine whether an apparent rise or fall in cases is statistically distinguishable from a flat trend, informing outbreak declarations — though autocorrelation in time-series counts often violates the independence assumption, requiring more advanced time-series-aware models that extend this basic framework.

Ask the tutor: How would these formulas change if I had unequal variance across x (heteroscedasticity) — what correction should I apply?