Residual Analysis
When we fit a regression line to data, we rarely expect the line to pass through every single point. The differences between what our model predicts and what we actually observe are called residuals. Residual analysis is the process of examining these differences to decide whether our regression model is a good fit for the data, or whether something important has been missed. A residual is calculated for each data point as the observed value minus the predicted value: e = y − ŷ. If the residuals are small and scattered randomly around zero with no obvious pattern, the linear model is likely appropriate. If the residuals show a curve, a fan shape, or clusters, the model may be flawed — perhaps the relationship is nonlinear, the variance is not constant, or an influential outlier is distorting the line. Residual analysis is a cornerstone of statistical practice. Professional statisticians, data scientists, and researchers in every field use residual plots before trusting any regression-based conclusion. Understanding residuals helps you move beyond simply calculating a line of best fit and toward critically evaluating whether that line is actually telling the truth about your data.
- Fit the Regression Line
- Calculate Every Residual
- Create a Residual Plot
- Interpret the Residual Plot
- Use the Coefficient of Determination R²
Fit the Regression Line
Begin by calculating the least-squares regression line ŷ = a + bx, where b = r(sy/sx) is the slope and a = ȳ − bx̄ is the y-intercept. This line minimises the sum of squared residuals Σ(y − ŷ)², which is why it is called least-squares regression.
Calculate Every Residual
For each data point (xᵢ, yᵢ), compute the predicted value ŷᵢ = a + bxᵢ and then the residual eᵢ = yᵢ − ŷᵢ. A positive residual means the actual value lies above the line; a negative residual means it lies below. The sum of all residuals in a least-squares fit always equals exactly zero: Σeᵢ = 0.
Create a Residual Plot
Plot the residuals eᵢ on the vertical axis against the explanatory variable xᵢ (or against ŷᵢ) on the horizontal axis. Add a horizontal reference line at e = 0. This plot is the primary diagnostic tool for regression. Do not plot residuals against the response variable y — that creates a misleading pattern even for a perfect model.
Interpret the Residual Plot
Look for three warning signs. (1) Curved pattern: residuals that bow upward or downward indicate the true relationship is nonlinear and a straight line is not appropriate. (2) Fan or funnel shape: residuals that grow larger as x increases indicate non-constant variance (heteroscedasticity), violating a key regression assumption. (3) Outliers: individual points with very large residuals (in absolute value) may be influential and could be pulling the regression line away from the bulk of the data. A good residual plot shows points scattered randomly and symmetrically around zero with roughly equal spread throughout.
Use the Coefficient of Determination R²
Alongside the residual plot, report R² = 1 − SSE/SST, where SSE = Σeᵢ² (sum of squared errors) and SST = Σ(yᵢ − ȳ)² (total variation in y). R² tells you what proportion of the variation in y is explained by the linear relationship with x. An R² of 0.85 means 85% of the variability in y is accounted for by the model. However, always pair R² with a residual plot — a high R² can still accompany a curved residual plot, revealing that the linear model is still wrong.
The Key Equation
- e_i — Residual (same as y): The vertical distance from the data point to the regression line; positive if above, negative if below.
- y_i — Observed value (same as y): The actual measured response value for the i-th observation.
- \hat{y}_i — Predicted value (same as y): The value the regression line predicts for the given xᵢ.
- a — y-intercept (same as y): The value of ŷ when x = 0; calculated as ȳ − bx̄.
- b — Slope (y-units per x-unit): The rate of change of ŷ with respect to x; calculated as r(sᵧ/sₓ).
Worked Example
Problem. A student records the number of hours studied (x) and exam score (y) for 5 students. The data are: (1, 52), (2, 58), (3, 70), (4, 73), (5, 95). The least-squares regression line is ŷ = 38.6 + 10.3x. Calculate all residuals, verify they sum to zero, and describe what the residual plot suggests about model fit.
Answer. Residuals: +3.1, −1.2, +0.5, −6.8, +4.9 (sum ≈ 0). R² = 0.926, meaning approximately 92.6% of the variability in exam scores is explained by hours studied. The residual plot shows no alarming pattern, supporting the use of a linear model.
Origins · 1 minFrancis Galton and how this was discovered
Francis Galton pioneered the concept of regression in the 1880s while studying how the heights of children relate to the heights of their parents. He noticed that children of very tall parents tended to be shorter than their parents, and children of very short parents tended to be taller — a phenomenon he called 'regression to the mean.' His collaborator Karl Pearson later formalised the mathematics of the least-squares regression line and the correlation coefficient, making residual analysis possible as a formal statistical tool. Galton was so obsessed with measurement that he once created a 'Beauty Map' of Britain by secretly rating the attractiveness of women he passed on the street in different cities, using a needle and paper in his pocket to keep a tally.
Related Concepts
- Least-Squares Regression Line — Residuals are defined relative to the least-squares regression line; you must understand how that line is constructed before you can analyse its residuals.
- Correlation Coefficient (r) — The correlation coefficient r measures the strength of the linear relationship, and R² = r² directly links correlation to the proportion of variation explained by the regression model.
- Outliers and Influential Points — Large residuals identify potential outliers, and influential points are those whose removal would dramatically change the regression line — a key concern in residual analysis.
- Scatterplots — A residual plot is a special type of scatterplot; understanding how to read and interpret scatterplots is essential for evaluating residual patterns.