Least Squares Regression

When we collect paired data — say, hours studied versus exam score — we want a mathematical model that summarises the relationship and lets us make predictions. Linear regression finds the straight line that best captures the overall trend in the data. But what does 'best' actually mean? The method of least squares gives a precise, mathematically rigorous answer: the best-fit line is the one that minimises the sum of the squared vertical distances between each data point and the line. The idea of squaring the distances (rather than simply adding them up) is crucial. Positive and negative deviations would cancel if we just summed them, giving a misleading picture. Squaring every deviation makes each contribution non-negative, so the minimisation process genuinely finds the line that is closest to all the points simultaneously. This approach also has elegant algebraic properties that produce clean, exact formulas for the slope and intercept. Least-squares regression is one of the most widely used statistical techniques in science, economics, medicine, and engineering. Understanding its properties — including the fact that the line always passes through the point of means, and that the residuals sum to zero — gives you powerful tools for interpreting data and critically evaluating claims based on statistical models.

Key takeaways

Define the residuals
Set up the objective: minimise SSE
Derive the least-squares slope
Find the intercept
Key properties of the fitted line

Define the residuals

For each data point (xᵢ, yᵢ), the residual eᵢ is the vertical gap between the observed value and the predicted value on the line: eᵢ = yᵢ − ŷᵢ = yᵢ − (a + bxᵢ). A positive residual means the point lies above the line; a negative residual means it lies below.

Set up the objective: minimise SSE

The Sum of Squared Errors (SSE) is SSE = Σeᵢ² = Σ(yᵢ − a − bxᵢ)². We choose the values of slope b and intercept a that make SSE as small as possible. This is a calculus problem: take the partial derivatives of SSE with respect to a and b, set them to zero, and solve.

Derive the least-squares slope

Setting ∂(SSE)/∂b = 0 and ∂(SSE)/∂a = 0 yields two 'normal equations'. Solving them gives the least-squares slope: b = Σ(xᵢ − x̄)(yᵢ − ȳ) / Σ(xᵢ − x̄)², which can also be written as b = Sxy / Sxx, where Sxy is the sample covariance numerator and Sxx is the sum of squared deviations of x. Equivalently, b = r · (sy / sx), where r is the correlation coefficient and sy, sx are the sample standard deviations.

Find the intercept

Once the slope b is known, the intercept follows directly from the property that the least-squares line always passes through the point of means (x̄, ȳ): a = ȳ − b · x̄. This is a key structural property — no matter what the data are, the regression line is guaranteed to pass through the centroid of the data cloud.

Key properties of the fitted line

The least-squares line has three important properties: (1) The residuals sum to zero: Σeᵢ = 0. (2) The line passes through (x̄, ȳ). (3) The mean of the fitted values equals the mean of the observed y-values: ȳ̂ = ȳ. These properties are guaranteed by the algebra of the normal equations and serve as useful checks when verifying calculations.

The Key Equation

\hat{y} = a + bx, \quad b = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n}(x_i - \bar{x})^2}, \quad a = \bar{y} - b\bar{x}

\hat{y} — Predicted value (same units as y): The value of y estimated by the regression line for a given x.
a — y-intercept (same units as y): The value of ŷ when x = 0; shifts the line up or down.
b — Slope (units of y per unit of x): The change in ŷ for each one-unit increase in x.
\bar{x}, \bar{y} — Sample means (units of x and y respectively): The arithmetic averages of the x-values and y-values; the line always passes through (x̄, ȳ).
n — Sample size (dimensionless): The number of data pairs collected.

Worked Example

Problem. A student records the number of hours spent studying (x) and the score (y) on a maths test for five classmates: (1, 52), (2, 58), (3, 65), (4, 72), (5, 78). Find the least-squares regression line and predict the score for 3.5 hours of study.

\hat{y} = 45.2 + 6.6x, \quad \hat{y}(3.5) = 68.3

Answer. The least-squares line is ŷ = 45.2 + 6.6x. The predicted score for 3.5 hours of study is 68.3.

Figure

Diagram

Origins · 1 minCarl Friedrich Gauss and how this was discovered

Carl Friedrich Gauss

1777–1855 · German

Gauss developed the method of least squares around 1795 (at age 18!) and used it to predict the orbit of the asteroid Ceres in 1801. He published a full account in his 1809 work Theoria Motus, where he also showed that if errors are normally distributed, least squares is the optimal estimation method — a result now known as the Gauss-Markov theorem. Gauss reportedly kept his discovery of least squares secret for years. When French mathematician Adrien-Marie Legendre published the method independently in 1805, a priority dispute erupted — one of the most famous in the history of mathematics.

Correlation Coefficient — The slope of the least-squares line equals r·(sy/sx), so the correlation coefficient r directly controls the steepness of the regression line and measures the strength of the linear relationship.
Residuals and Residual Plots — Residuals are the building blocks of least squares. Analysing their patterns reveals whether the linear model is appropriate or whether a different model would fit better.
Coefficient of Determination (R²) — R² = 1 − SSE/SST measures what proportion of the total variation in y is explained by the least-squares regression line, directly quantifying the quality of the fit.
Scatter Plots — Scatter plots are the essential first step before regression — they reveal whether the data have a linear trend that justifies applying least-squares fitting.