Regression

Regression finds the line (or curve) that best fits a set of data points, letting you predict one variable from another. The key result: the least-squares line

\overset{y}{^} = a + b x

minimizes the total squared vertical distance from points to the line. It applies anywhere you have paired numerical data — economics, biology, physics labs, machine learning. First developed by Francis Galton (1880s) and formalized using least squares by Gauss and Legendre (~1805). Regression is a statistical method for modeling the relationship between a dependent variable (the response,

y

) and one or more independent variables (predictors,

x

). The most common form, simple linear regression, fits a straight line through a scatter of points so that the line best summarizes the trend. It matters because it turns messy real-world data into a usable predictive equation and quantifies how strongly and in what direction two quantities are related. A surprising fact: the name "regression" comes from Galton's observation that tall parents tend to have children closer to the average height — a "regression toward the mean" — a phenomenon unrelated to the line-fitting machinery that now bears the name.

The Core Insight

Among all possible straight lines you could draw through a cloud of points, exactly one minimizes the sum of squared vertical errors. That line is unique, computable from simple sums of the data, and always passes through the point $(\overset{x}{ˉ}, \overset{y}{ˉ})$ — the center of mass of the data. Squaring the errors (rather than taking absolute values) makes the problem solvable with calculus and uniquely determined.

Formal Statement

Simple Linear Regression Model

We assume each observed $y_{i}$ relates to $x_{i}$ by a true line plus random error:

y_{i} = α + β x_{i} + ε_{i}

⟳this is the assumed model, not derived —

ε_{i}

captures noise with mean zero.

Here $α$ is the true intercept, $β$ the true slope, and $ε_{i}$ the random error for point $i$ . We estimate $α, β$ with $a, b$ , giving the fitted line:

\overset{y}{^}_{i} = a + b x_{i}

⟳

\overset{y}{^}_{i}

is the predicted value; the hat denotes an estimate.

The residual for point $i$ is the gap between observed and predicted:

e_{i} = y_{i} - \overset{y}{^}_{i} = y_{i} - (a + b x_{i})

⟳vertical distance from data point to the fitted line.

The Least-Squares Criterion

We choose $a$ and $b$ to minimize the sum of squared residuals $S$ :

S (a, b) = i = 1 \sum n e_{i}^{2} = i = 1 \sum n (y_{i} - a - b x_{i})^{2}

⟳total squared vertical error as a function of the two unknowns.

Least-Squares Solution

The values of $a$ and $b$ that minimize $S$ are:

b = \frac{\sum _{i = 1}^{n} ( x _{i} - x ˉ ) ( y _{i} - y ˉ )}{\sum _{i = 1}^{n} ( x _{i} - x ˉ ) ^{2}}

⟳covariance of

x, y

divided by variance of

x

; comes from setting derivatives to zero.

a = \overset{y}{ˉ} - b \overset{x}{ˉ}

⟳forces the line through

(\overset{x}{ˉ}, \overset{y}{ˉ})

where $\overset{x}{ˉ} = \frac{1}{n} \sum x_{i}$ and $\overset{y}{ˉ} = \frac{1}{n} \sum y_{i}$ are the means.

Correlation Coefficient

The strength and direction of the linear relationship are measured by Pearson's $r$ :

r = \frac{\sum ( x _{i} - x ˉ ) ( y _{i} - y ˉ )}{\sum ( x _{i} - x ˉ ) ^{2} \sum ( y _{i} - y ˉ ) ^{2}}

⟳standardized covariance; always lies in

[- 1, 1]

The coefficient of determination $r^{2}$ gives the fraction of variance in $y$ explained by $x$ :

r^{2} = 1 - \frac{\sum ( y _{i} - y ^ _{i} ) ^{2}}{\sum ( y _{i} - y ˉ ) ^{2}}

⟳1 minus (unexplained variation / total variation).

Why It Holds / Derivation

Deriving the Least-Squares Formulas

We minimize $S (a, b) = \sum (y_{i} - a - b x_{i})^{2}$ over $a$ and $b$ .

Step 1. Take the partial derivative with respect to

a

and set it to zero:

\frac{\partial S}{\partial a} = - 2 \sum (y_{i} - a - b x_{i}) = 0

⟳chain rule, derivative of inner

(\cdot)

w.r.t.

a

- 1

Step 2. Dividing by

- 2

and rearranging gives the first normal equation:

\sum y_{i} = na + b \sum x_{i}

⟳sum the constant

a

across

n

points.

Dividing through by $n$ yields $\overset{y}{ˉ} = a + b \overset{x}{ˉ}$ , which rearranges to equation [6].

Step 3. Take the partial derivative with respect to

b

and set it to zero:

\frac{\partial S}{\partial b} = - 2 \sum x_{i} (y_{i} - a - b x_{i}) = 0

⟳derivative of inner

(\cdot)

w.r.t.

b

- x_{i}

Step 4. This gives the second normal equation:

\sum x_{i} y_{i} = a \sum x_{i} + b \sum x_{i}^{2}

⟳distribute

x_{i}

inside the sum.

Step 5. Substitute

a = \overset{y}{ˉ} - b \overset{x}{ˉ}

from Step 2 into [12], and after algebra (collecting

b

terms) you obtain equation [5]. Because

S

is a sum of squares — a convex paraboloid in

(a, b)

— this critical point is the unique global minimum.

■

Carl Friedrich Gauss | 1777–1855

Gauss claimed to have used the method of least squares as early as 1795 to predict the orbit of the dwarf planet Ceres, famously relocating it after it disappeared behind the Sun. Adrien-Marie Legendre published the method first in 1805, sparking a priority dispute. Gauss later connected least squares to the normal (Gaussian) distribution of errors, showing the method is optimal when errors are normally distributed — a result underpinning the Gauss–Markov theorem.

Building Intuition

Picture a scatter of points and imagine threading a rigid rod through them. Connect each point to the rod with a vertical spring. Each spring stores energy proportional to the square of its stretch. The least-squares line is the resting position where total spring energy is minimal — the rod settles into the configuration where it pulls equally hard from above and below.

Why square the errors rather than just add absolute distances? Squaring (1) makes large errors count disproportionately, so the line "cares" about outliers; (2) gives a smooth function we can differentiate; and (3) produces a unique, closed-form answer. The slope $b$ is essentially "how much $y$ tends to move per unit move in $x$ ," computed as the covariance of $x$ and $y$ normalized by the spread of $x$ .

Least-squares line minimizing squared vertical residuals

<svg viewBox="0 0 460 320" xmlns="http://www.w3.org/2000/svg">
<rect width="460" height="320" fill="#0a0e17"/>
<line x1="50" y1="280" x2="430" y2="280" stroke="#94a3b8" stroke-width="1.5"/>
<line x1="50" y1="280" x2="50" y2="30" stroke="#94a3b8" stroke-width="1.5"/>
<text x="438" y="284" fill="#e2e8f0" font-size="13">x</text>
<text x="44" y="26" fill="#e2e8f0" font-size="13">y</text>
<!-- regression line -->
<line x1="60" y1="250" x2="420" y2="70" stroke="#d4a853" stroke-width="2.5"/>
<text x="350" y="95" fill="#d4a853" font-size="12">ŷ = a + bx</text>
<!-- data points + residuals -->
<line x1="110" y1="220" x2="110" y2="235" stroke="#94a3b8" stroke-width="1" stroke-dasharray="3"/>
<circle cx="110" cy="220" r="4" fill="#e2e8f0"/>
<line x1="180" y1="195" x2="180" y2="180" stroke="#94a3b8" stroke-width="1" stroke-dasharray="3"/>
<circle cx="180" cy="180" r="4" fill="#e2e8f0"/>
<line x1="250" y1="160" x2="250" y2="178" stroke="#94a3b8" stroke-width="1" stroke-dasharray="3"/>
<circle cx="250" cy="178" r="4" fill="#e2e8f0"/>
<line x1="320" y1="120" x2="320" y2="108" stroke="#94a3b8" stroke-width="1" stroke-dasharray="3"/>
<circle cx="320" cy="108" r="4" fill="#e2e8f0"/>
<line x1="390" y1="90" x2="390" y2="100" stroke="#94a3b8" stroke-width="1" stroke-dasharray="3"/>
<circle cx="390" cy="100" r="4" fill="#e2e8f0"/>
<text x="125" y="225" fill="#94a3b8" font-size="11">residual eᵢ</text>
</svg>

Worked Examples

Predicting Exam Score from Study Hours

Given 5 students with hours studied

x

and score

y

| $x$ | 1 | 2 | 3 | 4 | 5 | |---|---|---|---|---|---| | $y$ | 52 | 58 | 67 | 71 | 82 |

Step 1 — Means:

\overset{x}{ˉ} = 3

\overset{y}{ˉ} = (52 + 58 + 67 + 71 + 82) /5 = 330/5 = 66

. Step 2 — Deviations and products:

| $x_{i} - \overset{x}{ˉ}$ | $y_{i} - \overset{y}{ˉ}$ | product | $(x_{i} - \overset{x}{ˉ})^{2}$ | |---|---|---|---| | $- 2$ | $- 14$ | $28$ | $4$ | | $- 1$ | $- 8$ | $8$ | $1$ | | $0$ | $1$ | $0$ | $0$ | | $1$ | $5$ | $5$ | $1$ | | $2$ | $16$ | $32$ | $4$ |

Sums: $\sum (x - \overset{x}{ˉ}) (y - \overset{y}{ˉ}) = 73$ , $\sum (x - \overset{x}{ˉ})^{2} = 10$ .

Step 3 — Slope (eq [5]):

b = 73/10 = 7.3

. Step 4 — Intercept (eq [6]):

a = 66 - 7.3 \times 3 = 66 - 21.9 = 44.1

. Fitted line:

\overset{y}{^} = 44.1 + 7.3 x (points)

Prediction for 6 hours:

\overset{y}{^} = 44.1 + 7.3 (6) = 87.9

points.

Computing the Correlation Coefficient

Given the same data. We need

\sum (y_{i} - \overset{y}{ˉ})^{2} = (- 14)^{2} + (- 8)^{2} + 1^{2} + 5^{2} + 1 6^{2} = 196 + 64 + 1 + 25 + 256 = 542

. Step 1 — Apply eq [7]:

r = \frac{73}{10 \cdot 542} = \frac{73}{5420} = \frac{73}{73.62} = 0.9916.

Step 2 — Coefficient of determination:

r^{2} = 0.991 6^{2} \approx 0.983

Interpretation: about 98.3% of the variation in exam scores is explained by hours studied — a very strong positive linear relationship.

Common Misconceptions

Correlation Is Not Causation

A high $r$ means $x$ and $y$ move together — it does not prove that $x$ causes $y$ . Ice-cream sales correlate with drowning deaths, but both are driven by a hidden third variable (hot weather). Regression describes association, not mechanism. Establishing causation requires controlled experiments or careful causal reasoning, not just a good fit.

Do Not Extrapolate Beyond the Data

The fitted line is only trustworthy within the range of observed $x$ values. Predicting an exam score for 20 hours of study with our line gives $\overset{y}{^} = 44.1 + 7.3 (20) = 190$ — impossible, since scores cap at 100. Extrapolation assumes the linear trend continues forever, which it almost never does.

Anscombe's Quartet

In 1973 statistician Frank Anscombe constructed four datasets with identical means, identical slope ( $b = 0.5$ ), identical intercept, and identical $r = 0.816$ — yet they look completely different. One is a clean line, one is a perfect parabola, one is a line plus a single wild outlier, and one is a vertical cluster dragged by one extreme point. Lesson: never trust regression numbers without plotting the data first.

Regression of $y$ on $x$ ≠ Regression of $x$ on $y$

The line minimizing vertical errors (predicting $y$ from $x$ ) is not the same as the line minimizing horizontal errors (predicting $x$ from $y$ ). They have different slopes and only coincide when $r = \pm 1$ . Choosing which variable is the response matters.

Connections

Regression sits at a crossroads of several ideas. The slope formula in [5] is the covariance of $x$ and $y$ over the variance of $x$ , linking directly to Variance and Standard Deviation and Covariance. The derivation uses Partial Derivatives to minimize a function of two variables — an application of Optimization. The correlation coefficient $r$ relates to the Pearson Correlation and, when squared, to the analysis of variance decomposition of total variation. Extending [1] to several predictors gives multiple regression, expressible compactly with Matrix Algebra and the normal equations $X^{⊤} X b = X^{⊤} y$ . Regression is also the simplest member of Supervised Learning in machine learning.

Real-World Applications

Economics — Phillips Curve: regressing inflation on unemployment to model their trade-off.
Medicine — dose–response: predicting blood-pressure reduction from drug dosage in clinical trials.
Climate science: estimating global temperature trends by regressing average temperature on year.
Finance — CAPM beta: a stock's $β$ is the regression slope of its returns against market returns.
Sports analytics — Moneyball: the Oakland A's regressed runs scored on on-base percentage to value players.
Engineering / physics labs: fitting $V = I R$ to measured voltage–current data to extract resistance as the slope.
Machine learning: linear and logistic regression are foundational baseline models for prediction and classification.

Historical Development

1805 — Adrien-Marie Legendre publishes the method of least squares in an appendix on comet orbits.
1809 — Carl Friedrich Gauss publishes least squares, claims prior use (1795) for Ceres, links it to the normal distribution.
1821–1823 — Gauss proves the Gauss–Markov theorem: least squares is the best linear unbiased estimator.
1877–1886 — Francis Galton studies heredity, coins "regression toward mediocrity" while studying parent–child heights.
1896 — Karl Pearson formalizes the correlation coefficient $r$ and the modern theory of regression.
1900s — Pearson and G. Udny Yule extend regression to multiple variables.
1922 — Ronald Fisher develops the statistical framework (variance, significance testing) underpinning regression inference.
1973 — Frank Anscombe publishes Anscombe's Quartet, dramatizing the need to visualize data.
1970s–present — regression generalizes into ridge/lasso, logistic, and the regression layers inside modern neural networks.

Subtleties & Limits

Ordinary least squares assumes errors are independent, have constant variance (homoscedasticity), and that $x$ is measured without error. Real data routinely violate all three: heteroscedasticity (spreading variance), autocorrelation (in time series), and measurement error in predictors each bias or destabilize estimates. Outliers can dominate the fit because errors are squared, motivating robust regression methods. With many correlated predictors, multicollinearity makes coefficients unstable — an active area mitigated by regularization (ridge, lasso). And linear regression can only capture linear structure; choosing the right transformation or model form for nonlinear data remains as much art as science.