Linear Regression Models

Before fitting any line, decide which variable is the explanatory variable (also called the independent or predictor variable, placed on the x-axis) and which is the response variable (also called the dependent variable, placed on the y-axis). For example, if you are studying how hours of study (x) predicts exam score (y), hours of study is the explanatory variable.

You need five summary statistics: the means x̄ and ȳ, the standard deviations sₓ and sᵧ, and the correlation coefficient r. These can be computed from your data set or read from a statistics calculator. The means tell you where the centre of the data lies, the standard deviations describe the spread, and r describes the strength and direction of the linear association (r ranges from −1 to +1).

The slope of the least-squares line is given by b₁ = r · (sᵧ / sₓ). The slope tells you, on average, how many units y changes for each one-unit increase in x. Notice that the slope has the same sign as r: if the association is positive (r > 0), the line slopes upward; if negative (r < 0), it slopes downward. The ratio sᵧ/sₓ rescales the correlation from standardised units into the original units of the data.

The intercept is found by using the fact that the least-squares line always passes through the point (x̄, ȳ), the intersection of the two means. Substituting into ŷ = b₀ + b₁x gives b₀ = ȳ − b₁x̄. The intercept is the predicted value of y when x = 0. Be cautious: this may not always be meaningful in context if x = 0 is outside the range of your data.

Write the equation as ŷ = b₀ + b₁x, replacing ŷ, b₀, b₁, x with the actual variable names and values. For example: predicted score = 42.3 + 5.8 × (hours studied). Always interpret the slope in context: 'For each additional hour studied, the predicted exam score increases by 5.8 points.' Check whether the intercept is meaningful. Finally, compute r² to describe how much of the variability in y is explained by the linear relationship with x.

A residual is the difference between an observed y value and the value predicted by the regression line: e = y − ŷ. By construction, the sum of all residuals equals zero for the least-squares line. Plotting residuals against x (a residual plot) is essential for checking whether a linear model is appropriate. A good fit shows residuals scattered randomly around zero with no pattern. A curved pattern in the residual plot signals that a linear model may not be the best choice.

• \hat{y} — Predicted value of y (units of y): The value of the response variable predicted by the regression model for a given x; pronounced 'y-hat'.
• b_0 — Intercept (units of y): The predicted value of y when x = 0; the point where the regression line crosses the y-axis.
• b_1 — Slope (units of y per unit of x): The average change in y for each one-unit increase in x.
• r — Correlation coefficient (dimensionless): Measures the strength and direction of the linear association between x and y; ranges from −1 to +1.
• s_y,\; s_x — Standard deviations of y and x (units of y; units of x): Measures of spread; their ratio rescales the correlation into slope units.
• \bar{x},\; \bar{y} — Means of x and y (units of x; units of y): The arithmetic means of each variable; the least-squares line always passes through (x̄, ȳ).

Problem. A statistics student records the number of hours seven classmates studied for a maths test and their resulting scores. The summary statistics are: x̄ = 4.0 hours, ȳ = 68.0 points, sₓ = 1.5 hours, sᵧ = 9.0 points, r = 0.85. (a) Find the least-squares regression line. (b) Predict the score for a student who studied for 5 hours. (c) Calculate and interpret r².

Answer. Regression line: ŷ = 47.6 + 5.1x. Predicted score for 5 hours = 73.1 points. r² ≈ 72.3%.

• Correlation Coefficient — The correlation coefficient r is a direct input to the least-squares slope formula, and r² measures how well the regression line fits the data.
• Residuals and Residual Plots — Residuals are the differences between observed and predicted values; analysing them is essential for verifying that a linear regression model is appropriate.
• Scatter Plots — Regression lines are plotted on scatter plots; identifying a linear pattern in a scatter plot is the first step before fitting a regression model.
• Coefficient of Determination (r²) — r² measures the proportion of variability in the response variable explained by the regression model, and is the primary measure of fit quality.
• Extrapolation and Prediction — Using a regression line to predict y for values of x outside the data range (extrapolation) is a common misuse of linear regression that students must recognise and avoid.