Coefficient of Determination

When we fit a curve or line to data, we need a way to measure how well that model actually captures the pattern in the data. The coefficient of determination, written R², gives us exactly that: it is a number between 0 and 1 that expresses the proportion of the total variability in the response variable that is explained by the regression model. An R² of 0.92, for example, means that 92% of the variation in y is accounted for by the model, while the remaining 8% is unexplained residual variation. R² is not limited to linear regression. It can be applied to any curve-fitting model — quadratic, exponential, power, sine, or anything else — making it one of the most versatile tools in statistics for comparing competing models. When deciding whether a quadratic fit is better than a linear one, or whether an exponential curve beats a power curve for a given data set, R² provides an objective, single-number comparison. Understanding R² requires familiarity with two key quantities: the total sum of squares (SS_tot), which measures how spread out the y-values are around their mean, and the residual sum of squares (SS_res), which measures how far the data points sit from the fitted curve. The relationship between these two quantities defines R² and makes it interpretable as an explained-variance ratio.

Key takeaways
  • Compute the mean of y
  • Calculate SS_tot (Total Sum of Squares)
  • Fit your model and compute SS_res (Residual Sum of Squares)
  • Compute R²
  • Interpreting R² for non-linear models

Compute the mean of y

Start with your n data pairs (x₁,y₁), (x₂,y₂), …, (xₙ,yₙ). Calculate the mean response ȳ = (1/n)∑yᵢ. This mean is the simplest possible 'model' — it predicts the same value for every x — and it serves as the baseline against which we judge any regression model.

Calculate SS_tot (Total Sum of Squares)

SS_tot = ∑(yᵢ − ȳ)² measures the total variability in the data around the mean. Every term (yᵢ − ȳ)² is the squared deviation of one data point from the mean. A large SS_tot means the data are very spread out; a small SS_tot means they cluster tightly around ȳ.

Fit your model and compute SS_res (Residual Sum of Squares)

After fitting a regression model — whether linear ŷ = mx + b, quadratic ŷ = ax² + bx + c, exponential ŷ = ae^(bx), power ŷ = ax^b, or sine ŷ = a·sin(bx + c) + d — compute the residuals eᵢ = yᵢ − ŷᵢ. These are the vertical distances between each observed yᵢ and the model's predicted value ŷᵢ. Then SS_res = ∑(yᵢ − ŷᵢ)² = ∑eᵢ². Smaller SS_res means a better-fitting model.

Compute R²

R² = 1 − SS_res / SS_tot. If the model fits perfectly (all points on the curve), SS_res = 0 and R² = 1. If the model is no better than just using ȳ, then SS_res ≈ SS_tot and R² ≈ 0. For nonlinear models, R² can theoretically be negative if the model fits worse than the horizontal mean line, although this is rare in practice.

Interpreting R² for non-linear models

For nonlinear regression (quadratic, exponential, power, sine), R² retains the same formula and the same interpretation. Compare models on the same data set: the model with the higher R² explains more of the variability. However, always inspect a residual plot as well — a high R² with a systematic curved pattern in the residuals may indicate model assumptions are violated or a different function family is needed.

The Key Equation

R2=1−SSresSStot=1−∑i=1n(yi−y^i)2∑i=1n(yi−yˉ)2R^2 = 1 - \frac{SS_{\text{res}}}{SS_{\text{tot}}} = 1 - \frac{\displaystyle\sum_{i=1}^{n}(y_i - \hat{y}_i)^2}{\displaystyle\sum_{i=1}^{n}(y_i - \bar{y})^2}
  • R^2 — Coefficient of determination (dimensionless): Proportion of total variability in y explained by the regression model; ranges from 0 to 1 for well-fitting models.
  • SS_{\text{res}} — Residual sum of squares (same units as y²): Sum of squared differences between each observed value and the model's predicted value: ∑(yᵢ − ŷᵢ)².
  • SS_{\text{tot}} — Total sum of squares (same units as y²): Sum of squared differences between each observed value and the mean ȳ: ∑(yᵢ − ȳ)².
  • \hat{y}_i — Predicted (fitted) value (same units as y): The value the regression model predicts for the i-th observation.
  • \bar{y} — Mean of y (same units as y): The arithmetic mean of all observed y-values, used as the baseline model.

Worked Example

Problem. A biologist measures the growth of a bacterial colony over time. She has 5 data points: (1, 2.1), (2, 4.9), (3, 10.8), (4, 22.3), (5, 48.6). She fits an exponential model ŷ = 1.05 · e^(1.00x), obtaining predicted values ŷ₁ = 2.86, ŷ₂ = 7.77, ŷ₃ = 21.1, ŷ₄ = 57.4, ŷ₅ = 155.9. She also tries a power model and wants to first find R² for the exponential model to decide whether to continue with it.

R2=1−12860.211430.77≈−7.99R^2 = 1 - \frac{12860.21}{1430.77} \approx -7.99

Answer. R² ≈ −7.99 for this exponential model with the given parameters, indicating a very poor fit.

Figure
Diagram
Decomposing Variability: SS_tot vs SS_resxyȳŷ (model)residual eᵢ(yᵢ − ȳ)(yᵢ − ŷᵢ)SS_tot = Σ(yᵢ − ȳ)²SS_res = Σ(yᵢ − ŷᵢ)²R² = 1 − SS_res/SS_tot
Origins · 1 minKarl Pearson and how this was discovered
KP
Karl Pearson
1857–1936 · British

Karl Pearson was the founder of modern mathematical statistics. He developed the Pearson correlation coefficient r, from which R² is derived for linear regression (R² = r² for simple linear models). He also created chi-squared tests, principal component analysis, and founded the first university statistics department at University College London in 1911. Pearson was so devoted to statistics that he kept meticulous daily records of his own health metrics for decades, effectively running a lifelong self-experiment — one of the earliest examples of personal data analytics.

  • Linear Regression — R² was first defined in the context of linear regression, and for simple linear models R² equals the square of Pearson's r. Understanding linear regression is the natural starting point before extending to nonlinear models.
  • Residuals and Residual Plots — SS_res is built from residuals, and residual plots are the companion diagnostic to R² — a high R² with a systematic residual pattern signals model inadequacy that R² alone cannot detect.
  • Pearson Correlation Coefficient — For simple linear regression, R² = r², directly linking the coefficient of determination to the Pearson correlation coefficient r.
  • Nonlinear Regression Models — R² is used to compare quadratic, exponential, power, and sine models. Understanding the forms of these models helps in choosing the best fit and interpreting R² results.