Chi-Squared Distribution and Goodness of Fit

The chi-squared distribution is where probability theory hands experimental physics its ruler. Mathematically it is the distribution of a sum of squares of independent standard Gaussians; physically it is the number that decides whether a model curve genuinely threads a cloud of noisy data or merely looks as if it does. The bridge is the central limit theorem: because measurement errors are so often approximately Gaussian, the squared, error-normalised residuals of a fit pile up into a chi-squared variable, and the tail of that distribution becomes the universal currency of goodness of fit — the $p$ -value quoted in every particle-physics discovery, every cosmological parameter contour, every calibration report. We build the object constructively. Start from one standard normal $Z\sim N(0,1)$ ; the change-of-variables formula gives the density of $Z^2$ , which is $\chi^2_1$ , equivalently a Gamma with shape $1/2$ and scale $2$ . Independence then adds degrees of freedom, handled most cleanly through the moment-generating function $(1-2t)^{-k/2}$ , whose product structure makes the additivity $\chi^2_{k_1}+\chi^2_{k_2}=\chi^2_{k_1+k_2}$ fall out in one line. The same variable reappears from the multivariate Gaussian, where the squared Mahalanobis distance is exactly $\chi^2_k$ — which is why error ellipsoids are level sets of a chi-squared. In practice we estimate parameters rather than knowing the true mean, each estimate costs one degree of freedom, and the Pearson statistic is compared against $\chi^2_\nu$ with $\nu=N-m$ . Reduced chi-squared, $p$ -values, and confidence regions all follow from a single tail integral of one density.

Key takeaways

The chi-squared with $k$ degrees of freedom is the law of $Q=\sum_{i=1}^{k}Z_i^2$ for independent standard Gaussians; its density is $f(x)=\frac{1}{2^{k/2}\Gamma(k/2)}x^{k/2-1}e^{-x/2}$ on $x>0$ .
It is exactly a Gamma with shape $k/2$ and scale $2$ , so it has mean $k$ , variance $2k$ , and mode $\max(k-2,0)$ .
The same variable is the squared Mahalanobis distance of a $k$ -dimensional Gaussian, which is why constant-probability error ellipsoids are its level sets.
In goodness-of-fit testing the Pearson statistic $X^2=\sum_i(O_i-E_i)^2/E_i$ is asymptotically $\chi^2_\nu$ with $\nu=N-m$ , one degree of freedom lost per fitted constraint.
A good fit has reduced chi-squared $\chi^2/\nu\approx 1$ ; the $p$ -value is the upper tail $P(\chi^2_\nu>X^2_{\mathrm{obs}})$ , small $p$ meaning tension with the data.
For large $k$ the distribution becomes Gaussian, $(Q-k)/\sqrt{2k}\to N(0,1)$ .
Key idea: A sum of $k$ independent squared standard Gaussians is $\chi^2_k$ ; normalised squared residuals of a Gaussian-error fit therefore follow $\chi^2_\nu$ , and the upper-tail probability $P(\chi^2_\nu>X^2_{\mathrm{obs}})$ quantifies model–data agreement.

Where this lives today

The chi-squared tail is the arbiter of modern quantitative science. When ATLAS and CMS announced the Higgs boson in 2012, the " $5\sigma$ " they quoted was a $p$ -value converted to sigmas — a statement that the background-only model fit the observed excess with probability below $3\times10^{-7}$ , computed (in its profile-likelihood generalisation) from precisely this tail machinery. Cosmologists fitting $\Lambda$ CDM to the Planck power spectrum report a global chi-squared against thousands of multipole bins, and draw the joint confidence ellipses in the $(H_0,\Omega_\Lambda)$ plane using exactly the $\Delta\chi^2=2.30,6.18,11.83$ contours of Example 3. Every particle-physics fit quoting a " $\chi^2/\mathrm{ndf}$ " applies equation (6), and the same distribution governs variance confidence intervals in metrology, nuclear-decay histograms, and gravitational-wave error ellipsoids. Wherever a physicist writes "the fit is good," a chi-squared tail integral is doing the certifying.

The physical picture

Picture a standard Gaussian dart-thrower aiming at the origin of a $k$ -dimensional space. Each coordinate of a throw is an independent $N(0,1)$ , and the quantity we watch is the squared distance from the bullseye, $Q=Z_1^2+\cdots+Z_k^2$ . In one dimension the darts crowd near zero, so $\chi^2_1$ piles up at the origin and diverges there; add dimensions and the typical squared radius grows to about $k$ , since every axis contributes one unit of variance on average. That single fact — mean $k$ , variance $2k$ — sets the whole shape: the peak drifts rightward to $k-2$ , the distribution broadens, and for large $k$ it rounds into a Gaussian. The first figure is that family of shapes.

Figure

Chi-squared densities for k = 1, 2, 4, 6 degrees of freedom. The mean sits at x = k and the interior peak at the mode x = k − 2, so the whole distribution marches rightward and flattens as k grows: k = 1 diverges at the origin, k = 2 is a pure exponential ½e^(−x/2), and k = 4, 6 show interior peaks at x = 2 and x = 4

Goodness of fit turns that geometry into a decision. If the model were correct and the error bars honest, each residual divided by its error is roughly a standard Gaussian, so squaring and summing converts "distance from the model" into a chi-squared whose expected value is the number of free residuals $\nu$ . A statistic near $\nu$ signals agreement; far above signals a model that misses the data or underestimated errors; far below hints at overestimated errors (or data too good to be true). The verdict is quantitative: shade the density beyond the observed value and that area is the $p$ -value. The second figure carries a real case — a fair-die test with $\nu=5$ , an observed statistic of $4.9$ , and the $5\%$ rejection region past the critical value $11.07$ .

Figure

Goodness of fit as a tail area on the chi-squared with 5 degrees of freedom. The observed statistic X² = 4.9 from the fair-die test of Example 1 lands just below the mean at x = 5, deep in the bulk, so the model is accepted. The red shaded region beyond the critical value x = 11.07 has area 0.05 — the conventional 5 percent rejection region

The core equations

Everything begins with the density — the law of a sum of $k$ independent squared standard normals. Its ingredients, the power $x^{k/2-1}$ , the exponential $e^{-x/2}$ , and the constant built from $\Gamma(k/2)$ , are exactly those of a Gamma distribution with shape $k/2$ and scale $2$ .

f_k(x)=\frac{1}{2^{k/2}\,\Gamma(k/2)}\,x^{k/2-1}\,e^{-x/2},\qquad x>0 \qquad (1)

where: $k$ — degrees of freedom (a positive integer in physics); $x$ — value of the chi-squared variable, $x\ge 0$ ; $\Gamma$ — the gamma function, with $\Gamma(n)=(n-1)!$ for integer $n$ and $\Gamma(1/2)=\sqrt\pi$ . Reading it: the power $x^{k/2-1}$ sets the origin behaviour (diverging for $k=1$ , flat for $k=2$ , vanishing for $k\ge3$ ), while $e^{-x/2}$ keeps every moment finite. The moment-generating function is the algebraic engine of the subject. Because independent variables multiply their MGFs, the single factor for one squared normal raised to the $k$ -th power gives the MGF of $\chi^2_k$ , whose product form makes degrees of freedom add.

M_Q(t)=\mathbb{E}\!\left[e^{tQ}\right]=(1-2t)^{-k/2},\qquad t<\tfrac12 \qquad (2)

where: $M_Q(t)$ — the MGF of $Q$ ; $t$ — the transform argument, restricted to $t<1/2$ so the defining integral converges. Reading it: the exponent $-k/2$ is additive in $k$ , so $M_{Q_1}M_{Q_2}=(1-2t)^{-(k_1+k_2)/2}$ — additivity of degrees of freedom is visible by eye. Differentiating $\ln M_Q$ at the origin extracts the cumulants; the first two anchor every rule of thumb for reading a fit.

\mathbb{E}[Q]=k,\qquad \operatorname{Var}(Q)=2k,\qquad \text{skewness}=\sqrt{8/k} \qquad (3)

where: $\mathbb{E}[Q]$ — the expected value, equal to the degrees of freedom; $\operatorname{Var}(Q)$ — the variance, twice that; the skewness $\sqrt{8/k}$ decays to zero as $k\to\infty$ . Reading it: a "typical" chi-squared value is $k\pm\sqrt{2k}$ , which is why a reduced chi-squared has spread $\sqrt{2/\nu}$ about $1$ . The cumulative distribution function is a regularised incomplete gamma function — no elementary closed form for general $k$ , but every statistics library evaluates it, and the goodness-of-fit $p$ -value is simply its upper tail.

F_k(x)=\frac{\gamma(k/2,\,x/2)}{\Gamma(k/2)}=P\!\left(\tfrac{k}{2},\tfrac{x}{2}\right),\qquad p=1-P\!\left(\tfrac{\nu}{2},\tfrac{X^2_{\mathrm{obs}}}{2}\right) \qquad (4)

where: $\gamma(s,y)$ — the lower incomplete gamma function; $P(s,y)$ — its regularised form; $\nu=N-m$ — the degrees of freedom after fitting $m$ parameters; $p$ — the probability, under the null, of a statistic at least as large as observed. Reading it: small $p$ means the model sits in the improbable tail and is disfavoured. The two working statistics — the Pearson form for binned counts and the weighted least-squares form for continuous measurements — are both sums of squared, error-normalised residuals that converge to $\chi^2_\nu$ under the null.

X^2=\sum_{i=1}^{N}\frac{(O_i-E_i)^2}{E_i},\qquad \chi^2=\sum_{i=1}^{N}\frac{\bigl(y_i-f(x_i;\boldsymbol\theta)\bigr)^2}{\sigma_i^2} \qquad (5)

where: $O_i,E_i$ — observed and model-expected counts in bin $i$ ; $y_i,\sigma_i$ — the measured value and its standard error at $x_i$ ; $f(x_i;\boldsymbol\theta)$ — the model prediction with parameters $\boldsymbol\theta$ ; $N$ — the number of bins or data points. Reading it: the Pearson denominator $E_i$ is the Poisson variance of the bin, so each term is a standardised fluctuation squared. The reduced chi-squared collapses the test to one number, and the Mahalanobis identity ties the distribution back to the multivariate Gaussian, showing why confidence ellipsoids are chi-squared level sets.

\chi^2_\nu\equiv\frac{\chi^2}{\nu}\approx 1\ (\text{good fit}),\qquad (\mathbf{x}-\boldsymbol\mu)^\top\Sigma^{-1}(\mathbf{x}-\boldsymbol\mu)\sim\chi^2_k \qquad (6)

where: $\chi^2_\nu$ — the reduced chi-squared (chi-squared per degree of freedom); $\mathbf{x}$ — a $k$ -dimensional Gaussian with mean $\boldsymbol\mu$ and covariance $\Sigma$ . Reading it: a reduced chi-squared far from $1$ diagnoses trouble, while the quadratic form is the squared distance a whitening transform turns into $\sum z_i^2$ .

Derivation AMoment-generating function and the additivity of degrees of freedom

The goal is to compute the MGF of a single squared standard normal from a Gaussian integral, show that independence multiplies MGFs so $k$ of them give $(1-2t)^{-k/2}$ , and read off both the additivity rule $\chi^2_{k_1}+\chi^2_{k_2}=\chi^2_{k_1+k_2}$ and the mean and variance from the cumulant expansion. This one transform does the work of a dozen convolution integrals and is the reason the chi-squared family is closed under addition.

Step 1. Take $Z\sim N(0,1)$ and form $M_1(t)=\mathbb{E}[e^{tZ^2}]=\int_{-\infty}^{\infty}\frac{1}{\sqrt{2\pi}}\,e^{tz^2}e^{-z^2/2}\,\mathrm{d}z$ . Combine the exponents into $-\tfrac12 z^2(1-2t)$ , a Gaussian in $z$ with variance $1/(1-2t)$ . Why: the whole calculation reduces to a Gaussian integral, and the coefficient $1-2t$ carries the entire distribution.

Step 2. Evaluate the Gaussian: $\int e^{-\tfrac12 z^2(1-2t)}\mathrm{d}z=\sqrt{2\pi/(1-2t)}$ , so $M_1(t)=(1-2t)^{-1/2}$ , valid whenever $1-2t>0$ , i.e. $t<1/2$ . Why: the condition $t<1/2$ is the boundary beyond which the tail $e^{-x/2}$ can no longer beat the growth of $e^{tx}$ .

Step 3. For $Q=\sum_{i=1}^k Z_i^2$ with independent $Z_i$ , the expectation of a product factorises, so $M_Q(t)=(1-2t)^{-k/2}$ . Two independent chi-squared variables then satisfy $M_{Q_1+Q_2}(t)=(1-2t)^{-(k_1+k_2)/2}$ , the MGF of $\chi^2_{k_1+k_2}$ ; by uniqueness the sum is $\chi^2_{k_1+k_2}$ . Why: because the MGF determines the distribution uniquely, matching MGFs is a complete proof — no convolution integral is needed to add degrees of freedom.

Step 4. Expand $\ln M_Q(t)=-\tfrac{k}{2}\ln(1-2t)$ ; its derivatives at $t=0$ give $\kappa_1=k$ (mean), $\kappa_2=2k$ (variance), and the standardised third cumulant $\sqrt{8/k}$ (skewness). Why: the vanishing skewness as $k\to\infty$ is the fingerprint of the central-limit approach to a Gaussian.

Go deeper · Derivation B · 1 minThe density from a single squared Gaussian and the Gamma match

Derivation BThe density from a single squared Gaussian and the Gamma match

The goal is to derive the $\chi^2_1$ density explicitly by the change-of-variables formula for the non-injective map $z\mapsto z^2$ , then recognise the general MGF $(1-2t)^{-k/2}$ as that of a Gamma distribution and read the full density off the Gamma family. Together these give two independent confirmations of equation (1): a direct construction for $k=1$ and a transform-based one for all $k$ .

Step 1. For $Y=Z^2$ with $Z\sim N(0,1)$ , work through the CDF because the map is two-to-one: $F_Y(y)=P(-\sqrt y\le Z\le\sqrt y)=\Phi(\sqrt y)-\Phi(-\sqrt y)=2\Phi(\sqrt y)-1$ , using $\Phi(-z)=1-\Phi(z)$ . Why: squaring folds the roots $\pm\sqrt y$ onto one value, so a one-branch Jacobian would miss a factor of two — the CDF handles the folding automatically.

Step 2. Differentiate with the chain rule: $f_Y(y)=2\,\phi(\sqrt y)\cdot\frac{1}{2\sqrt y}=\frac{1}{\sqrt{2\pi y}}\,e^{-y/2}$ . This matches equation (1) at $k=1$ because $2^{1/2}\Gamma(1/2)=\sqrt{2\pi}$ . Why: the appearance of $\Gamma(1/2)=\sqrt\pi$ is exactly what turns the general normalising constant into the familiar $\sqrt{2\pi}$ , closing the loop with the standard normal.

Step 3. For general $k$ , compare $(1-2t)^{-k/2}$ with the Gamma MGF $(1-\beta t)^{-\alpha}$ of shape $\alpha$ and scale $\beta$ ; matching gives $\alpha=k/2$ , $\beta=2$ , so $\chi^2_k$ is exactly Gamma $(k/2,\,2)$ . Why: the chi-squared is not a distinct species but a slice of the Gamma family, so its density, moments, and CDF are all Gamma results at half-integer shape and scale two.

Step 4. Insert $\alpha=k/2$ , $\beta=2$ into the Gamma density $\frac{x^{\alpha-1}e^{-x/\beta}}{\beta^\alpha\Gamma(\alpha)}$ to recover $f_k(x)=\frac{x^{k/2-1}e^{-x/2}}{2^{k/2}\Gamma(k/2)}$ . Check $k=2$ : this is $\tfrac12 e^{-x/2}$ , an exponential of mean $2$ ; check $k=1$ : it reproduces Step 2. Why: the two cases the density must satisfy — exponential at $k=2$ , folded Gaussian at $k=1$ — both fall out, confirming the match for every $k$ .

Go deeper · Derivation C · 1 minMahalanobis distance and the whitening transform

Derivation CMahalanobis distance and the whitening transform

The goal is to prove that the squared Mahalanobis distance of a $k$ -dimensional Gaussian is exactly $\chi^2_k$ , by constructing a linear "whitening" transform that turns a correlated Gaussian vector into $k$ independent standard normals and rewrites the quadratic form as a sum of squares. This underlies every error ellipse and joint confidence region in experimental physics — the geometry of correlated uncertainties is chi-squared geometry in disguise.

Step 1. Because $\Sigma$ is symmetric positive-definite, it has a symmetric positive-definite square root $\Sigma^{1/2}$ with $\Sigma=\Sigma^{1/2}\Sigma^{1/2}$ , and hence an inverse $\Sigma^{-1/2}$ . Why: the square root exists precisely when the covariance is non-degenerate, encoding the assumption that no linear combination of the data has zero variance.

Step 2. Define the whitened vector $\mathbf{z}=\Sigma^{-1/2}(\mathbf{x}-\boldsymbol\mu)$ . A linear transform of a Gaussian is Gaussian, and its covariance is $\operatorname{Cov}(\mathbf{z})=\Sigma^{-1/2}\,\Sigma\,\Sigma^{-1/2}=I_k$ , so $\mathbf{z}\sim N(\mathbf{0},I_k)$ : independent standard normals. Why: whitening is the multivariate analogue of dividing a residual by its error bar — it strips out correlations, leaving unit-variance independent coordinates.

Step 3. Rewrite the quadratic form: $D^2=(\mathbf{x}-\boldsymbol\mu)^\top\Sigma^{-1}(\mathbf{x}-\boldsymbol\mu)=\mathbf{z}^\top\mathbf{z}=\sum_{i=1}^k z_i^2$ , a sum of $k$ independent squared standard normals, which is $\chi^2_k$ . Why: this identifies the error ellipsoids (surfaces of constant $D^2$ ) as level sets of a chi-squared, so their enclosed probabilities come straight from the $\chi^2_k$ CDF.

Step 4. Check one dimension: $\Sigma=\sigma^2$ gives $D^2=(x-\mu)^2/\sigma^2$ , the squared standardised residual, distributed as $\chi^2_1$ . Why: the familiar "residual over sigma, squared" is just the $k=1$ Mahalanobis distance, connecting it to the weighted least-squares statistic of equation (5).

Limiting cases

The character of $\chi^2_k$ shifts qualitatively with the degrees of freedom, and each regime has a clean closed form worth memorising.

Case	Result	Meaning
One degree of freedom	$f(x)=\dfrac{1}{\sqrt{2\pi x}}\,e^{-x/2}$	The density of $Z^2$ for one standard normal; diverges as $x\to0^+$ but remains integrable.
Two degrees of freedom	$f(x)=\tfrac12 e^{-x/2}$	Exactly an exponential of mean $2$ — the simplest closed form, finite at the origin and monotone decreasing.
Mode location	$x_{\mathrm{mode}}=\max(k-2,\,0)$	For $k\ge2$ the peak sits at $k-2$ ; for $k=1,2$ the density is monotone with no interior maximum.
Large $k$ (central limit)	$\dfrac{Q-k}{\sqrt{2k}}\xrightarrow{d}N(0,1)$	The standardised chi-squared becomes standard normal; Fisher's $\sqrt{2Q}\approx N(\sqrt{2k-1},1)$ converges even faster.
Sample variance	$\dfrac{(n-1)S^2}{\sigma^2}\sim\chi^2_{n-1}$	For $n$ i.i.d. normal samples the scaled sample variance is chi-squared — the basis of variance intervals and Student's $t$ .

Common trapConfusing N with the degrees of freedom, and misusing Pearson at small counts

The degrees of freedom are $\nu=N-m$ , not $N$ : every parameter estimated from the data imposes a linear constraint on the residuals, so a straight-line fit to $N$ points has $\nu=N-2$ . Forgetting this raises the expected chi-squared above the true $\nu$ and makes good fits look bad — Pearson made exactly this error in 1900, and Fisher corrected it in 1922. A second, independent trap is applying the Pearson statistic when expected counts are small ( $E_i\lesssim 5$ ): the Gaussian approximation to each Poisson bin fails, and one must merge bins or use an exact or likelihood-ratio test. A third: reduced chi-squared is only informative when $\nu$ is not tiny, because its spread is $\sqrt{2/\nu}$ — with $\nu=4$ a good fit routinely returns values near $2$ .

Worked examples

1 · Testing a die for fairness (fully worked)

A six-sided die is rolled $120$ times with observed counts $O=(16,19,25,14,22,24)$ . Under the fair-die hypothesis each face is expected $E=120/6=20$ times. Is the die consistent with fair?

Step 1. Form $X^2=\sum_i (O_i-E_i)^2/E_i$ with $E_i=20$ . The squared deviations are $16,1,25,36,4,16$ , summing to $98$ ; dividing by $E=20$ gives $X^2=98/20=\mathbf{4.9}$ .

Step 2. Count degrees of freedom. There are $N=6$ bins, but one constraint is imposed because the counts must sum to the fixed total $120$ , so $\nu=6-1=\mathbf{5}$ . No further parameters were fitted — the fair-die model has none to estimate.

Step 3. Compare with $\chi^2_5$ , whose mean is $\nu=5$ and standard deviation $\sqrt{2\nu}\approx3.16$ . The observed $X^2=4.9$ sits just below the mean, well inside the bulk (second figure). The upper-tail probability is $p=P(\chi^2_5>4.9)=\mathbf{0.43}$ .

Step 4. Interpret. A $p$ -value of $0.43$ means that if the die were fair, a discrepancy this large or larger would occur about $43\%$ of the time — unremarkable. There is no evidence against fairness; the reduced chi-squared $X^2/\nu=0.98\approx1$ says the same. Rejecting at the $5\%$ level would have needed $X^2>11.07$ , more than double what we saw.

2 · Reading a weighted straight-line fit (fully worked)

Five voltage measurements $V=(2.05,3.98,6.10,7.92,10.03)\ \mathrm{V}$ , each with error $\sigma_V=0.05\ \mathrm{V}$ , are taken at currents $I=(1,2,3,4,5)\ \mathrm{A}$ . Fit Ohm's law $V=RI$ (a line through the origin) and assess the fit.

Step 1. For a one-parameter model through the origin with equal errors, the weighted least-squares estimate is $R=\dfrac{\sum_i I_i V_i}{\sum_i I_i^2}=\dfrac{110.14}{55}=\mathbf{2.003\ \Omega}$ , with error $\sigma_R=\sigma_V/\sqrt{\sum I_i^2}=0.05/\sqrt{55}=\mathbf{0.0067\ \Omega}$ .

Step 2. The residuals $V_i-RI_i$ are $+0.047,\,-0.025,\,+0.092,\,-0.090,\,+0.017$ volts; standardising by $\sigma_V=0.05$ gives $+0.95,\,-0.50,\,+1.85,\,-1.80,\,+0.35$ standard errors.

Step 3. Square and sum: $\chi^2=0.90+0.25+3.41+3.25+0.12=\mathbf{7.94}$ . With $N=5$ points and $m=1$ fitted parameter, $\nu=4$ , so the reduced chi-squared is $\chi^2/\nu=\mathbf{1.98}$ .

Step 4. Interpret carefully. Naively $\chi^2_\nu\approx2$ looks alarming, but the spread of reduced chi-squared at $\nu=4$ is $\sqrt{2/4}\approx0.71$ , so $1.98$ lies only about $1.4\sigma$ high. The exact tail is $p=P(\chi^2_4>7.94)=\mathbf{0.094}$ — mildly high but not significant. The fit is acceptable; the $+1.85\sigma$ residual at $I=3$ is the largest and the first to re-measure, but four degrees of freedom cannot distinguish a poor model from an unlucky draw.

3 · Two-parameter confidence ellipse (worked relation)

A least-squares fit returns a best pair of parameters $(a,b)$ with a $2\times2$ covariance. What $\Delta\chi^2$ contour encloses the joint $68.3\%$ , $95.4\%$ , and $99.7\%$ confidence regions?

Step 1. The offset $\Delta\chi^2=\chi^2-\chi^2_{\min}$ is the squared Mahalanobis distance of $(a,b)$ from the best fit, which is $\chi^2_2$ , so the enclosed probability is the closed-form CDF $F_2(\Delta\chi^2)=1-e^{-\Delta\chi^2/2}$ .

Step 2. Invert for each level, $\Delta\chi^2=-2\ln(1-c)$ : the one-, two-, and three-sigma joint levels $c=0.6827,\,0.9545,\,0.9973$ give $\Delta\chi^2=\mathbf{2.30,\ 6.18,\ 11.83}$ , the standard contour heights for parameter ellipses in cosmology and accelerator physics. For a single parameter of interest the levels are instead $\Delta\chi^2=1,\,4,\,9$ (a $\chi^2_1$ projection) — confusing the two doubles the error bars.

Physics of the portrait

The chi-squared goodness-of-fit test is the invention that let physics say, quantitatively, whether a theory fits — and it belongs above all to Karl Pearson.

Origins · 1 minKarl Pearson and how this was discovered

Karl Pearson

1857–1936 · the chi-squared goodness-of-fit test

Karl Pearson was the founder of mathematical statistics as a discipline, and the chi-squared test is his most enduring gift to the experimental sciences. Trained at Cambridge and steeped in German philosophy — he spent enough time in Germany to respell his given name "Carl" as "Karl" — he held the University College London chair in applied mathematics and, spurred by the biologist Walter Weldon and the polymath Francis Galton, turned to measuring whether data agree with a hypothesised law. In an 1900 paper he introduced $X^2=\sum(O_i-E_i)^2/E_i$ and proved that, under the null, it follows the distribution Helmert and others had already met in the theory of Gaussian errors. This was the first general, distribution-free way to attach a probability to "the model fits the data," and it remains the workhorse of hypothesis testing a century later. Pearson also gave us the method of moments, the modern correlation coefficient, and the term "standard deviation." His one famous error concerned this very test: he used $\nu=N-1$ regardless of how many parameters were fitted, a mistake Fisher corrected in 1922 — the source of a long and bitter feud between the two men.

History

1876

— Helmert derives the distribution of the Gaussian sample variance

1900

— Pearson introduces the chi-squared goodness-of-fit statistic

1908

— Gosset, writing as Student, builds the t distribution on an independent chi-squared

1922

— Fisher corrects the degrees of freedom to ν = N − m

1936

— Mahalanobis defines the generalised distance whose square is chi-squared

Going further

Texts. Stone & Goldbart, Mathematics for Physics, probability and statistics chapter · Arfken & Weber, Mathematical Methods for Physicists, probability chapter · Riley, Hobson & Bence, Mathematical Methods for Physics and Engineering, Ch. 30 · Cowan, Statistical Data Analysis (Oxford) · Press et al., Numerical Recipes, "Modeling of Data" chapter. The key checkpoints in this article are: the Gaussian integral yields the single-normal MGF $(1-2t)^{-1/2}$ ; the change of variables for $\chi^2_1$ uses $\Gamma(1/2)=\sqrt\pi$ ; the cumulants $\kappa_1=k$ and $\kappa_2=2k$ follow from differentiating $\ln M_Q$ ; and the whitening identity gives $\operatorname{Cov}(\Sigma^{-1/2}(\mathbf x-\boldsymbol\mu))=I_k$ . The worked numbers are: the die test gives $X^2=4.9$ , $\nu=5$ , $p=0.43$ ; the resistor fit gives $R=2.003\ \Omega$ , $\chi^2=7.94$ , $\nu=4$ , $p=0.094$ ; the contour levels $\Delta\chi^2=2.30,6.18,11.83$ for $68.3/95.4/99.7\%$ follow from $-2\ln(1-c)$ .

Prerequisites. Gaussian (Normal) Distribution · Multivariate Gaussian Distribution · Cumulative Distribution Functions and Change of Variables · Moment-Generating and Characteristic Functions · Expectation, Variance and Moments. Leads to. Central Limit Theorem · Conditional Probability and Bayes' Theorem · Poisson Distribution

Ask the tutor: Want to walk through your own dataset — its chi-squared, degrees of freedom, and p-value — or see why the Mahalanobis distance turns error ellipses into chi-squared contours?

Why exactly does each fitted parameter remove one degree of freedom?

Because each parameter estimated from the data forces a linear constraint on the residuals. A least-squares straight line is defined by setting the derivatives of $\chi^2$ with respect to slope and intercept to zero, which makes the weighted sum of residuals and their $x$ -moment vanish. Those two equations leave only $N-2$ residuals free to fluctuate, so the chi-squared's effective dimension drops to $\nu=N-m$ for $m$ fitted parameters. Using $\nu=N$ overstates the expected chi-squared and wrongly penalises good fits — the mistake Fisher corrected in 1922.

What counts as a good value of reduced chi-squared?

As a rule of thumb $\chi^2/\nu\approx1$ signals a healthy fit, since the mean of $\chi^2_\nu$ is $\nu$ . But "close to 1" must be judged against its spread $\sqrt{2/\nu}$ : with $\nu=100$ a value outside roughly $1\pm0.14$ is notable, whereas with $\nu=4$ even $1.98$ is unremarkable. Values well above $1$ point to a model that misses the data or to underestimated errors; values well below $1$ suggest overestimated errors, over-fitting, or correlated data treated as independent. It is a first diagnostic, not a verdict — always convert to a $p$ -value before rejecting a model.

How is the p-value actually computed if the CDF has no closed form?

The chi-squared CDF is the regularised lower incomplete gamma function $P(\nu/2,\,x/2)$ , so the $p$ -value is its complement $1-P(\nu/2,\,X^2/2)$ . For general $\nu$ there is no elementary formula, but the incomplete gamma function is computed to machine precision by a continued fraction or series, and every library — `scipy.stats.chi2.sf`, ROOT's `TMath::Prob`, a spreadsheet's `CHISQ.DIST.RT` — returns it instantly. Closed forms exist for special cases: $\nu=2$ gives $p=e^{-X^2/2}$ . For large $\nu$ the Gaussian approximation $p\approx1-\Phi((X^2-\nu)/\sqrt{2\nu})$ , or the sharper Wilson-Hilferty formula, suffices.

When does the Pearson statistic fail, and what replaces it?

The Pearson statistic relies on each Poisson bin being approximately Gaussian, which holds only when the expected count is not too small — the usual guideline is $E_i\gtrsim5$ . With sparse bins the discreteness matters, the chi-squared approximation breaks down, and the $p$ -value can be badly misstated. The standard fixes are to merge adjacent sparse bins until each expected count clears the threshold, or to switch to an exact multinomial test, a likelihood-ratio ( $G$ -test) statistic $2\sum O_i\ln(O_i/E_i)$ , or a Monte-Carlo null distribution. In particle physics with very low counts, unbinned likelihood methods are preferred outright.

Why is the squared Mahalanobis distance chi-squared, and why does it matter?

If $\mathbf{x}$ is Gaussian with mean $\boldsymbol\mu$ and covariance $\Sigma$ , the quadratic form $(\mathbf{x}-\boldsymbol\mu)^\top\Sigma^{-1}(\mathbf{x}-\boldsymbol\mu)$ is exactly $\chi^2_k$ . The proof is a change of variables: whitening, $\mathbf{z}=\Sigma^{-1/2}(\mathbf{x}-\boldsymbol\mu)$ , produces $k$ independent standard normals, so the form becomes $\sum z_i^2$ . This matters because the constant-probability surfaces of a multivariate Gaussian — the error ellipsoids around a best fit — are then level sets of a chi-squared, so the probability enclosed by an ellipse of "radius" $\Delta\chi^2$ is read directly from the chi-squared CDF. That is how joint confidence regions are built.

Is the chi-squared distribution related to the Gaussian, and how do they blur together?

Intimately. The chi-squared is built from Gaussians by squaring and summing, so for one degree of freedom it is literally the law of $Z^2$ . For large $k$ the central limit theorem takes over: $Q$ is a sum of $k$ i.i.d. squared normals, so $(Q-k)/\sqrt{2k}$ tends to a standard Gaussian and the once-skewed density rounds into a symmetric bell (Fisher's $\sqrt{2Q}\approx N(\sqrt{2k-1},1)$ converges faster still). A chi-squared with a few degrees of freedom is markedly skewed, but with a hundred it is nearly Gaussian — which is why for large $\nu$ you assess a fit by counting how many $\sqrt{2\nu}$ the statistic sits from $\nu$ .