Chi-Square Inference

One-line statement: Chi-square inference uses the chi-square (

χ^{2}

) distribution to test hypotheses about categorical data and variances by comparing observed counts to expected counts. Key result in words: When data are grouped into categories, the sum of standardized squared deviations between observed and expected frequencies follows (approximately) a chi-square distribution, enabling goodness-of-fit, independence, and homogeneity tests. Where it applies: Categorical/count data, contingency tables, variance estimation for normal populations. Who/when: Karl Pearson introduced the chi-square goodness-of-fit test in 1900; R.A. Fisher corrected the degrees-of-freedom theory in the 1920s.

Chi-square inference is a family of statistical procedures built on the chi-square distribution, a continuous probability distribution arising as the sum of squared independent standard normal variables. Its central use is testing whether observed categorical frequencies are consistent with frequencies expected under a null hypothesis, measured by the Pearson chi-square statistic. It also underlies inference about the variance of a normally distributed population. The method matters because vast amounts of real data are categorical—survey responses, biological classifications, defect counts—where means and standard deviations are not the natural objects of study. A surprising fact: although the statistic is built from discrete counts, its sampling distribution is the continuous chi-square curve, valid only as a large-sample approximation, and the "correct" number of degrees of freedom famously eluded Pearson himself for over two decades.

The Core Insight

A single standardized normal deviation squared, $Z^{2}$ , has a chi-square distribution with 1 degree of freedom. If we standardize each category's deviation $(O - E) / E$ and square-and-sum across categories, the result behaves—approximately—like a sum of such squared normals. Large discrepancies between what we observe and what we expect produce large $χ^{2}$ values; rare large values are evidence against the null hypothesis. The whole apparatus reduces complicated categorical comparisons to one number and one reference curve.

Formal Statement

The Chi-Square Distribution

A random variable $X$ has a chi-square distribution with $k$ degrees of freedom, written $X \sim χ_{k}^{2}$ , if it is the sum of squares of $k$ independent standard normal random variables:

X = i = 1 \sum k Z_{i}^{2}, Z_{i} \sim iid N (0, 1)

⟳This is a definition, but

Z_{i}^{2}

being the building block follows from the change-of-variables formula applied to a single

N (0, 1)

density.

Its probability density function on $x > 0$ is

f_{k} (x) = \frac{1}{2 ^{k /2} Γ ( k /2 )} x^{k /2 - 1} e^{- x /2}

⟳This is the Gamma

(k /2, 2)

density; obtain it by computing the moment generating function

(1 - 2 t)^{- k /2}

and recognizing it.

Here $Γ (\cdot)$ is the gamma function, $k$ is a positive integer (the degrees of freedom), $x$ is the variable, and $e$ is Euler's number. The mean is $k$ and the variance is $2 k$ .

Pearson Chi-Square Statistic

For data sorted into $m$ mutually exclusive categories with observed counts $O_{i}$ and expected counts $E_{i}$ (under the null hypothesis), the Pearson statistic is

χ^{2} = i = 1 \sum m \frac{( O _{i} - E _{i} ) ^{2}}{E _{i}}

⟳Each term standardizes the deviation

O_{i} - E_{i}

by its approximate variance

E_{i}

(Poisson-like); summing approximately reproduces a sum of squared normals.

where $O_{i}$ is the observed frequency in category $i$ , $E_{i} = n p_{i}$ is the expected frequency, $n$ is the total sample size, and $p_{i}$ is the hypothesized probability of category $i$ .

Pearson's Limit Theorem

Under the null hypothesis $H_{0} : (p_{1}, \dots, p_{m})$ fixed and known, as the sample size $n \to \infty$ , the Pearson statistic in [3] converges in distribution to a chi-square random variable:

χ^{2} d χ_{m - 1}^{2}

⟳The standardized count vector tends to a multivariate normal with one linear constraint (

\sum O_{i} = n

); the quadratic form of this constrained normal is

χ^{2}

with

m - 1

degrees of freedom.

If the null model contains $r$ parameters estimated from the data, the degrees of freedom drop to $m - 1 - r$ .

Why It Holds / Derivation

Sketch of Pearson's Theorem

Step 1 — Multinomial setup. The vector of counts

(O_{1}, \dots, O_{m})

follows a multinomial distribution with parameters

n

and

(p_{1}, \dots, p_{m})

, where

\sum p_{i} = 1

. Step 2 — Multivariate CLT. Define the standardized deviations

U_{i} = \frac{O _{i} - n p _{i}}{n p _{i}} .

⟳This is

(O_{i} - E_{i}) / E_{i}

; by the multivariate central limit theorem the vector

(U_{1}, \dots, U_{m})

converges to a multivariate normal.

Step 3 — Covariance structure. The limiting normal has mean

0

and covariance matrix

Σ = I - p p^{T}

, where

p

is the vector of

p_{i}

. This matrix is a projection of rank

m - 1

. Step 4 — Quadratic form. The statistic is

χ^{2} = \sum U_{i}^{2} = U^{T} U

. A standard normal vector projected onto an

(m - 1)

-dimensional subspace yields a quadratic form distributed as

χ_{m - 1}^{2}

U^{T} U d χ_{rank (Σ)}^{2} = χ_{m - 1}^{2} .

⟳The eigenvalues of the projection

Σ

are

m - 1

ones and one zero; a sum of squared normals with these weights is exactly

χ_{m - 1}^{2}

Step 5 — Estimated parameters. Each independently estimated parameter imposes one further linear constraint, lowering the rank—hence the subtraction of

r

from the degrees of freedom.

■

Chi-square densities for several degrees of freedom; the right tail beyond the critical value is the rejection region

Karl Pearson | 1857–1936

Karl Pearson, founder of the world's first university statistics department at University College London, introduced the chi-square goodness-of-fit test in his 1900 paper "On the criterion that a given system of deviations...". A polymath trained in mathematics and law, Pearson sought objective criteria for judging how well theoretical curves fit real frequency data—motivated by biological and evolutionary problems from his collaboration with Francis Galton and Walter Weldon. His original derivation, however, used $m - 1$ degrees of freedom even when parameters were estimated; this error was decisively corrected by Ronald Fisher.

Building Intuition

Imagine rolling a die 600 times. Under fairness you expect 100 of each face. If you get $(105, 98, 110, 92, 95, 100)$ , the deviations are small relative to $100 = 10$ , so each standardized term is around 1 or less, and their sum is modest. But $(150, 60, 130, 70, 90, 100)$ produces deviations of 50, 40, etc., whose squares dwarf $E_{i}$ , inflating $χ^{2}$ . The statistic essentially asks: "In units of natural sampling noise, how far off are we, totalled across all categories?" The chi-square distribution tells us how large that total tends to get by chance alone, so an observed value in the far right tail signals a genuine departure from the model.

Worked Examples

Goodness-of-Fit: Testing a Fair Die

Given: A die is rolled

n = 120

times. Observed counts:

O = (28, 14, 22, 18, 20, 18)

. Test

H_{0}

: the die is fair (

p_{i} = 1/6

) at significance

α = 0.05

. Step 1 — Expected counts.

E_{i} = n p_{i} = 120 \times \frac{1}{6} = 20

for each face. Step 2 — Compute terms.

\frac{( 28 - 20 ) ^{2}}{20} + \frac{( 14 - 20 ) ^{2}}{20} + \frac{( 22 - 20 ) ^{2}}{20} + \frac{( 18 - 20 ) ^{2}}{20} + \frac{( 20 - 20 ) ^{2}}{20} + \frac{( 18 - 20 ) ^{2}}{20}

= \frac{64 + 36 + 4 + 4 + 0 + 4}{20} = \frac{112}{20} = 5.6.

Step 3 — Degrees of freedom.

df = m - 1 = 6 - 1 = 5

. No parameters estimated. Step 4 — Critical value.

χ_{0.05, 5}^{2} = 11.07

. Step 5 — Decision. Since

5.6 < 11.07

, we fail to reject

H_{0}

χ^{2} = 5.6, df = 5, fail to reject; data consistent with a fair die

Test of Independence: Treatment vs. Outcome

Given: A

2 \times 2

contingency table of patients:

| | Recovered | Not recovered | Row total | |---|---|---|---| | Drug | 50 | 30 | 80 | | Placebo | 35 | 45 | 80 | | Col total | 85 | 75 | 160 |

Test independence of treatment and outcome at $α = 0.05$ .

Step 1 — Expected counts

E_{ij} = \frac{( row _{i} ) ( col _{j} )}{n}

E_{11} = \frac{80 \times 85}{160} = 42.5

E_{12} = \frac{80 \times 75}{160} = 37.5

E_{21} = 42.5

E_{22} = 37.5

. Step 2 — Statistic.

χ^{2} = \frac{( 50 - 42.5 ) ^{2}}{42.5} + \frac{( 30 - 37.5 ) ^{2}}{37.5} + \frac{( 35 - 42.5 ) ^{2}}{42.5} + \frac{( 45 - 37.5 ) ^{2}}{37.5}

= \frac{56.25}{42.5} + \frac{56.25}{37.5} + \frac{56.25}{42.5} + \frac{56.25}{37.5} = 1.324 + 1.5 + 1.324 + 1.5 = 5.648.

Step 3 — Degrees of freedom.

df = (r - 1) (c - 1) = (2 - 1) (2 - 1) = 1

. Step 4 — Critical value.

χ_{0.05, 1}^{2} = 3.841

. Step 5 — Decision.

5.648 > 3.841

, so reject

H_{0}

χ^{2} = 5.65, df = 1, reject independence: treatment associated with outcome

Common Misconceptions

The chi-square test requires sufficiently large expected counts. A common rule of thumb is that every

E_{i}

should be at least 5 (some allow a few as low as 1 if most exceed 5). When expected counts are too small, the continuous chi-square approximation to the discrete multinomial is poor and p-values are unreliable. Remedies include combining categories or using Fisher's exact test. Also note: the statistic uses counts, never proportions or percentages—plugging in percentages is a frequent and fatal error. The chi-square test is not a test of the strength of association. Consider two

2 \times 2

tables with identical proportions but different sample sizes. With

n = 40

a moderate association may give

χ^{2} = 1.5

(not significant), while the same proportions with

n = 4000

give

χ^{2} = 150

(wildly significant). The statistic scales with

n

, so a "significant" result with huge samples can reflect a trivially small effect. Significance

\neq =

importance—report an effect-size measure such as Cramér's

V

alongside the test.

Connections

Chi-square inference sits in a web of related ideas. It is a special case of the Likelihood Ratio Test, whose statistic $G^{2} = 2 \sum O_{i} ln (O_{i} / E_{i})$ is asymptotically equivalent to Pearson's $χ^{2}$ . The distribution itself is the Gamma Distribution with shape $k /2$ and scale 2, and squaring a single Standard Normal Distribution variable gives $χ_{1}^{2}$ . The ratio of two independent chi-squares (scaled) defines the F-Distribution used in Analysis of Variance, while a normal divided by a scaled chi-square root gives the Student t-Distribution. Inference about a normal variance uses $(n - 1) s^{2} / σ^{2} \sim χ_{n - 1}^{2}$ , directly linking it to Confidence Intervals.

Real-World Applications

Genetics (Mendelian ratios): Testing whether observed offspring phenotype counts match expected 9:3:3:1 dihybrid ratios.
Quality control: Comparing observed defect-type counts across production lines to detect a shifted process.
Market research: Testing independence between customer demographic and product preference in survey contingency tables.
Epidemiology: Assessing association between exposure and disease status in case-control studies.
Political science: Testing homogeneity of candidate support across geographic regions.
Ecology: Goodness-of-fit of observed species-abundance counts to a hypothesized distribution.
Cryptanalysis / randomness testing: The chi-square test on byte-frequency counts checks whether a data stream behaves like uniform random output.

Historical Development

1875: Friedrich Helmert derives the distribution of the sample variance for normal data—an early implicit chi-square result.
1900: Karl Pearson publishes the goodness-of-fit chi-square test, naming the statistic and tabulating its distribution.
1904: Pearson extends the method to contingency tables and association.
1908–1915: Refinements by Pearson and others on applying the test to estimated parameters (with the degrees-of-freedom error persisting).
1922–1924: R.A. Fisher corrects the degrees of freedom, showing that estimating $r$ parameters reduces $df$ by $r$ , sparking a public dispute with Pearson.
1924: Fisher publishes tables and clarifies the relationship among $χ^{2}$ , $t$ , and $F$ distributions.
1934: Fisher introduces the exact test for small $2 \times 2$ tables, addressing the small-count limitation.
1954: William Cochran articulates practical rules for minimum expected frequencies and validity conditions.

Subtleties & Limits

The chi-square approximation degrades for sparse tables, and there is no universally agreed threshold for "large enough" expected counts—Cochran's rules are heuristics, not theorems. For multi-way contingency tables, log-linear models and the question of which interaction terms to include push beyond classical chi-square testing into model-selection territory where the asymptotic degrees of freedom may not be integers in practice. Continuity corrections (Yates's correction for $2 \times 2$ tables) remain debated: they can over-correct and reduce power. Modern alternatives—exact permutation tests, Monte Carlo p-values, and Bayesian model comparison—address these limits but raise their own questions about computational feasibility and prior sensitivity in high dimensions.