Foundations of Statistics

Foundations of Statistics concerns the philosophical and mathematical principles that justify how we infer properties of a population from a sample. Key result in words: rival schools (frequentist, Bayesian, likelihoodist) interpret probability and inference differently, but each rests on a coherent axiomatic core — probability theory plus a rule connecting data to belief or decision. Where it applies: every domain that draws conclusions from data — science, medicine, economics, machine learning. Who/when: crystallized 1920s–1950s through R.A. Fisher, Jerzy Neyman, Egon Pearson, Harold Jeffreys, and Abraham Wald.

The foundations of statistics study what it means to learn from data: how a probabilistic model of a random experiment licenses statements about unknown quantities. At its core sits a probability space and a statistical model — a family of distributions indexed by parameters — together with principles (sufficiency, likelihood, conditionality) and a chosen interpretation of probability. The subject matters because every applied statistical method silently assumes one of these foundations, and choosing wrongly produces confident nonsense. A surprising fact: the Likelihood Principle, which most statisticians find intuitively compelling, is logically incompatible with standard frequentist procedures like p-values and confidence intervals — so the everyday toolkit of science violates a principle its users often endorse.

The Core Insight

All of statistical inference reduces to one question: given data $x$ generated by an unknown distribution $P_{θ}$ from a known family ${P_{θ} : θ \in Θ}$ , what can we say about $θ$ ? The likelihood function $L (θ) = p (x ∣ θ)$ — the data's probability viewed as a function of the parameter — is the bridge between observation and inference. The deep disagreements in statistics are not about mathematics but about what to do with the likelihood: maximize it (Fisher), embed it in long-run error control (Neyman–Pearson), or combine it with a prior via Bayes' theorem (Bayesian).

Formal Statement

Statistical Model

A statistical model (or experiment) is a triple $(X, A, {P_{θ} : θ \in Θ})$ where $X$ is the sample space of possible data, $A$ a $σ$ -algebra of events, and ${P_{θ}}$ a family of probability measures on $(X, A)$ indexed by a parameter $θ$ ranging over the parameter space $Θ$ . The model is parametric if $Θ \subseteq R^{k}$ , nonparametric if $Θ$ is infinite-dimensional. A statistic is any measurable function $T : X \to R^{m}$ of the data alone.

When the measures admit densities $p (x ∣ θ)$ with respect to a common dominating measure $μ$ , the central object is:

L (θ ∣ x) = p (x ∣ θ), ℓ (θ ∣ x) = lo g L (θ ∣ x)

⟳the density evaluated at the observed

x

is read as a function of

θ

with

x

fixed; the log-likelihood

ℓ

is monotone in

L

and turns products of independent observations into sums.

For $n$ independent identically distributed observations $x = (x_{1}, \dots, x_{n})$ :

L (θ ∣ x) = i = 1 \prod n p (x_{i} ∣ θ), ℓ (θ ∣ x) = i = 1 \sum n lo g p (x_{i} ∣ θ)

⟳independence factorizes the joint density into a product; the logarithm of a product is the sum of logarithms.

Sufficiency

A statistic $T$ is sufficient for $θ$ if the conditional distribution of the data $X$ given $T = t$ does not depend on $θ$ . Intuitively, $T$ captures all information in the sample relevant to $θ$ .

Fisher–Neyman Factorization

A statistic $T$ is sufficient for $θ$ if and only if the density factorizes as

p (x ∣ θ) = g (T (x), θ) h (x)

where

g

depends on the data only through

T

and

h

does not depend on

θ

⟳write the joint density, condition on the level set

{T = t}

; the

θ

-free conditional distribution forces the

θ

-dependence to enter only via

g (T, θ)

, while

h

absorbs the rest.

Likelihood Principle

If two experiments yield data $x$ and $y$ (possibly from different sample spaces) such that the likelihood functions are proportional,

L_{1} (θ ∣ x) = c L_{2} (θ ∣ y) for all θ, c > 0,

then

x

and

y

carry identical evidence about

θ

, and any inference must be the same. Birnbaum (1962) proved this follows from the Sufficiency Principle plus the Conditionality Principle.

⟳Birnbaum's argument constructs a mixed experiment that randomly selects between the two; the Conditionality Principle says inference ignores the unrealized branch, while sufficiency says proportional likelihoods are equivalent — together forcing the conclusion.

Why It Holds / Derivation

Birnbaum's Theorem — Sufficiency + Conditionality ⟹ Likelihood

Setup. Let $E_{1}, E_{2}$ be two experiments with the same parameter $θ$ , yielding outcomes $x$ (from $E_{1}$ ) and $y$ (from $E_{2}$ ) with proportional likelihoods, $L_{1} (θ ∣ x) = c L_{2} (θ ∣ y)$ .

Mixed experiment. Define $E^{*}$ : flip a fair coin; if heads run $E_{1}$ , if tails run $E_{2}$ . An outcome of $E^{*}$ is a pair $(j, z)$ recording which experiment was run and its result.

Conditionality Principle (C). Inference from observing $(j, z)$ in $E^{*}$ must equal inference from running $E_{j}$ alone and observing $z$ — the unperformed experiment is irrelevant. Thus $Ev (E^{*}, (1, x)) = Ev (E_{1}, x)$ and $Ev (E^{*}, (2, y)) = Ev (E_{2}, y)$ .

Sufficiency within $E^{*}$ . Construct a statistic $T$ on $E^{*}$ that maps both $(1, x)$ and $(2, y)$ to a common value (legitimate because their likelihoods are proportional, so the conditional distribution given $T$ is $θ$ -free). $T$ is sufficient.

Sufficiency Principle (S). Two outcomes with the same value of a sufficient statistic carry the same evidence: $Ev (E^{*}, (1, x)) = Ev (E^{*}, (2, y))$ .

Conclusion. Chaining steps 3 and 5: $Ev (E_{1}, x) = Ev (E_{2}, y)$ . Proportional likelihoods imply identical evidence. $■$

Birnbaum's theorem is genuinely unsettling: nearly all statisticians accept Sufficiency and Conditionality as obvious, yet the Likelihood Principle they jointly imply forbids frequentist procedures whose answers depend on the whole sample space (the unobserved data), such as p-values, confidence levels, and stopping rules. The theorem itself has been challenged (Durbin 1970, Evans 2013) on the grounds that the two principles are stated over different domains; the debate is unresolved.

Ronald A. Fisher | 1890–1962

The towering figure of 20th-century statistics. Fisher introduced likelihood (distinguishing it sharply from probability), maximum likelihood estimation, sufficiency, information (the Fisher Information), and the design of experiments with randomization and factorial layouts. He fought bitterly with Neyman and Pearson over the meaning of inference, defending fiducial arguments and rejecting their decision-theoretic framing. His 1925 Statistical Methods for Research Workers and 1935 The Design of Experiments shaped how science is done. Brilliant, combative, and also a population geneticist who unified Mendelism with Darwin, Fisher's foundational concepts remain the vocabulary of the field even where his philosophy is rejected.

Building Intuition

Think of the parameter $θ$ as a hidden dial setting on a machine that spits out data. You see the output but not the dial. The likelihood asks, for each candidate dial setting, "how plausibly would this machine have produced exactly what I saw?" A setting under which your data is wildly improbable is a poor explanation; a setting under which it is probable is a good one.

The three schools then differ on the final move. The frequentist refuses to assign probability to the dial itself (it has one fixed unknown value) and instead asks: "If I used this procedure repeatedly across many hypothetical datasets, how often would I be right?" — error rates over imagined replications. The Bayesian treats the dial setting as genuinely uncertain, places a prior probability on it, and updates to a posterior using Bayes' rule — probability as degree of belief. The likelihoodist stays minimal: report the likelihood curve itself and let relative likelihoods quantify evidence, refusing both error rates and priors. Same data, same likelihood, three philosophies of what counts as an answer.

The likelihood function for a binomial sample: 7 successes in 10 trials. The peak is the maximum likelihood estimate.

Worked Examples

Maximum Likelihood for a Binomial Proportion

Given: A coin is flipped

n = 10

times yielding

k = 7

heads. Model:

X \sim Binomial (n, θ)

θ

= probability of heads. Step 1 — Likelihood.

L (θ) = (7 10) θ^{7} (1 - θ)^{3} .

Step 2 — Log-likelihood (drop constant).

ℓ (θ) = 7 ln θ + 3 ln (1 - θ) + const .

Step 3 — Differentiate and set to zero.

\frac{d ℓ}{d θ} = \frac{7}{θ} - \frac{3}{1 - θ} = 0 \Rightarrow 7 (1 - θ) = 3 θ \Rightarrow 7 = 10 θ .

Step 4 — Solve.

\hat{θ} = 7/10

. Second derivative is negative, confirming a maximum. Step 5 — Fisher information & standard error. For one observation

I (θ) = 1/ [θ (1 - θ)]

; for

n

trials the total is

n / [θ (1 - θ)]

. At

\hat{θ}

SE = \frac{θ ^ ( 1 - θ ^ )}{n} = \frac{0.7 \times 0.3}{10} = 0.021 \approx 0.145.

\hat{θ} = 0.70, SE \approx 0.145 (dimensionless)

Bayesian Update with a Beta Prior

Given: Same data (

k = 7

heads in

n = 10

). Prior belief:

θ \sim Beta (2, 2)

— symmetric, mild preference for fairness. Step 1 — Prior density.

p (θ) \propto θ^{2 - 1} (1 - θ)^{2 - 1} = θ (1 - θ)

. Step 2 — Likelihood.

L (θ) \propto θ^{7} (1 - θ)^{3}

. Step 3 — Posterior (Bayes).

p (θ ∣ x) \propto θ (1 - θ) \cdot θ^{7} (1 - θ)^{3} = θ^{8} (1 - θ)^{4} .

This is

Beta (9, 5)

— conjugacy:

Beta (a, b)

updates to

Beta (a + k, b + n - k) = Beta (2 + 7, 2 + 3)

. Step 4 — Posterior mean.

E [θ ∣ x] = \frac{9}{9 + 5} = \frac{9}{14} \approx 0.643.

Step 5 — 95% credible interval (numerically, Beta(9,5) quantiles): approximately

[0.39, 0.86]

Posterior = Beta (9, 5), E [θ ∣ x] \approx 0.643, 95% CrI \approx [0.39, 0.86]

Note the Bayesian estimate (0.643) is pulled below the MLE (0.70) toward the prior mean (0.5).

Common Misconceptions

The p-value is not P(H₀ is true)

A p-value is $P (data as or more extreme ∣ H_{0})$ — a probability about the data given the hypothesis, computed under the assumption that $H_{0}$ holds. It is not $P (H_{0} ∣ data)$ , the probability the hypothesis is true. These can differ enormously: for a borderline result $p = 0.05$ the posterior probability of $H_{0}$ can exceed 0.30 under reasonable priors (Berger–Sellke 1987). Confusing the two — the "prosecutor's fallacy" — is the single most consequential error in applied statistics.

A 95% confidence interval does not contain θ with probability 0.95

Once computed, a confidence interval like $[0.41, 0.99]$ either contains the fixed true $θ$ or it does not — there is no probability left. The "95%" refers to the procedure: across many hypothetical datasets, 95% of the intervals it generates would cover $θ$ . The probability lives in the random interval before you see data, not in the realized one. Only a Bayesian credible interval licenses the statement " $θ$ lies here with probability 0.95."

Stopping rules break frequentist inference but not likelihood

Two experimenters both observe 7 heads in 10 flips. Experimenter A fixed $n = 10$ in advance (Binomial model). Experimenter B flipped until getting 3 tails (Negative Binomial model). The likelihood functions are proportional — $\propto θ^{7} (1 - θ)^{3}$ — so by the Likelihood Principle their evidence about $θ$ is identical. Yet the frequentist p-value for $H_{0} : θ = 0.5$ differs between the two designs ( $0.172$ for Binomial vs $0.0327$ for Negative Binomial), because the "more extreme" sample space differs. The same data yields significance under one stopping rule and not the other — a direct collision between frequentist practice and the Likelihood Principle.

Connections

The foundations connect every branch of inference. Maximum Likelihood Estimation operationalizes the likelihood as a point estimator; Fisher Information quantifies the curvature of $ℓ (θ)$ that determines estimator precision via the Cramér–Rao Bound. The Neyman–Pearson Lemma founds optimal hypothesis testing on likelihood ratios. Bayes' Theorem supplies the engine of Bayesian inference, with Decision Theory (Wald) providing a unifying framework where frequentist and Bayesian methods appear as different ways of handling risk. Sufficiency and the Exponential Family structure determine which models admit clean inference, while Asymptotic Theory (consistency, asymptotic normality) explains why all schools tend to agree as data accumulate — a deep reconciliation.

Real-World Applications

Clinical trial design (FDA). The choice between fixed-sample frequentist tests and Bayesian adaptive designs directly invokes the stopping-rule controversy; the FDA's 2010 and 2019 guidance on Bayesian trials reflects foundational debates about whether interim looks "spend" error.
Genome-wide association studies (GWAS). Testing millions of SNPs forces explicit multiple-comparison corrections (Bonferroni, FDR) — a frequentist concern about the family of tests that a strict likelihoodist would dismiss.
Spam filters and medical diagnosis. Naive Bayes classifiers and diagnostic test interpretation depend on base rates (priors); ignoring them produces the prosecutor's fallacy in courtroom DNA evidence.
Particle physics (Higgs boson, 2012). The "5-sigma" discovery threshold is a frequentist tail-probability convention; physicists debate it against Bayesian model-comparison via Bayes factors.
A/B testing in tech (Google, Microsoft). "Peeking" at running experiments inflates false-positive rates under fixed-horizon frequentist tests, driving adoption of sequential and Bayesian methods.
Insurance and actuarial credibility theory. Bühlmann credibility is a Bayesian shrinkage estimator combining individual and pooled data, foundational to premium-setting.
Machine learning regularization. L2 penalties correspond exactly to Gaussian priors (MAP estimation), making "ridge regression" a Bayesian procedure in frequentist clothing.

Historical Development

1654 — Pascal and Fermat exchange letters on games of chance, founding probability and implicitly the classical (equiprobable) interpretation.
1763 — Thomas Bayes' essay (published posthumously by Richard Price) introduces inverse probability — updating belief about a cause from observed effects.
1812 — Laplace's Théorie analytique des probabilités systematizes inverse probability and the principle of insufficient reason; dominates 19th-century inference.
1922 — R.A. Fisher's On the Mathematical Foundations of Theoretical Statistics introduces likelihood, sufficiency, efficiency, and consistency, displacing inverse probability.
1933 — Neyman and Pearson publish their lemma, founding hypothesis testing on error rates and the long-run frequency interpretation; Kolmogorov axiomatizes probability measure-theoretically.
1937 — Neyman introduces confidence intervals as a frequentist alternative to fiducial/Bayesian intervals.
1939 — Harold Jeffreys' Theory of Probability revives and rigorizes Bayesian inference with objective priors, against the prevailing frequentist tide.
1950 — Abraham Wald's Statistical Decision Functions unifies estimation and testing under decision theory, proving Bayes rules are essentially complete (admissibility).
1954 — L.J. Savage's Foundations of Statistics derives subjective probability and expected utility from axioms of rational preference, grounding the modern Bayesian program.
1962 — Allan Birnbaum proves that Sufficiency + Conditionality imply the Likelihood Principle, sharpening the conflict between schools.

Subtleties & Limits

Several foundational questions remain genuinely open or contested. (1) The status of Birnbaum's theorem: its premises are stated over different experimental domains, and critics (Evans, Mayo) argue the proof equivocates — so whether Sufficiency and Conditionality really force the Likelihood Principle is unsettled. (2) The problem of priors: Bayesian inference requires a prior, but "objective" or "non-informative" priors (Jeffreys, reference priors) are not invariant under reparametrization in general and can yield improper or paradoxical posteriors (marginalization paradoxes). (3) Severity and error-statistics: Deborah Mayo's program argues that good inference requires that a claim has survived a severe test, a property invisible to the Likelihood Principle — challenging whether evidence can be likelihood-only. (4) Calibration of subjective belief: how, if at all, personal probabilities connect to objective frequencies (de Finetti's exchangeability gives a partial bridge) remains philosophically live. (5) The replication crisis has reopened all of this: whether the foundations themselves, or merely their misuse, are to blame for irreproducible science is actively debated.