Sampling Distributions

A sampling distribution is the probability distribution of a statistic (such as the sample mean) computed over all possible samples of a fixed size drawn from a population. Key result: the sample mean of $n$ independent observations has mean $μ$ and standard deviation $σ / n$ , and for large $n$ its distribution is approximately normal regardless of the population shape. Applied throughout inferential statistics — confidence intervals, hypothesis testing, quality control. Formalized in the early 20th century by Gosset ("Student"), Fisher, and others.

A sampling distribution describes how a sample statistic varies from sample to sample if the sampling process were repeated indefinitely under identical conditions. It is the conceptual bridge between a single observed dataset and statements about an unknown population parameter: without it, a number like "the sample mean is 5.3" carries no measure of reliability. Every confidence interval, $p$ -value, and standard error is computed from a sampling distribution. The surprising fact is that even when individual observations come from a wildly non-normal population — skewed, bimodal, discrete — the distribution of their average becomes smooth and bell-shaped as the sample grows, a phenomenon governed by the Central Limit Theorem. This universality is what makes normal-theory inference applicable far beyond normally distributed data.

The Core Insight

A statistic is itself a random variable, because it is a function of random data. Repeating the sampling experiment produces a spread of statistic values; that spread — its center, dispersion, and shape — is the sampling distribution. Inference works by treating the one observed statistic as a single draw from this distribution and reasoning backward to the parameter.

Formal Statement

Population, Sample, Statistic, Sampling Distribution

Let a population be modeled by a random variable $X$ with cumulative distribution $F$ , mean $μ = E [X]$ , and variance $σ^{2} = Var (X)$ . A random sample of size $n$ is a sequence $X_{1}, X_{2}, \dots, X_{n}$ of independent and identically distributed (i.i.d.) copies of $X$ .

A statistic is any measurable function $T = g (X_{1}, \dots, X_{n})$ that does not depend on unknown parameters. The sampling distribution of $T$ is the probability distribution of the random variable $T$ .

The most important statistic is the sample mean:

\overset{ˉ}{X}_{n} = \frac{1}{n} i = 1 \sum n X_{i}

⟳This is a definition; note it is a linear combination of the

X_{i}

, which makes its mean and variance easy to compute.

Its first two moments are:

E [\overset{ˉ}{X}_{n}] = μ, Var (\overset{ˉ}{X}_{n}) = \frac{σ ^{2}}{n}

⟳Apply linearity of expectation to the sum; for the variance, use independence so cross-covariance terms vanish, giving

\frac{1}{n ^{2}} \sum Var (X_{i}) = \frac{n σ ^{2}}{n ^{2}}

The quantity $σ_{\overset{ˉ}{X}} = σ / n$ is called the standard error of the mean. Here $μ$ is the population mean, $σ^{2}$ the population variance, $n$ the sample size, and $X_{i}$ the $i$ -th observation.

Central Limit Theorem (CLT)

Let $X_{1}, X_{2}, \dots$ be i.i.d. with finite mean $μ$ and finite variance $σ^{2} > 0$ . Then the standardized sample mean converges in distribution to a standard normal:

Z_{n} = \frac{X ˉ _{n} - μ}{σ / n} d N (0, 1) as n \to \infty

⟳Show the characteristic function (or moment generating function) of

Z_{n}

converges to

e^{- t^{2} /2}

via a Taylor expansion to second order; invoke Lévy's continuity theorem.

Equivalently, for large $n$ , $\overset{ˉ}{X}_{n} \approx N (μ, σ^{2} / n)$ .

Sampling Distributions Under Normality

If $X_{1}, \dots, X_{n}$ are i.i.d. $N (μ, σ^{2})$ , then exactly (not just asymptotically):

\overset{ˉ}{X}_{n} \sim N (μ, \frac{σ ^{2}}{n}), \frac{( n - 1 ) S ^{2}}{σ ^{2}} \sim χ_{n - 1}^{2}

⟳

\overset{ˉ}{X}

is a linear combination of normals, hence normal. For

S^{2}

, use the orthogonal (Helmert) transformation showing

\sum (X_{i} - \overset{ˉ}{X})^{2} / σ^{2}

is a sum of

n - 1

squared independent standard normals.

where $S^{2} = \frac{1}{n - 1} \sum_{i = 1}^{n} (X_{i} - \overset{ˉ}{X})^{2}$ is the sample variance. Moreover $\overset{ˉ}{X}$ and $S^{2}$ are independent, and the studentized mean follows a $t$ -distribution:

T = \frac{X ˉ _{n} - μ}{S / n} \sim t_{n - 1}

⟳Write

T = Z / V / (n - 1)

where

Z = (\overset{ˉ}{X} - μ) / (σ / n) \sim N (0, 1)

and

V = (n - 1) S^{2} / σ^{2} \sim χ_{n - 1}^{2}

are independent; this ratio is the definition of a

t

variable.

Why It Holds / Derivation

:::proof[Mean and variance of $\overset{ˉ}{X}$ (Equation [2])]

Expectation. By linearity of expectation, which requires no independence:

E [\overset{ˉ}{X}_{n}] = E [\frac{1}{n} i = 1 \sum n X_{i}] = \frac{1}{n} i = 1 \sum n E [X_{i}] = \frac{1}{n} \cdot n μ = μ .

⟳Each

E [X_{i}] = μ

because the

X_{i}

are identically distributed.

Variance. Using $Var (aY) = a^{2} Var (Y)$ and additivity of variance for independent variables:

Var (\overset{ˉ}{X}_{n}) = \frac{1}{n ^{2}} Var (i = 1 \sum n X_{i}) = \frac{1}{n ^{2}} i = 1 \sum n Var (X_{i}) = \frac{1}{n ^{2}} \cdot n σ^{2} = \frac{σ ^{2}}{n} .

⟳Independence kills the

(2 n)

covariance cross-terms; only the diagonal variances survive.

Interpretation. The mean is unbiased; the variance shrinks as $1/ n$ , so the standard error decays as $1/ n$ . Quadrupling the sample size halves the standard error. $■$

Sketch of the CLT via characteristic functions

Center and scale: let $Y_{i} = (X_{i} - μ) / σ$ , so $E [Y_{i}] = 0$ , $Var (Y_{i}) = 1$ . Then $Z_{n} = \frac{1}{n} \sum_{i = 1}^{n} Y_{i}$ .

The characteristic function of $Z_{n}$ factors by independence:

φ_{Z_{n}} (t) = [φ_{Y} (\frac{t}{n})]^{n} .

⟳

φ_{\sum} (t) = \prod φ (t)

for independent sums, and

φ_{c Y} (t) = φ_{Y} (c t)

Taylor-expand $φ_{Y}$ near 0 using $φ_{Y} (0) = 1$ , $φ_{Y}^{'} (0) = i E [Y] = 0$ , $φ_{Y}^{''} (0) = - E [Y^{2}] = - 1$ :

φ_{Y} (\frac{t}{n}) = 1 - \frac{t ^{2}}{2 n} + o (\frac{1}{n}) .

⟳Finite variance guarantees the second-order Taylor term exists with the stated coefficient.

Therefore

φ_{Z_{n}} (t) = (1 - \frac{t ^{2}}{2 n} + o (1/ n))^{n} ⟶ e^{- t^{2} /2} .

⟳Use

lim_{n \to \infty} (1 + a_{n} / n)^{n} = e^{a}

when

a_{n} \to a

The limit $e^{- t^{2} /2}$ is exactly the characteristic function of $N (0, 1)$ . By Lévy's continuity theorem, pointwise convergence of characteristic functions implies convergence in distribution. $■$

William Sealy Gosset ("Student") | 1876–1937

Gosset was a chemist and statistician employed by the Guinness brewery in Dublin, where small-sample experiments on barley and yeast were routine. Forbidden by company policy from publishing under his own name, he wrote as "Student." In his 1908 paper The Probable Error of a Mean he derived the sampling distribution of $T = (\overset{ˉ}{X} - μ) / (S / n)$ — the Student's $t$ -distribution — recognizing that for small $n$ the substitution of $S$ for the unknown $σ$ inflates the tails beyond the normal curve. R. A. Fisher later supplied the rigorous geometric derivation and the degrees-of-freedom concept. Gosset's insight launched the modern theory of small-sample inference.

Building Intuition

Imagine a giant urn (the population) with a known average. You scoop out $n$ balls, average their numbers, write the result on a card, and return the balls. Repeat millions of times. The histogram of all those cards is the sampling distribution of the mean — it is not about any single scoop but about the variability of the scooping procedure.

Three things happen as $n$ grows: (1) the histogram centers ever more tightly on the true population mean (it stays centered at $μ$ but narrows); (2) its width shrinks like $1/ n$ ; and (3) regardless of how lumpy or skewed the urn's contents are, the histogram of averages smooths into a bell. The averaging process "blends" the irregularities of individual draws, and the more you blend, the more normal the result. This is why a single sample, properly analyzed, lets you quantify uncertainty about the whole population.

Sampling distribution of the mean narrows and normalizes as $n$ increases

Worked Examples

Standard error and a normal approximation

Given: A population of light-bulb lifetimes has mean

μ = 1200

hours and standard deviation

σ = 300

hours. A sample of

n = 36

bulbs is taken. Find the mean and standard error of

\overset{ˉ}{X}

, and approximate

P (\overset{ˉ}{X} > 1250)

. Steps:

Mean of the sampling distribution: $E [\overset{ˉ}{X}] = μ = 1200$ hours.
Standard error: $σ_{\overset{ˉ}{X}} = σ / n = 300/ 36 = 300/6 = 50$ hours.
By the CLT, $\overset{ˉ}{X} \approx N (1200, 5 0^{2})$ . Standardize:

z = \frac{1250 - 1200}{50} = 1.0

⟳subtract the mean and divide by the standard error to convert to a standard-normal

z

-score.

From normal tables, $P (Z > 1.0) = 1 - 0.8413 = 0.1587$ .

Answer:

E [\overset{ˉ}{X}] = 1200 h, σ_{\overset{ˉ}{X}} = 50 h, P (\overset{ˉ}{X} > 1250) \approx 0.159

Small-sample inference with the $t$-distribution

Given: From a normal population,

n = 10

measurements give sample mean

\overset{x}{ˉ} = 52.0

and sample standard deviation

s = 4.0

. The population standard deviation

σ

is unknown. Construct a 95% confidence interval for

μ

. Steps:

Because $σ$ is unknown and $n$ is small, the studentized mean follows $t_{n - 1} = t_{9}$ .
Standard error estimate: $s / n = 4.0/ 10 = 1.265$ .
Critical value: for 95% confidence with 9 degrees of freedom, $t_{0.025, 9} = 2.262$ .
Margin of error: $E = t \cdot s / n = 2.262 \times 1.265 = 2.861$ .
Interval:

\overset{x}{ˉ} \pm t_{0.025, 9} \frac{s}{n} = 52.0 \pm 2.861

⟳invert the pivot

T = (\overset{ˉ}{X} - μ) / (S / n) \sim t_{n - 1}

to bracket

μ

Answer:

(49.14, 54.86) — a 95% confidence interval for μ

Common Misconceptions

The sampling distribution is not the population distribution, nor the data distribution. Three distinct objects are routinely confused:

The population distribution describes individual values of $X$ (mean $μ$ , sd $σ$ ).
The data (empirical) distribution is the histogram of your one observed sample.
The sampling distribution describes a statistic across hypothetical repeated samples (e.g., $\overset{ˉ}{X}$ has sd $σ / n$ ).

Increasing

n

makes the sampling distribution narrower and more normal — but it does not change the population's shape, and it does not make your single data histogram normal.

The CLT does not save a heavy-tailed population

Consider a Cauchy population, whose density is $f (x) = \frac{1}{π ( 1 + x ^{2} )}$ . It has no finite mean or variance. The sample mean $\overset{ˉ}{X}_{n}$ of $n$ i.i.d. Cauchy variables is itself Cauchy with the same scale — it does not become normal and does not concentrate as $n \to \infty$ :

\overset{ˉ}{X}_{n} \sim Cauchy (0, 1) for every n .

⟳the Cauchy characteristic function is

e^{- ∣ t ∣}

; averaging gives

(e^{- ∣ t ∣/ n})^{n} = e^{- ∣ t ∣}

, unchanged.

The CLT's finite-variance hypothesis is essential — it is not a mere technicality.

Connections

The sampling distribution sits at the hub of inferential statistics. Its existence and shape are dictated by the Central Limit Theorem and the Law of Large Numbers (which guarantees the center converges to $μ$ ). The exact distributions under normality — the Chi-Squared Distribution for $S^{2}$ , Student's t-Distribution for the studentized mean, and the F-Distribution for ratios of variances — are all sampling distributions of specific statistics. Every Confidence Interval is built by inverting a pivot whose sampling distribution is known, and every Hypothesis Testing $p$ -value is a tail probability under the sampling distribution assumed by the null. The Standard Error is simply the standard deviation of a sampling distribution, and the Bootstrap Method estimates a sampling distribution empirically by resampling when analytic forms are unavailable.

Real-World Applications

Quality control (Shewhart control charts): $\overset{ˉ}{X}$ -charts plot sample means with $\pm 3 σ / n$ control limits derived from the sampling distribution to detect process drift — used by manufacturers like Toyota and Intel.
Election polling: A poll of $n \approx 1000$ voters reports a margin of error $\approx 1/ n \approx 3%$ , computed directly from the sampling distribution of a sample proportion.
Clinical trials (FDA drug approval): Confidence intervals and $t$ -tests for treatment effect sizes rely on the sampling distribution of the difference in sample means.
A/B testing in tech (Google, Netflix): Comparing conversion rates between website variants uses the sampling distribution of proportions and the CLT for two-sample tests.
Metrology and physics measurement: The standard uncertainty of a measured mean is reported as $σ / n$ , the standard error from the sampling distribution (NIST guidelines).
Survey sampling (census and labor statistics): The U.S. Bureau of Labor Statistics computes standard errors for unemployment estimates from sampling-distribution theory adjusted for complex designs.
Finance (Value at Risk): Sampling distributions of returns and their estimated quantiles underpin risk estimates, with bootstrap methods used when normality fails.

Historical Development

1733 — Abraham de Moivre approximates the binomial by the normal curve for large $n$ , the first special case of the CLT (for sample proportions).
1810 — Pierre-Simon Laplace generalizes de Moivre's result toward a broad central limit theorem in his Théorie analytique des probabilités.
1900 — Karl Pearson introduces the chi-squared distribution and goodness-of-fit testing, an early use of a statistic's sampling distribution.
1908 — William Gosset ("Student") publishes the $t$ -distribution for the small-sample mean with unknown variance.
1915–1925 — Ronald A. Fisher gives rigorous geometric derivations of the $t$ , $χ^{2}$ , and $F$ sampling distributions, formalizes degrees of freedom, and develops likelihood and sufficiency.
1922 — Fisher clarifies the conceptual distinction between population parameters and sampling distributions of estimators.
1920–1937 — Lindeberg and Lévy establish modern rigorous versions of the CLT, with Lindeberg's condition giving the precise requirements for asymptotic normality of sums.
1933 — Neyman & Egon Pearson build hypothesis-testing theory squarely on sampling distributions under null and alternative hypotheses.
1979 — Bradley Efron introduces the bootstrap, allowing sampling distributions to be approximated computationally without analytic formulas.

Subtleties & Limits

Several practical and theoretical issues complicate the clean asymptotic picture. How large is "large $n$ "? The CLT is asymptotic; for highly skewed populations (e.g., income, insurance claims) the sample mean may remain noticeably non-normal even at $n = 100$ , and there is no universal cutoff — the required $n$ depends on the population's skewness and kurtosis (quantified via the Berry–Esseen bound, which gives an $O (n^{- 1/2})$ convergence rate). Dependence and non-identical distributions: real data are often correlated (time series, clustered surveys) or non-i.i.d., requiring modified CLTs and effective-sample-size corrections. Heavy tails: when variance is infinite or undefined (as for Cauchy or some power-law distributions), the normal CLT fails entirely and stable-law limits with non- $n$ scaling take over. Sampling distributions of complex statistics (medians, quantiles, ratios, machine-learning model parameters) often have no closed form, motivating resampling and asymptotic-expansion methods — an active area where exact finite-sample behavior remains incompletely understood.