The Law of Large Numbers

One-line statement: As the number of independent trials grows, the sample average converges to the true expected value. Key result in words: Random fluctuations average out: the empirical mean

\overset{ˉ}{X}_{n}

n

i.i.d. random variables approaches the population mean

μ

n \to \infty

. Where it applies: Probability, statistics, gambling theory, Monte Carlo methods, insurance, statistical mechanics — anywhere repeated random sampling occurs. Who/when: First proved by Jacob Bernoulli (published posthumously, 1713) for the binomial case; generalized through Chebyshev (1867), Khinchin (1929), and Kolmogorov (1930s).

The Law of Large Numbers (LLN) is the foundational theorem connecting abstract probability to observable frequency. Precisely, if $X_{1}, X_{2}, \dots$ are independent and identically distributed (i.i.d.) random variables with finite mean $μ = E [X_{i}]$ , then the sample mean $\overset{ˉ}{X}_{n} = \frac{1}{n} \sum_{i = 1}^{n} X_{i}$ converges to $μ$ — in probability (Weak LLN) or almost surely (Strong LLN). It matters because it justifies the entire enterprise of estimating unknown quantities by averaging measurements, and it gives the frequentist interpretation of probability its rigorous footing. The surprising fact is how weak the assumptions can be: finite mean alone (no variance needed) suffices for Khinchin's and Kolmogorov's theorems. Equally surprising is what the LLN does not say — it makes no claim that "luck balances out" in any individual sequence.

The Core Insight

Individual random outcomes are unpredictable, but their average is not. As you accumulate more independent samples, the contributions of unusual outcomes are diluted by a factor of $1/ n$ , while the systematic center $μ$ stays fixed. The randomness doesn't vanish — it gets divided away. The spread of the sample mean shrinks (its standard deviation falls like $1/ n$ ), forcing the average into an ever-tightening neighborhood of the true mean.

Formal Statement

Sample Mean and Setup

Let $X_{1}, X_{2}, \dots, X_{n}$ be random variables on a common probability space. The sample mean (empirical average) is

\overset{ˉ}{X}_{n} = \frac{1}{n} i = 1 \sum n X_{i} .

⟳This is just an arithmetic average of realized values, promoted to a random variable since each

X_{i}

is random.

Here:

$X_{i}$ — the $i$ -th observation, a real-valued random variable;
$n$ — the sample size (number of trials);
$μ = E [X_{i}]$ — the common (population) mean, assumed finite, $∣ μ ∣ < \infty$ ;
$σ^{2} = Var (X_{i})$ — the common variance (assumed finite for the elementary version);
"i.i.d." — independent and identically distributed: the $X_{i}$ share one distribution and are mutually independent.

Weak Law of Large Numbers (WLLN)

Let $X_{1}, X_{2}, \dots$ be i.i.d. with finite mean $μ$ . Then $\overset{ˉ}{X}_{n}$ converges to $μ$ in probability: for every $ε > 0$ ,

n \to \infty lim P (\overset{ˉ}{X}_{n} - μ \geq ε) = 0.

⟳Equivalently

P (∣ \overset{ˉ}{X}_{n} - μ ∣ < ε) \to 1

; this is the definition of convergence in probability,

\overset{ˉ}{X}_{n} P μ

(Khinchin's theorem requires only finite mean; the elementary proof below additionally assumes finite variance.)

Strong Law of Large Numbers (SLLN)

Let $X_{1}, X_{2}, \dots$ be i.i.d. with finite mean $μ$ (Kolmogorov, 1933). Then $\overset{ˉ}{X}_{n}$ converges to $μ$ almost surely:

P (n \to \infty lim \overset{ˉ}{X}_{n} = μ) = 1.

⟳Almost sure convergence means the set of sample sequences whose averages fail to converge to

μ

has probability zero — a strictly stronger conclusion than (2).

Weak vs. Strong — They Are Not the Same

The SLLN implies the WLLN, but not conversely. The WLLN says: for each large $n$ , the average is probably close to $μ$ . The SLLN says: with probability 1, the average settles down and stays close, eventually, forever. There exist sequences satisfying the WLLN but not the SLLN. The distinction is the difference between convergence in probability and almost-sure convergence.

Why It Holds / Derivation

WLLN via Chebyshev's Inequality (finite-variance case)

Step 1 — Mean of the sample mean. By linearity of expectation and the identical distribution,

E [\overset{ˉ}{X}_{n}] = \frac{1}{n} i = 1 \sum n E [X_{i}] = \frac{1}{n} \cdot n μ = μ .

⟳Expectation is linear regardless of dependence; each

E [X_{i}] = μ

Step 2 — Variance of the sample mean. Using independence (so covariances vanish) and

Var (c X) = c^{2} Var (X)

Var (\overset{ˉ}{X}_{n}) = \frac{1}{n ^{2}} i = 1 \sum n Var (X_{i}) = \frac{1}{n ^{2}} \cdot n σ^{2} = \frac{σ ^{2}}{n} .

⟳

Var (\frac{1}{n} \sum X_{i}) = \frac{1}{n ^{2}} Var (\sum X_{i})

; independence makes the variance of the sum the sum of variances.

Step 3 — Apply Chebyshev's inequality. For any random variable

Y

with mean

μ_{Y}

and variance

σ_{Y}^{2}

, and any

ε > 0

P (∣ Y - μ_{Y} ∣ \geq ε) \leq σ_{Y}^{2} / ε^{2}

. With

Y = \overset{ˉ}{X}_{n}

P (\overset{ˉ}{X}_{n} - μ \geq ε) \leq \frac{Var ( X ˉ _{n} )}{ε ^{2}} = \frac{σ ^{2}}{n ε ^{2}} .

⟳Substitute (4) and (5) into Chebyshev's bound; the only

n

-dependence is the

1/ n

factor.

Step 4 — Take the limit. Fix

ε > 0

. As

n \to \infty

, the right side

σ^{2} / (n ε^{2}) \to 0

. Since probabilities are nonnegative, the squeeze theorem gives

n \to \infty lim P (\overset{ˉ}{X}_{n} - μ \geq ε) = 0,

⟳

0 \leq P (\dots) \leq σ^{2} / (n ε^{2}) \to 0

forces the middle term to 0.

which is exactly (2). $■$

Sketch of the SLLN (Kolmogorov, finite 4th moment shortcut)

The full proof uses Kolmogorov's maximal inequality and the three-series theorem. A digestible route (assuming $E [X_{i}^{4}] < \infty$ , WLOG $μ = 0$ ) bounds the fourth moment of the partial sum $S_{n} = \sum X_{i}$ :

E [S_{n}^{4}] = n E [X_{1}^{4}] + 3 n (n - 1) (E [X_{1}^{2}])^{2} \leq C n^{2} .

⟳Expand

(\sum X_{i})^{4}

; independence and

E [X_{i}] = 0

kill all terms except those with each index appearing an even number of times — giving

n

fourth-power terms and

3 n (n - 1)

paired-square terms.

Then by Markov's inequality applied to $\overset{ˉ}{X}_{n}^{4} = S_{n}^{4} / n^{4}$ ,

n = 1 \sum \infty P (∣ \overset{ˉ}{X}_{n} ∣ \geq ε) \leq n = 1 \sum \infty \frac{E [ S _{n}^{4} ]}{n ^{4} ε ^{4}} \leq \frac{C}{ε ^{4}} n = 1 \sum \infty \frac{1}{n ^{2}} < \infty.

⟳

P (∣ \overset{ˉ}{X}_{n} ∣ \geq ε) = P (\overset{ˉ}{X}_{n}^{4} \geq ε^{4}) \leq E [\overset{ˉ}{X}_{n}^{4}] / ε^{4}

, then use (8); the resulting series is a convergent

p

-series (

p = 2

By the Borel–Cantelli lemma, since the probabilities are summable, $∣ \overset{ˉ}{X}_{n} ∣ \geq ε$ occurs only finitely often with probability 1. Letting $ε \to 0$ through a sequence yields $\overset{ˉ}{X}_{n} \to 0$ almost surely. Kolmogorov's achievement was removing the moment assumptions down to just finite mean. $■$

Jacob Bernoulli | 1655–1705

The Swiss mathematician Jacob (Jakob) Bernoulli proved the first version of the LLN — his "Golden Theorem" — in Ars Conjectandi, published posthumously in 1713 by his nephew Nicolaus. He showed that for repeated trials of a binary event with probability $p$ , the observed frequency converges (in probability) to $p$ , and crucially he derived an explicit bound on how many trials guarantee a given accuracy with given confidence — a remarkably modern, quantitative result. Bernoulli reportedly worked on the theorem for twenty years; he regarded it as so important that he wanted it to anchor probability as a serious mathematical discipline rather than a gambler's toy.

Building Intuition

Why Averages Are Tamer Than Individuals

Imagine each $X_{i}$ as a noisy vote about the true value $μ$ . A single vote can be wildly off. But independent errors point in random directions: some too high, some too low. When you average, the errors partially cancel. The key quantitative fact is (5): the standard deviation of the average is $σ / n$ , not $σ$ . To halve your uncertainty you need four times the data. This $n$ scaling is the price and the promise of the LLN — convergence is guaranteed but slow.

A useful mental image: the histogram of a single $X$ has fixed width $σ$ . The histogram of $\overset{ˉ}{X}_{n}$ is centered at the same place $μ$ but is narrower by $n$ , collapsing toward a spike at $μ$ .

Concentration of the sample mean: the distribution of $\bar X_n$ narrows around $\mu$ as $n$ grows

Worked Examples

Fair Coin: Frequency of Heads

Given: A fair coin,

P (heads) = p = 0.5

. Let

X_{i} = 1

if toss

i

is heads, else

0

(a Bernoulli variable). Here

μ = p = 0.5

and

σ^{2} = p (1 - p) = 0.25

. We flip it

n = 10, 000

times. Question: Bound the probability that the observed fraction of heads

\overset{ˉ}{X}_{n}

deviates from

0.5

by at least

ε = 0.01

. Steps:

Variance of the mean (eq. 5): $Var (\overset{ˉ}{X}_{n}) = σ^{2} / n = 0.25/10000 = 2.5 \times 1 0^{- 5}$ .
Chebyshev bound (eq. 6): $P (∣ \overset{ˉ}{X}_{n} - 0.5∣ \geq 0.01) \leq \frac{0.25}{10000 \times ( 0.01 ) ^{2}} = \frac{0.25}{1.0} = 0.25.$
Interpretation: at least a 75% chance the head-fraction lands within $[0.49, 0.51]$ . (Chebyshev is loose; the Central Limit Theorem gives the true probability $\approx 0.0455$ , i.e. ~95% inside.)

Answer:

P (∣ \overset{ˉ}{X}_{n} - 0.5∣ \geq 0.01) \leq 0.25

(dimensionless probability). The LLN guarantees this bound

\to 0

n \to \infty

Monte Carlo Estimation of an Integral

Given: Estimate

I = \int_{0}^{1} e^{x^{2}} d x

by sampling

U_{i} \sim Uniform (0, 1)

and setting

X_{i} = e^{U_{i}^{2}}

. Then

μ = E [X_{i}] = \int_{0}^{1} e^{x^{2}} d x = I

exactly, so

\overset{ˉ}{X}_{n} \to I

by the SLLN. Suppose we draw

n = 1, 000, 000

samples and find

\overset{ˉ}{X}_{n} = 1.4627

, with sample standard deviation

s = 0.494

. Question: Estimate

I

and quantify the error. Steps:

Point estimate: $\hat{I} = \overset{ˉ}{X}_{n} = 1.4627$ .
Standard error of the mean: $SE = s / n = 0.494/ 1 0^{6} = 0.494/1000 = 4.94 \times 1 0^{- 4}$ .
Approximate 95% interval (CLT): $\hat{I} \pm 1.96 SE = 1.4627 \pm 0.00097$ .
The true value is $I \approx 1.4627$ , inside the interval.

Answer:

I \approx 1.463 \pm 0.001

(dimensionless). Reducing the error tenfold requires

100 \times

more samples — the

1/ n

tax.

Common Misconceptions

The Gambler's Fallacy

The LLN does not say that deviations get corrected. After 10 heads in a row, tails is not "due." The coin has no memory; future tosses are independent. What actually happens is that the absolute number of excess heads can grow without bound — only the fraction tends to $1/2$ , because we divide by an ever-larger $n$ . The imbalance is diluted, not repaired.

Finite Mean Is Required

If $E [∣ X_{i} ∣] = \infty$ , the LLN can fail outright. The sample mean of i.i.d. variables with no finite mean does not converge to anything. Always check the mean exists before invoking the law.

Cauchy Distribution — No Convergence

Let $X_{1}, X_{2}, \dots$ be i.i.d. standard Cauchy (density $\frac{1}{π ( 1 + x ^{2} )}$ ). The mean does not exist (the defining integral diverges). A striking fact: the sample mean $\overset{ˉ}{X}_{n}$ of $n$ Cauchy variables is itself exactly standard Cauchy, for every $n$ .

\overset{ˉ}{X}_{n} = \frac{1}{n} i = 1 \sum n X_{i} = d X_{1} for all n .

⟳The Cauchy characteristic function is

φ (t) = e^{- ∣ t ∣}

; the sum of

n

of them has c.f.

e^{- n ∣ t ∣}

, and dividing by

n

rescales

t \to t / n

, returning

e^{- ∣ t ∣}

So averaging more Cauchy samples does not reduce the spread at all — $\overset{ˉ}{X}_{n}$ stays as wildly variable as a single observation. The LLN simply does not apply.

Connections

Where the LLN Sits in Probability

The LLN is the first of the two great limit theorems. Its partner, the Central Limit Theorem, refines it: while the LLN says $\overset{ˉ}{X}_{n} \to μ$ , the CLT describes the shape of the fluctuation, asserting $n (\overset{ˉ}{X}_{n} - μ) / σ \to N (0, 1)$ in distribution. The LLN sets the target; the CLT measures the wobble around it at scale $1/ n$ . The proof of the WLLN relies on Chebyshev's Inequality and on the Variance of a sum of Independent Random Variables. The SLLN rests on the Borel–Cantelli Lemma. Both laws presuppose a clean notion of Expected Value and the distinction between Convergence in Probability and Almost Sure Convergence.

Real-World Applications

Where the Law of Large Numbers Operates

Insurance and actuarial science. An insurer cannot predict any single policyholder's claim, but the average claim over millions of policies is highly predictable, letting premiums be set rationally. This is the literal business model of risk pooling.
Monte Carlo methods. Numerical integration, particle-physics simulations, and financial option pricing all estimate expectations by averaging random samples — valid precisely because of the SLLN (and error-quantified by the CLT).
Polling and survey statistics. A sample of a few thousand voters estimates a national proportion; the LLN guarantees consistency of the estimator as sample size grows.
Casino economics. With a fixed house edge per bet, the casino's average return per wager converges to that edge over millions of plays, making aggregate profit nearly deterministic even though each game is random.
Statistical mechanics. Macroscopic thermodynamic quantities (pressure, temperature) are averages over ~ $1 0^{23}$ molecules; their stability is essentially the LLN in physical guise.
Quality control and reliability. Estimating a defect rate by sampling a production line; the empirical defect frequency converges to the true rate.
Machine learning. Empirical risk minimization assumes the training-set average loss approximates the true expected loss — a direct LLN argument underpinning generalization theory.

Historical Development

From Bernoulli to Kolmogorov

1713 — Jacob Bernoulli's Ars Conjectandi (posthumous) proves the "Golden Theorem," the WLLN for binomial trials, with explicit error bounds.
1733–1738 — Abraham de Moivre develops the normal approximation to the binomial, foreshadowing both the LLN's refinement and the CLT.
1810s — Siméon Denis Poisson coins the phrase "la loi des grands nombres" (the law of large numbers) and extends Bernoulli's result to varying probabilities.
1837 — Poisson publishes a generalized version allowing non-identical trial probabilities.
1867 — Pafnuty Chebyshev gives a clean, general proof of the WLLN using his inequality, requiring only finite variance and not the binomial structure.
1909 — Émile Borel proves a Strong Law for binary digits (normal numbers), the first almost-sure version.
1917–1929 — Francesco Cantelli and Aleksandr Khinchin sharpen the strong and weak laws; Khinchin's WLLN needs only a finite mean.
1930–1933 — Andrey Kolmogorov proves the definitive SLLN: for i.i.d. variables, finite mean is necessary and sufficient for almost-sure convergence; he also gives his maximal inequality and three-series theorem.
1933 — Kolmogorov's Grundbegriffe axiomatizes probability via measure theory, placing the LLN on fully rigorous foundations.

Subtleties & Limits

Boundaries and Refinements

The classical LLN is settled, but its frontiers remain active. (1) Non-i.i.d. and dependent data: for stationary ergodic sequences, Birkhoff's ergodic theorem plays the LLN's role; for martingales there are martingale strong laws; for mixing or long-range-dependent processes, conditions are delicate and incomplete. (2) Rates of convergence: the LLN gives no rate by itself; the Berry–Esseen theorem, large-deviation principles (Cramér's theorem), and the law of the iterated logarithm (which pins the exact fluctuation size $2 σ^{2} n lo g lo g n$ of $S_{n}$ ) refine it, but optimal constants under minimal moments are still studied. (3) Heavy tails: when variance is infinite but the mean is finite, convergence holds but is slow and CLT-style normalization fails; the right framework is stable laws. (4) High-dimensional and matrix settings: laws of large numbers for random matrices, empirical measures, and U-statistics extend the idea but require new tools and have open quantitative questions.