Laws of Large Numbers and Convergence of Random Variables

A measurement repeated $n$ times returns a scatter of different numbers, yet every physicist reports the arithmetic mean and trusts it to approximate a "true" value. The Laws of Large Numbers make that trust into a theorem: for independent, identically distributed random variables with a finite mean, the sample average $\bar X_n$ converges to the expectation $\mu$ . The whole subject is deeper than it looks because of one word — converges. Random variables are functions on a sample space, and a sequence of functions can approach a limit in several genuinely inequivalent senses. Four modes organize everything. Almost-sure convergence ( $\bar X_n\to\mu$ along almost every infinite path) is strongest and underlies the Strong Law. Convergence in $L^p$ — mean-square when $p=2$ — controls the size of the fluctuations. Convergence in probability (the chance of a deviation vanishes) is weaker and gives the Weak Law. Convergence in distribution (only the CDFs need converge) is weakest, forgets the sample space entirely, and is the home of the Central Limit Theorem — which keeps the $O(1/\sqrt n)$ fluctuation the LLN discards and shows it is Gaussian. The toolkit is compact. Markov and Chebyshev inequalities turn moments into tail bounds; the Borel–Cantelli lemmas turn a summable tail bound into an almost-sure statement; and characteristic functions, through Lévy's continuity theorem, convert convergence of distributions into pointwise convergence of analytic transforms. With them the Weak Law is three lines and the Strong Law a little more, and all of ensemble averaging, Monte Carlo integration, and statistical error analysis stands firm: the LLN gives the answer, the CLT gives the error bar.

Key takeaways

The sample average $\bar X_n=\tfrac1n\sum_{i=1}^n X_i$ of i.i.d. random variables converges to the mean $\mu=\mathbb{E}[X]$ — the central message of the Laws of Large Numbers.
The Weak Law (WLLN) gives convergence in probability, $\bar X_n\xrightarrow{P}\mu$ , straight from Chebyshev's inequality when the variance is finite, with rate $\sigma^2/(n\varepsilon^2)$ .
The Strong Law (SLLN) gives the stronger almost-sure convergence $\bar X_n\xrightarrow{\text{a.s.}}\mu$ and needs only $\mathbb{E}|X|<\infty$ (Kolmogorov, 1930).
Four modes — almost sure, in $L^p$ , in probability, in distribution — form a strict hierarchy; only the weakest governs the CLT, while the LLN lives one or two rungs up.
Convergence in distribution is exactly pointwise convergence of characteristic functions (Lévy), the engine shared by the LLN ( $\varphi\to e^{i\mu t}$ ) and the CLT ( $\varphi\to e^{-\sigma^2t^2/2}$ ).
Key idea: sample averages converge to expectations, but "converge" has a precise hierarchy — almost sure ⇒ in probability ⇒ in distribution, and $L^2$ ⇒ in probability — and the LLN lives at the probability and almost-sure levels while the CLT lives at the distributional level.

The core insight

Sample averages converge to expectations, but "converge" has a precise hierarchy — almost sure $\Rightarrow$ in probability $\Rightarrow$ in distribution, and $L^2\Rightarrow$ in probability — and the Law of Large Numbers lives at the probability and almost-sure levels while the Central Limit Theorem lives at the distributional level.

Where this lives today

The Laws of Large Numbers are not a historical curiosity; they are the running assumption of every quantitative science. In lattice quantum field theory and statistical mechanics, Monte Carlo methods estimate high-dimensional path integrals by sampling: the SLLN guarantees the estimator converges to the true observable, and the CLT sets the $O(1/\sqrt n)$ error bar that dominates the computing budget. The ergodic hypothesis — that a single long time average equals the ensemble average — is Birkhoff's ergodic theorem, the dynamical generalization of the SLLN. Every quoted result of the form $\bar X_n\pm\hat\sigma/\sqrt n$ , from a tabletop $g$ to the mass of the Higgs boson, rests on the LLN for the central value and the CLT for the uncertainty; and the frequentist meaning of probability is literally the SLLN applied to indicators, $\hat p_n\to\mathbb{P}(A)$ almost surely. The open frontier is where the classical hypotheses fail: heavy-tailed data where $\mathbb{E}|X|=\infty$ and stable laws replace the Gaussian, and strongly dependent systems near critical points where covariances decay too slowly for the naive LLN.

The physical picture

Picture the sample mean $\bar X_n$ as a random point on the real line, and picture its sampling distribution. Its centre sits at $\mu$ for every $n$ , but its width is $\sigma/\sqrt n$ , so as $n$ grows it narrows into an ever-taller, ever-thinner spike over $\mu$ . The Law of Large Numbers is precisely this collapse to a spike; the Central Limit Theorem says that just before collapsing the spike is Gaussian. The first figure makes the narrowing quantitative.

Figure

The sampling distribution of the mean X̄ₙ collapses onto μ as n grows: its width σ/√n shrinks (for σ = 1) from 0.50 at n = 4 to 0.25 at n = 16 to 0.125 at n = 64 — the Law of Large Numbers is this collapse to a spike

The difference between the two laws is a difference between snapshots and whole paths. The Weak Law says the spike at each large fixed $n$ is narrow — it controls the distribution of a single $\bar X_n$ . The Strong Law constrains the entire infinite sequence at once: almost every path $\bar X_1,\bar X_2,\dots$ eventually enters any band around $\mu$ and never leaves. A sequence can jump out and back infinitely often while the probability of being out at any fixed instant still shrinks — convergence in probability without almost-sure convergence, exactly what the second figure's hierarchy keeps straight.

Figure

The four modes form a strict hierarchy — almost sure ⇒ in probability ⇒ in distribution, and L² ⇒ in probability — with the Weak Law at in-probability, the Strong Law at almost-sure, the CLT at in-distribution; no reverse arrow holds in general

Characteristic functions unify both laws and the CLT. Averaging multiplies them and rescales the argument, $\varphi_{\bar X_n}(t)=[\varphi_X(t/n)]^n$ : keep only the first moment and the limit is $e^{i\mu t}$ , a point mass at $\mu$ (the LLN); rescale by $\sqrt n$ and keep the second moment and it is $e^{-\sigma^2t^2/2}$ , a Gaussian (the CLT) — one machine, two attractors, selected by how hard you rescale.

The core equations

The starting point is bookkeeping: the mean of the average is the mean, and independence collapses the variance of the average by a factor of $n$ . Everything downstream follows from these two facts and the inequalities that exploit them.

\bar X_n=\frac{1}{n}\sum_{i=1}^{n}X_i,\qquad \mathbb{E}[\bar X_n]=\mu,\qquad \operatorname{Var}(\bar X_n)=\frac{\sigma^2}{n} \qquad (1)

where: $X_i$ — i.i.d. random variables; $\bar X_n$ — sample average of the first $n$ ; $\mu$ — common mean $\mathbb{E}[X_i]$ ; $\sigma^2$ — common variance, assumed finite here. Reading it: the estimator is unbiased at every $n$ , and its spread shrinks like $1/\sqrt n$ — that shrinking variance is the entire quantitative content of the Weak Law. Markov's inequality is the atom from which all the tail bounds are built: it converts a single moment into a bound on the probability of a large value.

\mathbb{P}\big(|Y|\ge a\big)\le\frac{\mathbb{E}|Y|^k}{a^k}\qquad(a>0,\ k>0) \qquad (2)

where: $Y$ — any random variable with finite $k$ -th absolute moment; $a$ — threshold; $k$ — any positive exponent, $k=1$ the classic Markov bound. Reading it: a big value is "expensive", contributing $a^k$ to the moment, so a finite moment forces large values to be rare. Applying Markov to the squared deviation $(X-\mu)^2$ gives Chebyshev's inequality — the workhorse that bounds fluctuations by the variance and powers the Weak Law.

\mathbb{P}\big(|X-\mu|\ge \varepsilon\big)\le\frac{\sigma^2}{\varepsilon^2} \qquad (3)

where: $\varepsilon$ — deviation tolerance; $\sigma^2$ — variance of $X$ . Reading it: the probability of straying more than $\varepsilon$ from the mean is capped by the variance in units of $\varepsilon^2$ , so a small variance forces the distribution to hug its mean. The Weak and Strong Laws are the two precise convergence statements; they differ only in the mode, and that difference is the crux of the subject.

\bar X_n\xrightarrow{P}\mu:\quad \lim_{n\to\infty}\mathbb{P}\big(|\bar X_n-\mu|\ge\varepsilon\big)=0\ \ \forall\varepsilon>0 \qquad (4)

\bar X_n\xrightarrow{\text{a.s.}}\mu:\quad \mathbb{P}\Big(\lim_{n\to\infty}\bar X_n=\mu\Big)=1 \qquad (5)

where: $\xrightarrow{P}$ — convergence in probability (each fixed- $n$ deviation probability vanishes); $\xrightarrow{\text{a.s.}}$ — almost-sure convergence (the whole path converges except on a null event). Reading them: (4) concerns the marginal law of one $\bar X_n$ ; (5) the joint behaviour of the entire sequence. The four modes are related by a fixed chain of implications; none of the reverse arrows holds in general.

X_n\xrightarrow{\text{a.s.}}X\ \Rightarrow\ X_n\xrightarrow{P}X\ \Rightarrow\ X_n\xrightarrow{d}X;\qquad X_n\xrightarrow{L^p}X\ \Rightarrow\ X_n\xrightarrow{P}X \qquad (6)

where: $\xrightarrow{d}$ — convergence in distribution (CDFs converge at every continuity point); $\xrightarrow{L^p}$ — convergence in $p$ -th mean, $\mathbb{E}|X_n-X|^p\to0$ ; almost-sure is finest, in-distribution coarsest. Lévy's continuity theorem makes characteristic functions decisive: convergence in distribution is equivalent to pointwise convergence of characteristic functions, provided the limit is continuous at the origin.

X_n\xrightarrow{d}X\ \iff\ \varphi_{X_n}(t)\to\varphi_X(t)\ \ \forall t,\quad \varphi_X\text{ continuous at }0 \qquad (7)

where: $\varphi_X(t)=\mathbb{E}[e^{itX}]$ — the characteristic function, the Fourier transform of the law. A hard question about distributions becomes an easy one about a smooth function. The first Borel–Cantelli lemma is the bridge from a summable tail bound to an almost-sure conclusion — the extra ingredient that upgrades the Weak Law to the Strong Law.

\sum_{n}\mathbb{P}(A_n)<\infty\ \Rightarrow\ \mathbb{P}\big(A_n\ \text{infinitely often}\big)=0 \qquad (8)

where: $A_n$ — a sequence of events; "infinitely often" is $\limsup_n A_n=\bigcap_N\bigcup_{n\ge N}A_n$ . Reading it: if the alarm probabilities are summable, then with probability one the alarm stops ringing after some finite time. Finally, the characteristic-function route packages the Weak Law in one line and shows why only a finite mean — not a finite variance — is truly needed.

\varphi_{\bar X_n}(t)=\Big[\varphi_X\!\big(t/n\big)\Big]^{n}\xrightarrow{\ n\to\infty\ } e^{\,i\mu t} \qquad (9)

where: $e^{i\mu t}$ — the characteristic function of the point mass at $\mu$ . Reading it: averaging drives the characteristic function to that of a constant, which by Lévy is convergence in distribution to $\mu$ — and a constant limit means convergence in probability.

Derivation AWeak Law of Large Numbers from Chebyshev

The goal is to prove the Weak Law in its most transparent form: assuming only a finite variance, we bound the probability that the sample mean strays from $\mu$ by a quantity vanishing like $1/n$ , so convergence in probability follows immediately with an explicit, usable rate. We compute the mean and variance of the average, feed the variance into Chebyshev's inequality, and take the limit — with mean-square ( $L^2$ ) convergence falling out for free from the same calculation.

Step 1. Take the mean of the average. By linearity, $\mathbb{E}[\bar X_n]=\tfrac1n\sum_{i=1}^n\mathbb{E}[X_i]=\tfrac1n\cdot n\mu=\mu$ . Why: the estimator is unbiased at every $n$ , so all convergence is about its spread, not its location.

Step 2. Take the variance of the average. Independence makes every cross-covariance vanish, so $\operatorname{Var}(\bar X_n)=\tfrac{1}{n^2}\sum_{i=1}^n\operatorname{Var}(X_i)=\tfrac{1}{n^2}\cdot n\sigma^2=\sigma^2/n$ . Why: this is where the whole effect lives — averaging $n$ independent copies divides the variance by $n$ .

Step 3. Apply Chebyshev to $\bar X_n$ about its own mean $\mu$ : $\mathbb{P}(|\bar X_n-\mu|\ge\varepsilon)\le \operatorname{Var}(\bar X_n)/\varepsilon^2=\sigma^2/(n\varepsilon^2)$ . Why: Chebyshev turns the shrinking variance into a shrinking deviation probability, with an explicit constant to size a sample.

Step 4. Fix $\varepsilon>0$ and let $n\to\infty$ . The bound $\sigma^2/(n\varepsilon^2)\to 0$ , so $\mathbb{P}(|\bar X_n-\mu|\ge\varepsilon)\to0$ for every $\varepsilon$ — this is $\bar X_n\xrightarrow{P}\mu$ . Since $\mathbb{E}[(\bar X_n-\mu)^2]=\sigma^2/n\to0$ is the same computation, we simultaneously have $L^2$ convergence. Why: one variance calculation delivers two modes at once, the $L^2\Rightarrow P$ side of the hierarchy.

Go deeper · Derivation B · 1 minStrong Law under a finite fourth moment via Borel–Cantelli

Derivation BStrong Law under a finite fourth moment via Borel–Cantelli

The goal is to upgrade convergence in probability to almost-sure convergence, which needs more than a $1/n$ bound: the Chebyshev bound $\sigma^2/(n\varepsilon^2)$ is not summable, so it cannot control the whole infinite path. We buy a summable $1/n^2$ bound with a fourth moment — expanding $\mathbb{E}[(\sum X_i)^4]$ and counting the terms that survive independence — then let the first Borel–Cantelli lemma turn summability into "large deviations happen only finitely often", exactly almost-sure convergence. Take $\mu=0$ and $\mathbb{E}[X^4]=K<\infty$ .

Step 1. Expand the fourth power of the sum. Every term with an odd power of a single mean-zero variable averages to zero by independence, leaving only the pure-quartic and paired-square terms: $\mathbb{E}\big[(\sum_i X_i)^4\big]=\sum_i\mathbb{E}[X_i^4]+3\sum_{i\ne j}\mathbb{E}[X_i^2]\mathbb{E}[X_j^2]$ . Why: independence and zero mean annihilate most of the $n^4$ terms, leaving two families to count.

Step 2. Count the survivors. There are $n$ quartic terms giving $nK$ , and $n(n-1)$ ordered pairs each contributing $\sigma^4$ with weight $3$ , giving $3n(n-1)\sigma^4$ . So the total is $nK+3n(n-1)\sigma^4\le Cn^2$ . Why: the paired-square terms number $\sim n^2$ and dominate, so the sum's fourth moment grows like $n^2$ , not $n^4$ .

Step 3. Divide by $n^4$ for the fourth moment of the average: $\mathbb{E}[\bar X_n^4]=n^{-4}\,\mathbb{E}[(\sum_i X_i)^4]\le C/n^2$ . Why: the decisive gain — a fourth-power average whose expectation decays like $1/n^2$ , fast enough to sum.

Step 4. Apply Markov with $k=4$ : $\mathbb{P}(|\bar X_n|\ge\varepsilon)\le \mathbb{E}[\bar X_n^4]/\varepsilon^4\le C/(n^2\varepsilon^4)$ . Since $\sum_n 1/n^2<\infty$ , the deviation probabilities are summable for every fixed $\varepsilon$ . Why: the fourth moment gives the extra power of $n$ that makes the series converge, which the variance could not.

Step 5. Invoke Borel–Cantelli. Summability implies $\mathbb{P}(|\bar X_n|\ge\varepsilon\ \text{i.o.})=0$ for each $\varepsilon$ ; taking $\varepsilon\downarrow0$ along a countable sequence gives $\bar X_n\to0$ almost surely. Why: "only finitely many exceedances of every band" is exactly almost-sure convergence — the whole path settles. *The fourth-moment expansion $\mathbb{E}[(\sum X_i)^4]=nK+3n(n-1)\sigma^4$ is the key counting result of this proof. Kolmogorov's full SLLN removes the fourth-moment assumption using a maximal inequality.*

Go deeper · Derivation C · 1 minWeak Law from characteristic functions with no variance assumption

Derivation CWeak Law from characteristic functions with no variance assumption

The goal is to prove the Weak Law under the weakest natural hypothesis — a finite mean alone, no variance — using characteristic functions. Averaging factorizes and rescales the characteristic function, a first-order Taylor expansion captures the finite mean, and $(1+z/n)^n\to e^z$ collapses the result onto the characteristic function of a point mass, which Lévy certifies as convergence in distribution to a constant, equivalent to convergence in probability. Assume only $\mathbb{E}|X|<\infty$ , so $\varphi_X'(0)=i\mu$ .

Step 1. Factorize. Independence turns the expectation of an exponential of a sum into a product: $\varphi_{\bar X_n}(t)=\mathbb{E}[e^{it\bar X_n}]=\prod_{i=1}^n\mathbb{E}[e^{i(t/n)X_i}]=[\varphi_X(t/n)]^n$ . Why: averaging multiplies characteristic functions and rescales the argument by $1/n$ .

Step 2. Expand at the origin. Because the first moment exists, $\varphi_X(s)=1+i\mu s+o(s)$ ; setting $s=t/n$ gives $\varphi_X(t/n)=1+i\mu t/n+o(1/n)$ . Why: only the first Taylor coefficient survives, exactly $i\mu$ — where the finite-mean hypothesis, and nothing more, enters.

Step 3. Take the $n$ -th power and pass to the limit: $[1+i\mu t/n+o(1/n)]^n\to e^{i\mu t}$ by the standard exponential limit, the $o(1/n)$ terms contributing nothing. Why: the exponential limit turns a product of near-one factors into a closed form.

Step 4. Identify the limit. $e^{i\mu t}$ is the characteristic function of a point mass at $\mu$ ; by Lévy's theorem $\bar X_n\xrightarrow{d}\mu$ , and convergence in distribution to a constant is equivalent to convergence in probability, so $\bar X_n\xrightarrow{P}\mu$ . Why: a degenerate limit collapses the two weakest modes, giving the Weak Law under a weaker hypothesis than Chebyshev needs. *The hypothesis $\mathbb{E}|X|<\infty$ used here is strictly weaker than the finite-variance assumption the Chebyshev proof requires.*

Limiting cases

The strength of what you can conclude depends on what you assume about the moments and independence of the $X_i$ :

Case	Result	Meaning
Finite variance	$\operatorname{Var}(X)=\sigma^2<\infty$	Chebyshev gives the WLLN in one line, $\bar X_n\xrightarrow{P}\mu$ , plus $L^2$ convergence since $\mathbb{E}[(\bar X_n-\mu)^2]=\sigma^2/n\to0$ .
Finite mean only	$\mathbb{E}\lvert X\rvert<\infty$	Kolmogorov's Strong Law still holds, $\bar X_n\xrightarrow{\text{a.s.}}\mu$ ; no variance needed, but the proof is far subtler than the fourth-moment case.
Infinite mean, heavy tails	$\mathbb{E}\lvert X\rvert=\infty$	Both laws fail. For the standard Cauchy, $\bar X_n$ is itself standard Cauchy for every $n$ and converges to nothing.
Weak dependence	$\operatorname{Cov}(X_i,X_j)\to0$ fast enough	Ergodic theorems extend the LLN to stationary dependent sequences; time averages equal ensemble averages (Birkhoff).

Common trapConvergence in probability is the same as convergence with probability one

It is not, and conflating them is the most common error in this subject. The Weak Law gives convergence in probability: at each large fixed $n$ the sample mean is very likely close to $\mu$ , but the sequence may still leave any band around $\mu$ infinitely often. The Strong Law gives convergence almost surely: almost every whole path eventually enters the band and stays forever. Almost-sure is strictly stronger — it implies in-probability, not conversely. Two related traps ride along: convergence in distribution constrains only the CDFs, not how close the variables are on a given sample; and $\mathbb{E}[\bar X_n]\to\mu$ alone implies no pathwise convergence. And never invoke Chebyshev without checking the variance is finite — for a Cauchy it is infinite and the bound is vacuous.

Worked examples

1 · How many trials does the Weak Law actually need? (fully worked)

You estimate the bias $p$ of a coin by the sample frequency $\bar X_n$ of heads. How large must $n$ be so that $\mathbb{P}(|\bar X_n-p|\ge 0.01)\le 0.05$ ? Take the worst case $p=\tfrac12$ , and compare Chebyshev with what the CLT promises.

Step 1. Each toss is Bernoulli with variance $\sigma^2=p(1-p)$ , maximized at $p=\tfrac12$ where $\sigma^2=0.25$ . The sample mean has variance $\sigma^2/n$ , so Chebyshev gives $\mathbb{P}(|\bar X_n-p|\ge\varepsilon)\le \sigma^2/(n\varepsilon^2)$ with $\varepsilon=0.01$ , so $\varepsilon^2=10^{-4}$ .

Step 2. Require the bound to be at most $0.05$ : $\dfrac{0.25}{n\cdot 10^{-4}}\le 0.05 \Rightarrow n\ge \dfrac{0.25}{0.05\cdot 10^{-4}}=\dfrac{0.25}{5\times10^{-6}}=\mathbf{50\,000}$ . So Chebyshev guarantees the target with fifty thousand tosses.

Step 3. Compare the CLT. Since $\bar X_n\approx N(p,\sigma^2/n)$ , a two-sided $95\%$ deviation needs $\varepsilon\sqrt n/\sigma=1.96$ , i.e. $\sqrt n=1.96\cdot 0.5/0.01=98$ , so $n\approx \mathbf{9\,604}$ . The CLT needs roughly five times fewer trials than Chebyshev demands.

Step 4. Read the gap. Chebyshev is distribution-free and therefore loose; the CLT uses the actual Gaussian shape and is sharp. Chebyshev: n ≥ 50 000. CLT: n ≈ 9 600. The LLN says the estimate converges; the CLT says how fast — the number a real experiment budgets against.

2 · Monte Carlo estimation of π and its error bar

Drop $n$ points uniformly in the unit square and let $\bar X_n$ be the fraction landing inside the quarter-circle $x^2+y^2\le1$ . Then $4\bar X_n$ estimates π. What is the error after $n=10^6$ points, and how many are needed for one more decimal?

Step 1. Each point is an indicator with success probability $p=\pi/4\approx 0.7854$ , so $\sigma^2=p(1-p)=0.1686$ . By the SLLN, $4\bar X_n\to\pi$ almost surely; by the CLT the standard error of the estimate of π is $\mathrm{SE}=4\sqrt{p(1-p)/n}$ .

Step 2. At $n=10^6$ : $\mathrm{SE}=4\sqrt{0.1686/10^6}=4\times 4.11\times10^{-4}=\mathbf{1.64\times10^{-3}}$ . So a million points give $\hat\pi=3.1416\pm0.0016$ at one sigma — only about three correct decimals.

Step 3. The error scales as $1/\sqrt n$ , so shrinking it tenfold (to $1.6\times10^{-4}$ ) costs a hundredfold more samples: $n\approx 10^8$ . Monte Carlo buys one extra decimal per hundredfold more work — the famous $O(1/\sqrt n)$ , which still beats grid quadrature in high dimension.

3 · Why the sample mean of Cauchy data never settles

Draw $X_i$ from the standard Cauchy distribution ( $\varphi_X(t)=e^{-|t|}$ ). What is the distribution of $\bar X_n$ ?

Step 1. The averaging rule gives $\varphi_{\bar X_n}(t)=[\varphi_X(t/n)]^n=[e^{-|t|/n}]^n=e^{-|t|}$ — the exponent $n\cdot(|t|/n)=|t|$ is independent of $n$ . So $\bar X_n$ is standard Cauchy for every $n$ : averaging a million draws is exactly as noisy as taking one. The LLN fails because $\mathbb{E}|X|=\infty$ — the finite-mean assumption is not a technicality but the load-bearing hypothesis. (The sample median, by contrast, does converge.)

Physics of the portrait

The Laws of Large Numbers are three centuries old as intuition but less than a century old as theorems, and the decisive figure both proved the general Strong Law and, three years later, wrote down the axioms that finally said what "converges" even means.

Origins · 1 minAndrey Kolmogorov and how this was discovered

Andrey Kolmogorov

1903–1987 · foundations of probability and the strong law of large numbers

Andrey Nikolaevich Kolmogorov was the twentieth century's foremost probabilist, contributing to turbulence, dynamical systems, and information theory as well as to the field he founded on rigorous ground. The Law of Large Numbers reached him as a two-hundred-year-old chain of partial results — Bernoulli's "golden theorem" of 1713, Poisson's naming of la loi des grands nombres in 1837, Chebyshev's inequality-based Weak Law, and Borel's 1909 Strong Law for the digits of a random number. In 1930 he proved the Strong Law in full generality: the sample mean of i.i.d. variables converges almost surely to $\mu$ if and only if $\mathbb{E}|X|<\infty$ , a necessary-and-sufficient statement that closed the classical theory. Then in 1933 his Grundbegriffe der Wahrscheinlichkeitsrechnung recast all of probability on measure theory — a probability is a measure of total mass one, a random variable a measurable function — and only in that framework can one even distinguish convergence almost surely from convergence in probability. The man who proved the Strong Law also supplied the language in which its subtlety can be stated.

History

1713

— Jacob Bernoulli proves the first law of large numbers, the "golden theorem", in Ars Conjectandi

1837

— Poisson coins the name "law of large numbers"

1867

— Chebyshev's inequality yields a short proof of the weak law

1909

— Borel proves the first strong law, for the binary digits of a random real number

1930

— Kolmogorov establishes the strong law under only a finite mean

Going further

Texts. Stone & Goldbart, Mathematics for Physics (CUP, 2009), probability chapter · Arfken, Weber & Harris, Mathematical Methods for Physicists, 7th ed. · Riley, Hobson & Bence, Mathematical Methods for Physics and Engineering, 3rd ed., Ch. 30 · Grimmett & Stirzaker, Probability and Random Processes, 3rd ed., Ch. 7 · Billingsley, Probability and Measure, 3rd ed., Ch. 1, 6, 22. The key checkpoints of this article are: the Chebyshev rate $\sigma^2/(n\varepsilon^2)$ ; the characteristic-function limit $[\varphi_X(t/n)]^n\to e^{i\mu t}$ ; the fourth-moment expansion $\mathbb{E}[(\sum X_i)^4]=nK+3n(n-1)\sigma^4$ ; the Cauchy stability $\varphi_{\bar X_n}=e^{-|t|}$ ; and the worked numbers — Chebyshev $n\ge 50\,000$ versus CLT $n\approx 9\,600$ for the coin, and $\mathrm{SE}=4\sqrt{p(1-p)/n}=1.64\times10^{-3}$ at $n=10^6$ for the Monte Carlo π. No constants beyond the combinatorial $3$ are asserted.

Prerequisites. Expectation, Variance and Moments · Random Variables and Probability Distributions · Axioms of Probability · Moment-Generating and Characteristic Functions. Leads to. Central Limit Theorem · Gaussian (Normal) Distribution · Poisson Distribution.

Ask the tutor: ask me to walk through any convergence mode with an explicit sequence that has it but fails the next one up, or to size the sample for your own experiment with Chebyshev and CLT estimates.

What is the difference between the Weak and the Strong Law in one sentence?

The Weak Law controls each snapshot — at every large fixed $n$ the sample mean is very probably close to $\mu$ — while the Strong Law controls the whole movie — almost every infinite sequence of running averages eventually enters any band around $\mu$ and never leaves. Formally the Weak Law is convergence in probability, the Strong Law convergence almost surely, and the latter is strictly stronger: it implies the former, not the reverse. In practice the Weak Law justifies "one more measurement is probably good", the Strong Law "keep measuring forever and the running average is guaranteed to settle".

Why is a finite variance not required for the Law of Large Numbers?

A finite variance is a convenience, not a necessity. The Chebyshev proof uses it because the variance is the moment Chebyshev bounds, but the characteristic-function proof needs only a finite mean: a first-order Taylor expansion of $\varphi_X$ captures $\mathbb{E}[X]=\mu$ and nothing else, which alone forces $\varphi_{\bar X_n}(t)\to e^{i\mu t}$ . Kolmogorov's Strong Law makes this sharp — the sample mean converges almost surely iff $\mathbb{E}|X|<\infty$ . So the true dividing line is a finite mean; what breaks the law is an infinite mean, as for the Cauchy, where there is no finite $\mu$ to converge to.

What exactly is convergence in distribution, and why is it the weakest mode?

Convergence in distribution asks only that the CDFs $F_{X_n}(x)\to F_X(x)$ at every continuity point of $F_X$ . It says nothing about whether $X_n$ and $X$ are ever close on a given sample — it constrains only the shape of the histogram. That is why it is weakest: i.i.d. standard normals converge in distribution to $N(0,1)$ trivially, yet successive terms never get close, so there is no in-probability limit. It is nonetheless the right and only mode for the Central Limit Theorem, whose statement is entirely about the limiting shape of the rescaled sum, not pathwise closeness.

How do characteristic functions unify the Law of Large Numbers and the Central Limit Theorem?

Both are one computation with different rescalings. Averaging gives $\varphi_{\bar X_n}(t)=[\varphi_X(t/n)]^n$ , and near the origin $\varphi_X(s)=1+i\mu s-\tfrac12\mathbb{E}[X^2]s^2+\cdots$ . For the LLN you scale the argument by $1/n$ , keep only the linear term, and get $e^{i\mu t}$ — a point mass at $\mu$ . For the CLT you subtract $\mu$ and scale by $1/\sqrt n$ , which keeps the quadratic term and gives $e^{-\sigma^2t^2/2}$ — a Gaussian. Lévy's theorem certifies in both cases that pointwise convergence to something continuous at $0$ is convergence in distribution: one machine, two attractors, chosen by how hard you rescale.

What role does the Borel–Cantelli lemma play, and why can't Chebyshev alone give the Strong Law?

Almost-sure convergence is a statement about the whole infinite path, and the bridge to it is the first Borel–Cantelli lemma: if the deviation probabilities are summable over $n$ , then with probability one only finitely many deviations occur — exactly almost-sure convergence. The trouble with Chebyshev is its bound $\sigma^2/(n\varepsilon^2)$ , whose sum $\sum 1/n$ diverges, so Borel–Cantelli says nothing. A fourth moment sharpens the bound to $C/(n^2\varepsilon^4)$ , whose sum $\sum 1/n^2$ converges, and the lemma fires. Kolmogorov's general proof avoids the fourth moment with a maximal inequality, but the logic is identical.

How does the Law of Large Numbers justify Monte Carlo integration and error bars?

To estimate $I=\int f\,\mathrm{d}\mu$ you sample $X_i\sim\mu$ and average $\bar f_n=\tfrac1n\sum f(X_i)$ . The Strong Law guarantees $\bar f_n\to I$ almost surely, so the method is consistent; the Central Limit Theorem supplies the error bar: for finite $\operatorname{Var}(f)$ the estimator is approximately $N(I,\operatorname{Var}(f)/n)$ , so the one-sigma error is $\sqrt{\operatorname{Var}(f)/n}$ — the celebrated $O(1/\sqrt n)$ , independent of dimension. That dimension-independence is why Monte Carlo beats grid quadrature for high-dimensional integrals in field theory and statistical mechanics, even though one extra decimal costs a hundredfold more samples.