The Weak Law of Large Numbers

The WLLN is the theoretical backbone of statistical estimation: an estimator $\hat\theta_n$ is called consistent if $\hat\theta_n \xrightarrow{P} \theta$, and the most common way to establish consistency for sample means, sample variances, and many M-estimators is a direct or generalized WLLN argument. In Monte Carlo methods across computational statistics, physics, and finance, one estimates an intractable expectation $E[g(X)]$ by simulating i.i.d. draws and averaging $\frac{1}{n}\sum g(X_i)$; the WLLN is exactly what certifies that this simulated average converges to the true integral as the number of simulated draws grows, which is the theoretical justification for every Monte Carlo integration scheme. In survey sampling and polling, the law underlies the claim that a sample proportion from a sufficiently large random sample approximates the true population proportion, with the finite-sample Chebyshev-type bound (as in Worked Example 1) giving explicit, distribution-free confidence guarantees used in designing sample sizes. In information theory, the Asymptotic Equipartition Property — the statement that typical sequences carry almost all the probability mass — is proved via a weak-law argument applied to the log-probabilities of a random sequence. In statistical mechanics, the law of large numbers is the rigorous justification for replacing fluctuating microscopic quantities (like the empirical kinetic energy of $10^{23}$ particles) by their expected values, underwriting the reliability of macroscopic thermodynamic observables. Finally, the WLLN is the conceptual stepping stone to The Strong Law of Large Numbers (upgrading the mode of convergence to almost-sure) and to The Central Limit Theorem (which refines the WLLN by characterizing the size and shape, namely Gaussian, of the fluctuations of $\bar X_n$ around $\mu$ at the finer scale $1/\sqrt{n}$).

Ask the tutor: How does the proof change if we only assume the $X_i$ have a finite mean but no finite variance, as in Khinchin's version of the Weak Law?

Picture the sample mean $\bar X_n$ as a random point that jitters around on the real line as $n$ grows. Each new observation nudges the average, but the nudges get proportionally smaller: adding one more term to an average of $n$ numbers changes it by an amount of order $1/n$. The WLLN formalizes the idea that this jitter collapses onto the single point $\mu$, not because any individual observation is special, but because variance — a measure of spread — shrinks like $\sigma^2/n$. The mechanism is purely about variance control. If $X_1, \dots, X_n$ are independent with common variance $\sigma^2$, then

which is small for large $n$ even though $\operatorname{Var}(X_i) = \sigma^2$ never changes for the individual summands. Chebyshev's Inequality then translates "small variance" directly into "small probability of being far from the mean." The picture below shows how the distribution of $\bar X_n$ (schematically) concentrates as $n$ increases, while the picture that follows shows the Chebyshev bound itself as a shrinking window.

Example 1: A skewed exponential-type claim size. An insurance analyst models individual claim sizes $X_1, X_2, \dots$ (in thousands of dollars) as i.i.d. draws from an exponential distribution with rate $\lambda = 0.4$, so that $E[X_i] = 1/\lambda = 2.5$ and $\operatorname{Var}(X_i) = 1/\lambda^2 = 6.25$. This distribution is strongly right-skewed — nothing like a symmetric bell curve — which makes it a good stress test of the theorem: the WLLN does not require symmetry, only finite mean and variance. The analyst wants to know: how many claims $n$ are needed so that the average claim $\bar X_n$ is within $\varepsilon = 0.5$ (i.e. $500) of the true mean $\mu = 2.5$, with probability at least $0.95$ that the deviation is not exceeded — using only the Chebyshev-based guarantee from the WLLN's proof (no CLT normal approximation)? We want $n$ such that the Chebyshev bound satisfies

Plugging in $\sigma^2 = 6.25$ and $\varepsilon^2 = 0.25$:

So $n = 500$ claims suffice to guarantee (via the conservative Chebyshev bound built into the WLLN's proof) that the sample average is within $500 of the true mean with probability at least 95%. This is a distribution-free, non-asymptotic guarantee — it required no assumption on the shape of the exponential distribution beyond its first two moments, illustrating that the WLLN's proof mechanism gives usable finite-$n$ numbers, not just a qualitative limit. (As a sanity check on scale: the Central Limit Theorem, which the WLLN foreshadows, would give a much tighter estimate here — roughly $n \approx 1.96^2 \cdot 6.25/0.25 \approx 96$ — since Chebyshev's inequality is known to be a loose, distribution-agnostic bound. The point of this example is precisely that the WLLN's Chebyshev proof gives a valid but conservative answer using only variance, no distributional shape assumption.)

Example 2: Uncorrelated but dependent, non-identically-distributed terms. The theorem as stated allows something weaker than i.i.d.: pairwise uncorrelated variables with a common mean and uniformly bounded variance, not necessarily independent or identically distributed. Here is an explicit, asymmetric instance showing the theorem's reach. Let $U$ be uniform on $\{1, 2, 3, 4\}$ and define

where $R_i = \cos(2\pi i U/4)$ for $U$ a single random variable shared across all $i$ (so the $X_i$ are certainly not independent — they are all deterministic functions of the same $U$), $\mu = 10$, $\sigma = 3$. Compute $E[R_i]$: for fixed $i$, as $U$ ranges uniformly over $\{1,2,3,4\}$,

For $i$ not a multiple of $4$, this sum of the four unit vectors at angles $2\pi i u /4$ for $u=1,2,3,4$ is a discrete Fourier sum that vanishes: $\sum_{u=1}^4 \cos(2\pi i u/4) = 0$ whenever $4 \nmid i$ (the four terms are cosines at four equally spaced points on the circle summing to zero by symmetry of the roots of unity, since $\sum_{u=0}^{3} e^{2\pi i \cdot i u/4} = 0$ for $4 \nmid i$). So for such $i$, $E[X_i] = \mu = 10$. Meanwhile $\operatorname{Var}(R_i) \leq 1$ always since $|R_i| \leq 1$, so $\operatorname{Var}(X_i) \leq \sigma^2 = 9$. One can further check $\operatorname{Cov}(X_i, X_j) = \sigma^2 \operatorname{Cov}(R_i, R_j)$, and for indices $i \ne j$ with $4 \nmid (i-j)$ and $4\nmid(i+j)$ suitable product-to-sum identities again force $E[R_iR_j]=0$, matching $E[R_i]E[R_j]=0$, giving zero covariance for a large sub-family of indices — enough, by restricting to $i \in \{1,2,3\} \pmod 4$ (excluding multiples of 4), to build an infinite pairwise-uncorrelated subsequence with common mean $10$ and variance bounded by $9$, even though every $X_i$ is a deterministic function of one single random variable $U$ and the sequence is far from independent. By the theorem (using hypotheses (1)–(3) in their pairwise-uncorrelated form, not full independence), the sample mean of this subsequence still converges in probability to $\mu = 10$. This shows the genuine hypothesis of the theorem is uncorrelatedness with bounded variance, strictly weaker than the i.i.d. assumption usually quoted, and the proof via Chebyshev goes through verbatim.