Convergence of Random Variables

Convergence of random variables describes the various precise senses in which a sequence

X_{1}, X_{2}, \dots

of random variables can "approach" a limiting random variable

X

. The four principal modes — almost sure, in probability, in

L^{p}

(mean), and in distribution — form a strict hierarchy of strength. They underpin the Law of Large Numbers and the Central Limit Theorem, and were forged into rigour by Kolmogorov, Lévy, and others in the early 20th century.

A sequence of random variables $(X_{n})$ is not a sequence of numbers, so " $X_{n} \to X$ " requires careful definition — and there is no single correct one. Almost sure convergence demands that the realized sample paths converge for all outcomes except a null set; convergence in probability asks only that the chance of a large discrepancy vanish; $L^{p}$ convergence controls an averaged error; and convergence in distribution asks merely that the cumulative distribution functions agree in the limit. These modes are linked by a fixed chain of implications, and the surprising fact is that the weakest mode, convergence in distribution, requires no common probability space at all — the variables may live in entirely separate worlds.

The Core Insight

A random variable is a function on a sample space, so "convergence" inherits the full ambiguity of how a sequence of functions can converge. The question "where does the error live?" — in the sample space (almost sure), in the probability of failure (in probability), in an integral (in mean), or only in the shape of the distribution (in law) — selects the mode. Strength is measured by how much information the mode preserves about the joint behaviour of the $X_{n}$ and $X$ .

Formal Statement

Throughout, fix a probability space $(Ω, F, P)$ on which the random variables $X_{n}$ and $X$ are defined (except for convergence in distribution, where this is unnecessary).

Almost Sure Convergence

X_{n}

converges to

X

almost surely (written

X_{n} a.s. X

) if the set of outcomes on which the numerical sequence converges has probability one:

P ({ω \in Ω : n \to \infty lim X_{n} (ω) = X (ω)}) = 1.

⟳This is ordinary pointwise convergence of the functions

ω \mapsto X_{n} (ω)

, but tolerated to fail on a set of probability zero (a null set).

Here $ω$ is a sample outcome, $X_{n} (ω)$ is the realized value, and the limit is the usual limit of real numbers.

Convergence in Probability

X_{n}

converges to

X

in probability (written

X_{n} P X

) if for every

ε > 0

n \to \infty lim P (∣ X_{n} - X ∣ > ε) = 0.

⟳Demand that the probability of an error exceeding any fixed tolerance

ε

shrinks to zero as

n

grows; the failure set may change identity with

n

The symbol $ε > 0$ is an arbitrary error tolerance; the event ${∣ X_{n} - X ∣ > ε}$ is the "bad set" at stage $n$ .

Convergence in $L^p$ (Mean of Order $p$)

For $p \geq 1$ , $X_{n}$ converges to $X$ in $L^{p}$ (written $X_{n} L^{p} X$ ) if all variables lie in $L^{p}$ and

n \to \infty lim E [∣ X_{n} - X ∣^{p}] = 0.

⟳Measure the error by its

p

-th moment (an integral against

P

) and require that integrated quantity to vanish;

p = 1

is "in mean",

p = 2

is "in mean square".

E [\cdot]

denotes expectation. The case

p = 2

(mean-square convergence) dominates statistics and signal processing.

Convergence in Distribution

X_{n}

converges to

X

in distribution (or in law, or weakly; written

X_{n} d X

) if the cumulative distribution functions

F_{X_{n}} (x) = P (X_{n} \leq x)

satisfy

n \to \infty lim F_{X_{n}} (x) = F_{X} (x) at every x where F_{X} is continuous .

⟳Compare the distribution functions pointwise, but excuse disagreement at jump points of the limiting law

F_{X}

, where a literal demand would be too rigid.

Crucially, $X_{n}$ and $X$ need not share a probability space — only their laws are compared.

The Hierarchy of Convergence

The four modes satisfy the following strict implications, none of which reverses in general:

(X_{n} a.s. X) ⟹ (X_{n} P X) ⟹ (X_{n} d X),

(X_{n} L^{p} X) ⟹ (X_{n} P X), and for q > p \geq 1, (X_{n} L^{q} X) ⟹ (X_{n} L^{p} X) .

⟳a.s.

\Rightarrow

in prob. follows from continuity of measure on shrinking tail events;

L^{p} \Rightarrow

in prob. is Markov's inequality; in prob.

\Rightarrow

in dist. follows by squeezing CDFs; the

L^{q} \Rightarrow L^{p}

nesting is Jensen/Lyapunov.

There is no general implication between almost sure and $L^{p}$ convergence: each can hold without the other.

Why It Holds / Derivation

Almost Sure ⇒ In Probability

Step 1. Fix

ε > 0

. Define the tail events

A_{n} = m \geq n ⋃ {∣ X_{m} - X ∣ > ε} .

⟳

A_{n}

is the event that

some future term beyond $n$ exceeds the tolerance. Step 2. The sets

A_{n}

are nested decreasing:

A_{1} \supseteq A_{2} \supseteq \dots

. On the event of a.s. convergence, for each such

ω

there exists

N (ω)

with

∣ X_{m} (ω) - X (ω) ∣ \leq ε

for all

m \geq N (ω)

, so

ω \in / A_{n}

for

n > N (ω)

. Hence

P (n = 1 ⋂ \infty A_{n}) \leq P ({X_{n} \neq \to X}) = 0.

⟳The intersection of all

A_{n}

is contained in the divergence set, which has probability zero by hypothesis.

Step 3. By continuity of measure from above (since

P (A_{1}) \leq 1 < \infty

n \to \infty lim P (A_{n}) = P (n ⋂ A_{n}) = 0.

⟳Downward continuity of a finite measure converts the nested intersection into a limit of probabilities.

Step 4. Since

{∣ X_{n} - X ∣ > ε} \subseteq A_{n}

, monotonicity gives

P (∣ X_{n} - X ∣ > ε) \leq P (A_{n}) \to 0

. As

ε

was arbitrary,

X_{n} P X

■

In Probability ⇒ In Distribution

Step 1. Let

x

be a continuity point of

F_{X}

, and fix

ε > 0

. Split using the law of total probability on the discrepancy event:

F_{X_{n}} (x) = P (X_{n} \leq x) \leq P (X \leq x + ε) + P (∣ X_{n} - X ∣ > ε) .

⟳If

X_{n} \leq x

but

X > x + ε

, then

∣ X_{n} - X ∣ > ε

; otherwise

X \leq x + ε

Step 2. Taking

lim sup

and using convergence in probability,

n lim sup F_{X_{n}} (x) \leq F_{X} (x + ε) .

⟳The error term vanishes, leaving the shifted CDF.

Step 3. Symmetrically,

P (X \leq x - ε) \leq F_{X_{n}} (x) + P (∣ X_{n} - X ∣ > ε)

, giving

F_{X} (x - ε) \leq lim inf_{n} F_{X_{n}} (x)

. Combining:

F_{X} (x - ε) \leq n lim inf F_{X_{n}} (x) \leq n lim sup F_{X_{n}} (x) \leq F_{X} (x + ε) .

⟳A two-sided squeeze brackets the limit between shifted values of

F_{X}

Step 4. Let

ε ↓ 0

. By continuity of

F_{X}

x

, both outer bounds tend to

F_{X} (x)

, forcing

F_{X_{n}} (x) \to F_{X} (x)

■

$L^p$ ⇒ In Probability (via Markov)

Step 1. For

ε > 0

, apply Markov's inequality to the nonnegative variable

∣ X_{n} - X ∣^{p}

P (∣ X_{n} - X ∣ > ε) = P (∣ X_{n} - X ∣^{p} > ε^{p}) \leq \frac{E [ ∣ X _{n} - X ∣ ^{p} ]}{ε ^{p}} .

⟳Markov:

P (Y \geq a) \leq E [Y] / a

for

Y \geq 0

, applied with

Y = ∣ X_{n} - X ∣^{p}

a = ε^{p}

Step 2. By hypothesis

E [∣ X_{n} - X ∣^{p}] \to 0

, so the right side

\to 0

for every fixed

ε

. Hence

X_{n} P X

■

Andrey Kolmogorov | 1903–1987

Kolmogorov's 1933 monograph Grundbegriffe der Wahrscheinlichkeitsrechnung placed probability on a measure-theoretic foundation, making " $X_{n} \to X$ " a precise question about measurable functions and integrals. His Strong Law of Large Numbers is the archetypal almost-sure convergence result, and his Three-Series and zero–one laws are the engines for proving a.s. statements. By identifying random variables with measurable functions, he unified the previously disparate analytic and probabilistic notions of convergence.

Building Intuition

Picture each $ω$ as a fixed "lottery draw" that, once chosen, determines the entire numerical sequence $X_{1} (ω), X_{2} (ω), \dots$ .

Almost sure asks: for almost every fixed draw, does the resulting number-sequence settle down? The drama happens inside individual paths.
In probability asks: at each time $n$ , is it unlikely that the current value is far off? It says nothing about whether the same draw keeps misbehaving — the bad set may hop around.
$L^{p}$ asks: is the average size of the error small? A rare but catastrophic error can spoil $L^{p}$ while leaving "in probability" intact.
In distribution forgets the joint picture entirely and asks only whether the histograms line up. Two independent fair coins are equal in distribution but never close path-by-path.

The gap between "in probability" and "almost sure" is the difference between the bad set being small at each instant and the bad set eventually vanishing for good.

The strict hierarchy of convergence modes; arrows point from stronger to weaker. Dashed boxes mark the conditions that reverse an arrow.

Worked Examples

Mean-Square Convergence of a Sample Mean

Given: Let

Y_{1}, Y_{2}, \dots

be i.i.d. with

E [Y_{i}] = μ = 3

and

Var (Y_{i}) = σ^{2} = 4

. Define the sample mean

X_{n} = \frac{1}{n} \sum_{i = 1}^{n} Y_{i}

. Does

X_{n} L^{2} μ

? Step 1 — Identify the error. Since

E [X_{n}] = μ

, the

L^{2}

error is the variance:

E [(X_{n} - μ)^{2}] = Var (X_{n}) .

⟳For an unbiased estimator the mean-square error equals the variance.

Step 2 — Compute the variance. By independence,

Var (X_{n}) = \frac{1}{n ^{2}} i = 1 \sum n Var (Y_{i}) = \frac{n σ ^{2}}{n ^{2}} = \frac{σ ^{2}}{n} = \frac{4}{n} .

⟳Variance of a sum of independent terms is the sum of variances; scaling by

1/ n

squares to

1/ n^{2}

Step 3 — Take the limit.

\frac{4}{n} \to 0

n \to \infty

E [(X_{n} - 3)^{2}] = \frac{4}{n} \to 0, so X_{n} L^{2} 3.

By the hierarchy [6] this also gives $X_{n} P 3$ — the Weak Law of Large Numbers. (Units: if $Y_{i}$ are in metres, the MSE is in metres².)

Convergence in Distribution to a Uniform Maximum

Given: Let

U_{1}, \dots, U_{n}

be i.i.d. Uniform

(0, 1)

and let

M_{n} = max_{i} U_{i}

. Define

Z_{n} = n (1 - M_{n})

. Find the limiting distribution. Step 1 — CDF of the maximum. For

0 \leq m \leq 1

P (M_{n} \leq m) = i = 1 \prod n P (U_{i} \leq m) = m^{n} .

⟳The max is

\leq m

iff all

n

independent variables are

\leq m

Step 2 — CDF of $Z_{n}$ . For

z \geq 0

, with

m = 1 - z / n

P (Z_{n} \leq z) = P (M_{n} \geq 1 - \frac{z}{n}) = 1 - (1 - \frac{z}{n})^{n} .

⟳

Z_{n} \leq z \Leftrightarrow n (1 - M_{n}) \leq z \Leftrightarrow M_{n} \geq 1 - z / n

; subtract the lower-tail probability from 1.

Step 3 — Take the limit. Using

(1 - z / n)^{n} \to e^{- z}

n \to \infty lim P (Z_{n} \leq z) = 1 - e^{- z}, z \geq 0.

⟳The standard exponential limit

(1 + a / n)^{n} \to e^{a}

with

a = - z

Z_{n} = n (1 - M_{n}) d Exponential (1) .

This is a classic extreme-value limit: the rescaled distance of the maximum from 1 becomes exponential.

Common Misconceptions

"Convergence in distribution means the variables get close." False. In-distribution convergence compares laws, not values. If

X \sim N (0, 1)

and

X_{n} = - X

, then

X_{n} d X

trivially (both are standard normal), yet

∣ X_{n} - X ∣ = 2∣ X ∣

never shrinks. Distribution convergence carries no information about the joint behaviour, and is the only mode for which a common probability space is unnecessary. "Almost sure and $L^{p}$ are the same." They are logically independent — neither implies the other (see the counterexample). The cause is that

L^{p}

penalises rare large errors that a.s. convergence happily tolerates. Slutsky reminder:

X_{n} d X

and

Y_{n} d Y

do not imply

X_{n} + Y_{n} d X + Y

unless one limit is a constant or the pair converges jointly.

In Probability but NOT Almost Surely — the "Typewriter" Sequence

On $Ω = [0, 1]$ with Lebesgue measure, define indicator variables on shrinking, sweeping intervals. For block size $k$ , let $X_{n}$ run through indicators of

[\frac{j}{k}, \frac{j + 1}{k}], j = 0, 1, \dots, k - 1, k = 1, 2, 3, \dots

⟳Enumerate intervals of width

1/ k

that sweep across

[0, 1]

, then shrink

k

; each

X_{n}

is the indicator of one such interval.

Then $P (X_{n} = 1) = 1/ k \to 0$ , so $X_{n} P 0$ . But for every fixed $ω \in [0, 1]$ , the sequence $X_{n} (ω)$ equals $1$ infinitely often (the sweeping interval revisits $ω$ in every block) and $0$ infinitely often. Hence $X_{n} (ω)$ diverges for every $ω$ — almost sure convergence fails completely.

Almost Surely but NOT in $L^1$ — Escaping Mass

On $Ω = [0, 1]$ , let $X_{n} = n \cdot 1_{[0, 1/ n]}$ . For any fixed $ω > 0$ , eventually $1/ n < ω$ , so $X_{n} (ω) = 0$ ; thus $X_{n} a.s. 0$ . However,

E [X_{n}] = n \cdot P ([0, 1/ n]) = n \cdot \frac{1}{n} = 1 \neq \to 0.

⟳A spike of height

n

on a set of measure

1/ n

carries constant area 1; pointwise it vanishes but its integral does not.

So $X_{n} L^{1} 0$ fails. The mass "escapes to infinity in height." Adding uniform integrability is exactly the condition that closes this gap, upgrading in-probability convergence to $L^{1}$ .

Connections

Convergence of random variables is the technical backbone of asymptotic probability. The Law of Large Numbers is a statement of in-probability (weak) and almost-sure (strong) convergence of sample means; the Central Limit Theorem is a statement of convergence in distribution to a Gaussian. The squeezing argument in the in-probability $\Rightarrow$ in-distribution proof relies on the structure of the Cumulative Distribution Function, while the Markov step rests on Markov's Inequality and its variance-bearing cousin Chebyshev's Inequality. The whole edifice presupposes the framework of Measure Theory and Probability Space, where random variables are measurable functions and expectation is the Lebesgue integral. Convergence in distribution is the probabilistic face of Weak Convergence of Measures, and the bridge back to almost-sure statements is the Skorokhod Representation Theorem.

Real-World Applications

Monte Carlo integration. Estimators $\frac{1}{n} \sum f (X_{i})$ converge a.s. and in $L^{2}$ to $E [f (X)]$ by the Strong Law; the $1/ n$ error rate comes from the CLT (convergence in distribution).
Maximum-likelihood estimation. Consistency of an estimator $\hat{θ}_{n}$ is the statement $\hat{θ}_{n} P θ_{0}$ ; asymptotic normality is $n (\hat{θ}_{n} - θ_{0}) d N (0, I^{- 1})$ .
Kalman filtering and signal processing. Mean-square ( $L^{2}$ ) convergence is the working notion for least-squares estimators and the stability of recursive filters.
Stochastic gradient descent (machine learning). Almost-sure convergence of iterates to a minimiser is established via the Robbins–Monro conditions on step sizes.
Extreme-value engineering (flood levels, material failure, finance tail risk). Convergence in distribution of maxima to Gumbel/Fréchet/Weibull laws (as in the Uniform-maximum example) drives risk design margins.
Empirical distribution functions. The Glivenko–Cantelli theorem gives uniform almost-sure convergence of $\hat{F}_{n}$ to $F$ , justifying the bootstrap and goodness-of-fit tests.
Queueing and insurance ruin theory. Heavy-traffic limits and ruin probabilities are convergence-in-distribution results for rescaled workload processes.

Historical Development

1713 — Jacob Bernoulli's Ars Conjectandi proves the first Law of Large Numbers: the relative frequency of an event converges (in what we now call probability) to its theoretical chance.
1837 — Poisson coins the phrase "law of large numbers" and extends Bernoulli's result.
1867 — Chebyshev gives his inequality, yielding a clean, general proof of the Weak Law and the first systematic use of moment bounds for convergence.
1900–1901 — Lyapunov proves the Central Limit Theorem under moment conditions, sharpening the meaning of convergence in distribution.
1909 — Émile Borel proves the Strong Law for Bernoulli trials, the first genuine almost-sure convergence theorem, introducing measure-zero reasoning.
1920s — Paul Lévy develops characteristic functions and the continuity theorem, the master tool for proving convergence in distribution; Khinchin and Kolmogorov refine LLN conditions.
1933 — Kolmogorov's Grundbegriffe axiomatises probability via measure theory, making all four convergence modes precise and proving the general Strong Law.
1956 — Skorokhod's representation theorem shows convergence in distribution can always be realised as almost-sure convergence on a suitable space, clarifying the relationship between the modes.
1968 — Billingsley's Convergence of Probability Measures systematises weak convergence on metric spaces, the modern functional viewpoint.

Subtleties & Limits

The clean undergraduate hierarchy hides genuine subtleties at the research frontier.

Rates and uniformity. Knowing $X_{n} d X$ says nothing about how fast. Quantifying the gap (Berry–Esseen bounds, Stein's method, Wasserstein distances) and obtaining sharp, dimension-free rates in high dimensions remains active.
Beyond the real line. For random variables in infinite-dimensional spaces (function-valued, measure-valued), the modes proliferate and tightness conditions become delicate; characterising weak compactness in such spaces is nontrivial.
Mode-mixing under operations. Determining minimal conditions under which a.s. or $L^{p}$ convergence is preserved through nonlinear functionals, conditioning, or stochastic integration is technically deep (e.g. for solutions of stochastic differential equations).
Almost-sure CLT. There exist refined "almost sure central limit theorems" where time-averages of normalized partial sums converge a.s.; their universality and rate are still being mapped out.

Convergence of Random Variables

The Core Insight

Formal Statement

Almost Sure Convergence

Convergence in Probability

Convergence in $L^p$ (Mean of Order $p$)

Convergence in Distribution

The Hierarchy of Convergence

Why It Holds / Derivation

Almost Sure ⇒ In Probability

In Probability ⇒ In Distribution

$L^p$ ⇒ In Probability (via Markov)

Andrey Kolmogorov | 1903–1987

Building Intuition

The strict hierarchy of convergence modes; arrows point from stronger to weaker. Dashed boxes mark the conditions that reverse an arrow.

Worked Examples

Mean-Square Convergence of a Sample Mean

Convergence in Distribution to a Uniform Maximum

Common Misconceptions

In Probability but NOT Almost Surely — the "Typewriter" Sequence

Almost Surely but NOT in $L^1$ — Escaping Mass

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also

Before this

Where this leads