Modes of Convergence of Random Variables

The four modes of convergence organize the entire toolkit of asymptotic statistics and probabilistic modeling. In statistics, the consistency of an estimator $\hat\theta_n$ is defined as $\hat\theta_n \xrightarrow{P} \theta$, while stronger "strong consistency" results demand almost sure convergence; the choice matters in proving properties of maximum likelihood estimators and in simulation-based (Monte Carlo) methods, where the Strong Law of Large Numbers justifies that simulated averages converge for (almost) every run of the simulator, not merely with high probability for each fixed sample size. Convergence in distribution underlies the Central Limit Theorem and all of its descendants (Berry–Esseen bounds, Edgeworth expansions), which drive confidence interval construction, hypothesis testing, and asymptotic normality claims throughout applied statistics. In stochastic processes and financial mathematics, $L^2$ convergence (mean-square convergence) is the natural mode for defining stochastic integrals (the Itô integral is built as an $L^2$ limit of simple-process approximations), while almost sure convergence underlies pathwise properties of stochastic processes such as the strong law for martingales. In information theory and statistical mechanics, almost sure convergence appears in ergodic theorems describing long-run time averages of dynamical systems. In numerical analysis and randomized algorithms, convergence in probability quantifies the reliability of randomized approximation schemes (e.g. Monte Carlo integration, randomized linear algebra), where one wants error bounds that hold with probability tending to one, without requiring every possible random seed to behave well. In cryptography, convergence-in-distribution style arguments (statistical indistinguishability of distributions) underlie definitions of security for pseudorandom generators, where two distributions must be shown to be asymptotically indistinguishable — a distributional, not pathwise, guarantee.

Ask the tutor: Can you show me why the Skorokhod representation theorem lets us "upgrade" convergence in distribution to almost sure convergence on some new probability space?

Picture each $X_n$ as a random point on the real line, and imagine watching the whole infinite sequence $X_1(\omega), X_2(\omega), \dots$ unfold for one fixed outcome $\omega$. Almost sure convergence says: for (Lebesgue-)almost every such $\omega$, this particular numerical sequence converges to $X(\omega)$. It is a statement about entire trajectories, and it is the strongest of the four notions—closest to the naive calculus meaning of "converges." Convergence in probability relaxes this drastically. It never asks the whole path to settle down; it asks only that, for each large $n$, the chance of being far from the limit is small. A sequence can converge in probability while, for every single outcome $\omega$, the path $X_n(\omega)$ never settles — it can have rare, occasional large excursions that keep recurring forever, provided those excursions become less and less probable. The classic image is a "moving spike": a bump of shrinking width but persistent height that sweeps across $[0,1]$ forever, hitting each point only rarely. Convergence in $L^p$ is an "average size of error" notion: it demands that the $p$-th absolute moment of the discrepancy vanish. This is often the easiest mode to verify by direct computation (e.g. via variance formulas), and Chebyshev's inequality is exactly the bridge that turns $L^2$ control into probability control. Convergence in distribution is the loosest of all: it is not even a statement that compares $X_n(\omega)$ to $X(\omega)$ pointwise, since $X_n$ and $X$ need not live on a common space or be at all correlated — it is a statement purely about the shapes of the histograms $F_n$ converging to the shape of $F$. This is precisely why the Central Limit Theorem, which describes the shape of the distribution of a standardized sum without any claim about pathwise behavior, is naturally a statement in this mode.

Example 1 (Verifying $L^2$, probability, and distribution convergence for a scaled, shifting spike). Let $U$ be a random variable uniform on $[0,1]$ (a genuine, non-symmetric base object), and for $n = 1, 2, \dots$ define

So $X_n$ equals $n$ with probability $1/n^3$ (when $U$ lands in the tiny interval $[0,1/n^3]$) and equals $0$ otherwise. Let $X \equiv 0$. Convergence in probability. For any $\varepsilon > 0$ and $n$ large enough that $n > \varepsilon$,

So $X_n \xrightarrow{P} 0$. Convergence in $L^2$. Compute directly:

So $X_n \xrightarrow{L^2} 0$ as well — consistent with Theorem part (ii), since $L^2$ convergence must imply convergence in probability, which we already confirmed directly. Almost sure convergence. Here we check it directly (not via the theorem, since a.s. does not follow from probability or $L^2$ convergence in general — see the counterexample below for a case where it fails). Fix $\omega$, i.e. fix the value $u = U(\omega) \in (0,1]$ (a single point, so this excludes only $U=0$, a probability-zero event). Since $u > 0$, choose $N$ with $1/N^3 < u$; then for all $n \geq N$, $u \notin [0,1/n^3]$ (as $1/n^3 \le 1/N^3 < u$), so $X_n(\omega) = 0$ for all $n\ge N$. Hence $X_n(\omega) \to 0$ for every $\omega$ with $U(\omega) > 0$, an event of probability $1$. So $X_n \xrightarrow{\text{a.s.}} 0$ too. Convergence in distribution. Since $X_n \xrightarrow{P} 0$, Theorem part (iii) already guarantees $X_n \xrightarrow{d} 0$; we can also see it directly: $F_n(x) = \mathbb P(X_n \le x)$ equals $1 - 1/n^3$ for $0 \le x < n$ and $1$ for $x \ge n$ and $0$ for $x<0$; at every fixed $x \neq 0$ this converges to $\mathbf 1_{\{x \geq 0\}} = F(x)$, the CDF of the constant $0$. This example shows all four modes agreeing; it is included to demonstrate the mechanics of checking each definition concretely with a genuinely asymmetric (not symmetric-around-zero) construction, before we exhibit the cases where the modes disagree.

Example 2 (A case where in-probability and $L^1$ convergence hold, but a.s. convergence fails: the "moving spike"). Let $U \sim \mathrm{Uniform}(0,1)$ again. We construct a sequence via a triangular indexing. For each $n \geq 1$, write $n = 2^k + j$ uniquely with $k \geq 0$ and $0 \leq j < 2^k$ (i.e. $k = \lfloor \log_2 n \rfloor$), and set

So as $n$ ranges over the block $2^k, 2^k+1, \dots, 2^{k+1}-1$, the interval $[j/2^k, (j+1)/2^k)$ sweeps once across $[0,1)$ in $2^k$ steps of width $2^{-k}$, and $X_n \in \{0,1\}$ marks whether $U$ currently lies in the moving window. Convergence in probability (to $X \equiv 0$). For $0 < \varepsilon < 1$,

where $k = \lfloor \log_2 n\rfloor \to \infty$ as $n \to \infty$. So $\mathbb P(|X_n|>\varepsilon) = 2^{-k} \to 0$, giving $X_n \xrightarrow{P} 0$. Convergence in $L^1$ (hence also matches part (ii) of the theorem). $\mathbb E|X_n| = \mathbb P(X_n=1) = 2^{-k} \to 0$, so $X_n \xrightarrow{L^1} 0$. Failure of almost sure convergence. Fix any $\omega$ with $u = U(\omega) \in (0,1)$ irrational (a set of probability $1$, since rationals have Lebesgue measure zero). We claim $X_n(\omega)$ does not converge to $0$ — in fact it does not converge at all. For every $k$, since the windows $[j/2^k,(j+1)/2^k)$ for $j=0,\dots,2^k-1$ partition $[0,1)$, exactly one value of $j=j(k)$ satisfies $u \in [j(k)/2^k, (j(k)+1)/2^k)$, and for the corresponding index $n = 2^k + j(k)$ we get $X_n(\omega) = 1$. Since this happens for every $k = 0,1,2,\dots$, there are infinitely many indices $n$ (one in every block) with $X_n(\omega) = 1$, while also (since each block has other indices with $X_n(\omega)=0$, as soon as $2^k \geq 2$) there are infinitely many indices with $X_n(\omega) = 0$. So $X_n(\omega)$ equals $1$ infinitely often and $0$ infinitely often: the sequence of real numbers $X_1(\omega), X_2(\omega), \dots$ has two subsequential limits, $0$ and $1$, hence does not converge. Since this failure occurs for every irrational $u \in (0,1)$, i.e. on an event of probability $1$, we conclude

so $X_n$ does not converge almost surely to $0$ (or to anything else, by the same argument). This is the standard demonstration that convergence in probability (even together with $L^1$ convergence) does not imply almost sure convergence — confirming that implication (i) in the theorem cannot be reversed.