Comparing Distributions

Comparing distributions means quantifying how similar or different two probability distributions are, using graphical tools (Q–Q plots, ECDF overlays), summary-statistic comparisons (mean, variance, quantiles), formal hypothesis tests (two-sample t, Kolmogorov–Smirnov, Mann–Whitney), and distance/divergence measures (total variation, KL divergence, Wasserstein). Key result: different tools answer different questions — "are these the same distribution?" vs. "are their means equal?" vs. "how far apart are they?" — and choosing the wrong one is the central practical error. Applies across statistics, machine learning, econometrics, and the experimental sciences. Foundational figures: Kolmogorov, Smirnov, Mann, Whitney, and Wasserstein (1969 metric).

Given two collections of data (or two theoretical models), comparing distributions asks whether they share the same underlying probability law, and if not, how and how much they differ. This matters because almost every inferential question — does a drug change outcomes, do two algorithms behave the same, is a sample drawn from a hypothesized model — reduces to comparing a sample distribution to another sample or to a reference. The discipline spans location (where the mass sits), scale (how spread out), shape (skew, tails), and whole-distribution comparisons. A surprising fact: two distributions can have identical means and variances yet be radically different (e.g., a normal vs. a bimodal mixture), so mean-based tests can completely miss a real effect — which is exactly why distribution-level tools exist.

The Core Insight

There is no single "distance" between distributions — there is a toolbox, and each tool privileges a different feature. Mean comparisons ask about the first moment; variance comparisons about the second; the Kolmogorov–Smirnov statistic about the largest vertical gap between cumulative distribution functions (CDFs); KL divergence about information loss; Wasserstein distance about the physical cost of moving probability mass. The art of comparing distributions is matching the question ("equal centers?" "same shape?" "indistinguishable laws?") to the right instrument, while respecting whether your data are paired or independent, continuous or discrete, and how heavy the tails are.

Formal Statement

Let $X$ and $Y$ be random variables with cumulative distribution functions $F (x) = P (X \leq x)$ and $G (y) = P (Y \leq y)$ . We have independent samples $x_{1}, \dots, x_{n}$ from $F$ and $y_{1}, \dots, y_{m}$ from $G$ .

Equality of Distributions

Two distributions are equal, written $F = G$ , if their CDFs agree everywhere:

F (t) = G (t) for all t \in R .

⟳the CDF uniquely determines a probability law on

R

, so equality of CDFs is equivalent to equality of distributions; this is the strongest sense in which two distributions can "be the same."

Empirical Cumulative Distribution Function (ECDF)

The ECDF of a sample estimates the true CDF by the fraction of observations at or below each point:

\hat{F}_{n} (t) = \frac{1}{n} i = 1 \sum n 1 {x_{i} \leq t},

where $1 {\cdot}$ is the indicator function (1 if true, 0 if false).

⟳

\hat{F}_{n} (t)

is an average of i.i.d. Bernoulli

(F (t))

indicators, so by the law of large numbers

\hat{F}_{n} (t) \to F (t)

; uniform convergence is the Glivenko–Cantelli theorem.

Kolmogorov–Smirnov Statistic

The two-sample KS statistic is the supremum (largest) vertical distance between the two ECDFs:

D_{n, m} = t \in R sup \hat{F}_{n} (t) - \hat{G}_{m} (t) .

⟳because both ECDFs are step functions that jump only at observed data points, the supremum is attained at one of the

n + m

sample values, making

D_{n, m}

computable by scanning the pooled sorted data.

Total Variation Distance

For distributions with densities $f$ and $g$ (or PMFs), the total variation distance is

TV (F, G) = A sup ∣ P_{F} (A) - P_{G} (A) ∣ = \frac{1}{2} \int ∣ f (x) - g (x) ∣ d x,

where the supremum runs over all measurable events $A$ .

⟳the optimal event is

A^{*} = {x : f (x) > g (x)}

; substituting and using

\int (f - g) = 0

over the whole space yields the

\frac{1}{2} \int ∣ f - g ∣

form.

Kullback–Leibler Divergence

The KL divergence from $G$ to $F$ measures expected information lost when $G$ is used to approximate $F$ :

D_{KL} (F ∥ G) = \int f (x) lo g \frac{f ( x )}{g ( x )} d x \geq 0.

⟳nonnegativity follows from Jensen's inequality applied to the convex function

- lo g

, since

E_{F} [- lo g (g / f)] \geq - lo g E_{F} [g / f] = - lo g 1 = 0

Wasserstein (Earth Mover's) Distance

The $p$ -Wasserstein distance is the minimal cost of transporting mass from $F$ to $G$ :

W_{p} (F, G) = (γ \in Γ (F, G) in f \int ∣ x - y ∣^{p} d γ (x, y))^{1/ p},

where $Γ (F, G)$ is the set of joint distributions (couplings) with marginals $F$ and $G$ .

⟳for

p = 1

R

, the optimal coupling matches quantiles, giving the closed form

W_{1} = \int_{0}^{1} ∣ F^{- 1} (u) - G^{- 1} (u) ∣ d u = \int_{- \infty}^{\infty} ∣ F (t) - G (t) ∣ d t

Glivenko–Cantelli

The ECDF converges uniformly to the true CDF almost surely:

t sup \hat{F}_{n} (t) - F (t) a.s. 0 as n \to \infty.

⟳pointwise a.s. convergence from the SLLN plus monotonicity of CDFs lets a finite grid control the whole real line; refining the grid drives the uniform error to zero.

Kolmogorov Limiting Distribution

Under $F = G$ (continuous), the scaled KS statistic has a distribution-free limit:

\frac{nm}{n + m} D_{n, m} d K, P (K \leq t) = 1 - 2 k = 1 \sum \infty (- 1)^{k - 1} e^{- 2 k^{2} t^{2}} .

⟳the empirical process

n (\hat{F}_{n} - F)

converges to a Brownian bridge whose supremum has the Kolmogorov distribution; the result is distribution-free because the time-change

t \mapsto F (t)

removes dependence on

F

Why It Holds / Derivation

The KS Test Is Distribution-Free Under the Null

Claim: Under

H_{0} : F = G

with

F

continuous, the null distribution of

D_{n, m}

does not depend on the underlying

F

Probability integral transform. If $X \sim F$ with $F$ continuous, then $U = F (X)$ is Uniform $(0, 1)$ . Apply $F$ to every observation in both samples.

ECDFs transform consistently. Since $F$ is monotone increasing, $x_{i} \leq t ⟺ F (x_{i}) \leq F (t)$ . Writing $u_{i} = F (x_{i})$ and $s = F (t)$ , the indicator $1 {x_{i} \leq t}$ equals $1 {u_{i} \leq s}$ .

The supremum is invariant. As $t$ ranges over $R$ , $s = F (t)$ ranges over $(0, 1)$ , so

D_{n, m} = t sup ∣ \hat{F}_{n} (t) - \hat{G}_{m} (t) ∣ = s \in (0, 1) sup ∣ \hat{F}_{n}^{U} (s) - \hat{G}_{m}^{U} (s) ∣,

where the right side uses ECDFs of the transformed uniform samples.

Conclusion. Under $H_{0}$ both transformed samples are i.i.d. Uniform $(0, 1)$ , irrespective of the original $F$ . Hence the distribution of $D_{n, m}$ depends only on $n$ and $m$ — not on $F$ . $■$

KL Divergence Is Nonnegative (Gibbs' Inequality)

Write $D_{KL} (F ∥ G) = E_{F} [- lo g \frac{g ( X )}{f ( X )}]$ where $X \sim f$ .
The function $ϕ (u) = - lo g u$ is convex on $(0, \infty)$ .
By Jensen's inequality, $E_{F} [ϕ (g / f)] \geq ϕ (E_{F} [g / f])$ .
Compute the inner expectation: $E_{F} [g (X) / f (X)] = \int f \cdot (g / f) d x = \int g d x = 1$ .
Therefore $D_{KL} \geq ϕ (1) = - lo g 1 = 0$ , with equality iff $g / f = 1$ a.e., i.e. $F = G$ . $■$

Two ECDFs and the Kolmogorov–Smirnov gap

Andrey Kolmogorov | 1903–1987

Kolmogorov axiomatized probability theory in 1933 and, the same year, introduced the goodness-of-fit statistic that compares an ECDF to a hypothesized CDF via their maximal gap. His insight that this gap's limiting distribution is governed by the Brownian bridge — and is free of the underlying distribution — made distribution comparison rigorous and universal. Nikolai Smirnov extended the idea to the two-sample case in 1939, giving the tool used to this day for asking "do these two samples come from the same law?"

Building Intuition

Picture each distribution as a pile of sand on a number line. Comparing means asks only where the balance point of each pile sits — you could shovel sand around wildly without moving it. Comparing variances asks how spread out each pile is. The KS statistic stacks the sand into a running cumulative wall and asks: what is the biggest height difference between the two walls anywhere? Wasserstein distance asks the most physical question of all: if I literally had to shovel one pile into the shape of the other, what is the minimum total "dirt × distance" work? Notice these can disagree dramatically: two piles with the same balance point can have a huge KS gap (different shapes), and two piles a KS test calls "barely different" can be far apart in Wasserstein if the difference lives in the tails. The choice of tool is a choice of what "different" means.

Worked Examples

Two-Sample t-Test for Equal Means

Given: A new fertilizer (group A) and control (group B). Yields (kg/plot): A:

n_{A} = 5

, mean

\overset{x}{ˉ}_{A} = 52

, sd

s_{A} = 4

. B:

n_{B} = 5

, mean

\overset{x}{ˉ}_{B} = 47

, sd

s_{B} = 3

. Goal: Test

H_{0} : μ_{A} = μ_{B}

α = 0.05

(two-sided), assuming equal variances. Step 1 — Pooled variance.

s_{p}^{2} = \frac{( n _{A} - 1 ) s _{A}^{2} + ( n _{B} - 1 ) s _{B}^{2}}{n _{A} + n _{B} - 2} = \frac{4 ( 16 ) + 4 ( 9 )}{8} = \frac{100}{8} = 12.5.

Step 2 — Standard error.

SE = s_{p} \frac{1}{n _{A}} + \frac{1}{n _{B}} = 12.5 0.4 = 3.536 \times 0.632 = 2.236.

Step 3 — t-statistic.

t = \frac{x ˉ _{A} - x ˉ _{B}}{SE} = \frac{52 - 47}{2.236} = 2.236.

Step 4 — Critical value. With

df = 8

t_{0.025, 8} = 2.306

. Step 5 — Decision. Since

∣ t ∣ = 2.236 < 2.306

, we fail to reject

H_{0}

t = 2.24, df = 8, p \approx 0.056 > 0.05 \Rightarrow no significant difference in mean yield (kg/plot)

Two-Sample KS Test for Whole-Distribution Difference

Given: Two small samples (sorted). A:

{1, 2, 4, 7}

(

n = 4

). B:

{3, 5, 6, 8}

(

m = 4

). Goal: Test

H_{0} : F_{A} = F_{B}

using

D_{n, m}

. Step 1 — Build ECDFs over the pooled sorted points

{1, 2, 3, 4, 5, 6, 7, 8}

. Each step is

1/4 = 0.25

| $t$ | $\hat{F}_{A}$ | $\hat{F}_{B}$ | $∣ \hat{F}_{A} - \hat{F}_{B} ∣$ | |----|------|------|------| | 1 | 0.25 | 0.00 | 0.25 | | 2 | 0.50 | 0.00 | 0.50 | | 3 | 0.50 | 0.25 | 0.25 | | 4 | 0.75 | 0.25 | 0.50 | | 5 | 0.75 | 0.50 | 0.25 | | 6 | 0.75 | 0.75 | 0.00 | | 7 | 1.00 | 0.75 | 0.25 | | 8 | 1.00 | 1.00 | 0.00 |

Step 2 — Maximum gap.

D_{4, 4} = 0.50

(attained at

t = 2

and

t = 4

). Step 3 — Critical value. For

n = m = 4

α = 0.05

, the exact critical value is

D_{crit} = 3/4 = 0.75

. Step 4 — Decision. Since

0.50 < 0.75

, we fail to reject

H_{0}

D_{4, 4} = 0.50 < 0.75 \Rightarrow samples consistent with the same distribution (small n has weak power)

Common Misconceptions

Equal means do NOT imply equal distributions. A two-sample t-test that "finds no difference" tests only the first moment. Two groups can have identical means and still differ wildly in variance, skew, or modality. Always ask which feature your test is sensitive to before declaring distributions "the same." A non-significant test is not evidence of equality. Failing to reject

H_{0} : F = G

means you lacked power to detect a difference, not that the distributions match. With small samples (as in the KS example above), the test can miss even large real differences. "Absence of evidence is not evidence of absence." KL divergence is not a distance. It is asymmetric,

D_{KL} (F ∥ G) \neq = D_{KL} (G ∥ F)

, fails the triangle inequality, and blows up to $+ \infty$ whenever

G

assigns zero probability where

F

does not. Do not use it as if it were a metric — use TV or Wasserstein for that. Identical mean and variance, totally different shape. Let

F = N (0, 1)

and let

G

be the two-point mixture placing mass

\frac{1}{2}

- 1

and

\frac{1}{2}

+ 1

. Both have mean

0

and variance

1

. A t-test and an F-test (variance test) both report "no difference." Yet

G

is discrete and bimodal while

F

is unimodal continuous — a KS or Wasserstein comparison detects a large gap. Moment-matching can perfectly disguise a structural difference.

Connections

Comparing distributions sits at the crossroads of several areas. The KS test rests on the Empirical Distribution Function and its convergence (Glivenko–Cantelli), while its limiting law is governed by the Brownian Bridge from stochastic processes. Mean comparisons descend from the Central Limit Theorem and Hypothesis Testing. Divergence measures like KL connect to Information Theory (entropy, cross-entropy) and to Maximum Likelihood Estimation, where minimizing KL to the empirical distribution is maximizing likelihood. Wasserstein distance links to Optimal Transport and underlies modern generative modeling. Rank-based comparisons (Mann–Whitney) belong to Nonparametric Statistics.

Real-World Applications

Clinical trials (FDA): Two-sample t-tests and Mann–Whitney tests compare treatment vs. placebo outcome distributions to license drugs.
A/B testing (tech industry): Companies like Netflix and Booking.com use distribution comparisons (KS, bootstrap) to decide whether a website variant changes user behavior, not just average clicks.
Machine learning — data drift detection: Production ML systems (e.g., at Uber, Amazon) run KS or Population Stability Index tests to flag when incoming feature distributions drift from training data.
Generative models (WGAN): Wasserstein distance is the training objective of Wasserstein GANs, comparing the generator's output distribution to the data distribution.
Quality control (manufacturing): Six Sigma programs compare a process's output distribution against specification limits and historical baselines.
Econometrics — inequality: Lorenz curves and stochastic-dominance tests compare income distributions across countries or policy regimes.
Particle physics (CERN): Goodness-of-fit comparisons between simulated and measured event distributions validate detector models and signal claims.

Historical Development

1908 — William Gosset ("Student") publishes the t-distribution, enabling small-sample mean comparison.
1933 — Andrey Kolmogorov introduces the one-sample goodness-of-fit statistic based on the maximal ECDF–CDF gap.
1933 — Glivenko and Cantelli prove uniform convergence of the ECDF to the true CDF.
1939 — Nikolai Smirnov extends Kolmogorov's idea to the two-sample setting, creating the KS two-sample test.
1947 — Henry Mann and Donald Whitney publish the rank-based U test for stochastic ordering of two distributions.
1948 — Claude Shannon's information theory provides the conceptual home for cross-entropy and (soon after) KL divergence.
1951 — Solomon Kullback and Richard Leibler formalize the directed divergence between distributions.
1969 — Leonid Vaserstein (Wasserstein) introduces the metric, rooted in Monge–Kantorovich optimal transport, that now bears his name.
2017 — Arjovsky, Chintala, and Bottou introduce Wasserstein GANs, propelling optimal-transport distribution comparison into deep learning.

Subtleties & Limits

Several deep issues remain practically and theoretically active. (1) High-dimensional comparison: the KS test has no canonical multivariate generalization, and estimating Wasserstein or KL distances in high dimensions suffers a curse of dimensionality (sample requirements grow exponentially). Entropic regularization (Sinkhorn distances) and kernel methods (Maximum Mean Discrepancy) are partial answers, but optimal sample complexity is still studied. (2) Calibrated equivalence testing: standard tests are built to detect differences, not to certify similarity; rigorous "two one-sided test" (TOST) equivalence frameworks exist but choosing the equivalence margin remains a modeling judgment. (3) Distribution-free guarantees under dependence: most theory assumes i.i.d. data, yet real time-series and networked data violate this, and the limiting distributions of comparison statistics under general dependence are incompletely characterized.