Joint Distributions, Covariance, and Correlation

Joint distributions describe how two or more random variables behave together; covariance

Cov (X, Y)

measures the linear co-movement of

X

and

Y

; correlation

ρ_{X Y}

is its dimensionless, scale-free normalization confined to

[- 1, 1]

. Used everywhere: statistics, finance, signal processing, machine learning. Concepts crystallized by Galton and Pearson (1880s–1890s).

A joint distribution specifies the probabilities of all combinations of values that a collection of random variables can take simultaneously, encoding not just each variable's behavior but their interdependence. Covariance extracts the first-order (linear) part of that dependence — the average product of the two variables' deviations from their means — while correlation rescales covariance so it no longer depends on units and lies between $- 1$ and $+ 1$ . These tools matter because almost no real system has independent components: stock returns move together, heights and weights correlate, sensor noises share sources. The surprising fact every student must internalize: zero correlation does NOT imply independence — two variables can be perfectly deterministically related (e.g. $Y = X^{2}$ ) yet have zero covariance.

The Core Insight

The full joint distribution contains everything about how variables relate, but it is a complicated object (a surface or a table). Covariance and correlation compress that interdependence into a single number capturing only the linear trend. This is both their power (one interpretable scalar) and their danger (they are blind to nonlinear structure). Correlation answers exactly one question: "As $X$ rises above its mean, does $Y$ tend to rise or fall above its mean, and how strongly — measured in standard-deviation units?"

Formal Statement

Joint Distribution

For two random variables $X, Y$ :

Discrete case — the joint PMF is

p_{X Y} (x, y) = P (X = x, Y = y), x \sum y \sum p_{X Y} (x, y) = 1.

⟳probabilities of a complete, mutually exclusive set of outcomes sum to 1.

Continuous case — the joint PDF

f_{X Y} (x, y) \geq 0

satisfies

P ((X, Y) \in A) = \iint_{A} f_{X Y} (x, y) d x d y, \int_{- \infty}^{\infty} \int_{- \infty}^{\infty} f_{X Y} = 1.

⟳probability is the volume under the joint density surface over region

A

The marginal distributions recover each variable alone by summing/integrating out the other:

f_{X} (x) = \int_{- \infty}^{\infty} f_{X Y} (x, y) d y, p_{X} (x) = y \sum p_{X Y} (x, y) .

⟳collapse the joint over all values of the variable being eliminated (law of total probability).

The conditional density is

f_{Y ∣ X} (y ∣ x) = \frac{f _{X Y} ( x , y )}{f _{X} ( x )}, f_{X} (x) > 0.

⟳restrict to the slice

X = x

and renormalize so the slice integrates to 1.

Here $x$ are realized values, $y$ are realized values, $A$ a region in the plane.

Independence

X

and

Y

are independent iff the joint factors into marginals for all

x, y

f_{X Y} (x, y) = f_{X} (x) f_{Y} (y) ⟺ f_{Y ∣ X} (y ∣ x) = f_{Y} (y) .

⟳substitute the product form into the conditional definition [4]; the marginal cancels.

Covariance

The covariance of $X$ and $Y$ with means $μ_{X} = E [X], μ_{Y} = E [Y]$ is

Cov (X, Y) = E [(X - μ_{X}) (Y - μ_{Y})] .

⟳average the product of deviations from the mean — positive when they deviate together, negative when oppositely.

An equivalent, computation-friendly form:

Cov (X, Y) = E [X Y] - E [X] E [Y] .

⟳expand the product in [6], use linearity of expectation, and cancel

- μ_{X} μ_{Y}

terms.

The expectation of a product for the continuous case is

E [X Y] = \int_{- \infty}^{\infty} \int_{- \infty}^{\infty} x y f_{X Y} (x, y) d x d y .

⟳law of the unconscious statistician applied to the function

g (x, y) = x y

Correlation Coefficient

The (Pearson) correlation coefficient is the normalized covariance:

ρ_{X Y} = \frac{Cov ( X , Y )}{σ _{X} σ _{Y}}, σ_{X} = Var (X), σ_{Y} = Var (Y) .

⟳divide covariance by the product of standard deviations to cancel all units and bound the result.

Symbols: $σ_{X}, σ_{Y}$ are standard deviations, $Var (X) = Cov (X, X) = E [(X - μ_{X})^{2}]$ .

Properties of Covariance and Correlation

For random variables and constants $a, b, c, d$ :

Cov (a X + b, c Y + d) = a c Cov (X, Y) .

⟳additive shifts

b, d

leave deviations unchanged; scale factors pull out of expectation.

Var (X + Y) = Var (X) + Var (Y) + 2 Cov (X, Y) .

⟳expand

E [((X + Y) - (μ_{X} + μ_{Y}))^{2}]

and collect the cross term.

- 1 \leq ρ_{X Y} \leq 1.

⟳apply Cauchy–Schwarz to the centered variables (see proof below).

∣ ρ_{X Y} ∣ = 1 ⟺ Y = a X + b almost surely, with a \neq = 0.

⟳equality in Cauchy–Schwarz holds iff the centered variables are linearly dependent.

Why It Holds / Derivation

Correlation is bounded: $|\rho_{XY}|\le 1$

We prove [12] via the Cauchy–Schwarz inequality on centered variables.

Step 1. Define the centered variables

U = X - μ_{X}

V = Y - μ_{Y}

, so

E [U] = E [V] = 0

Cov (X, Y) = E [U V]

Var (X) = E [U^{2}]

Var (Y) = E [V^{2}]

. Step 2. For any real

t

, the quantity

E [(U - t V)^{2}] \geq 0

since it is the expectation of a square. Expand:

E [(U - t V)^{2}] = E [U^{2}] - 2 t E [U V] + t^{2} E [V^{2}] \geq 0.

⟳square the binomial inside, apply linearity of expectation.

Step 3. This is a quadratic in

t

that is always

\geq 0

, so its discriminant must be

\leq 0

(2 E [U V])^{2} - 4 E [V^{2}] E [U^{2}] \leq 0 ⟹ (E [U V])^{2} \leq E [U^{2}] E [V^{2}] .

⟳a non-negative parabola opening upward has at most one real root;

b^{2} - 4 a c \leq 0

Step 4. Substitute back:

(Cov (X, Y))^{2} \leq Var (X) Var (Y) ⟹ ρ_{X Y}^{2} \leq 1.

⟳divide [15] by

Var (X) Var (Y)

and take square roots.

Step 5 (equality case). Equality holds iff the discriminant is zero, i.e.

E [(U - t^{*} V)^{2}] = 0

for some

t^{*}

, which forces

U = t^{*} V

almost surely — a perfect linear relationship

X - μ_{X} = t^{*} (Y - μ_{Y})

, proving [13].

■

Independence ⟹ Zero Covariance

Step 1. If

X, Y

independent,

f_{X Y} = f_{X} f_{Y}

by [5]. Step 2. Then

E [X Y] = \iint x y f_{X} (x) f_{Y} (y) d x d y = (\int x f_{X} d x) (\int y f_{Y} d y) = E [X] E [Y] .

⟳the double integral separates into a product of single integrals because the integrand factors.

Step 3. By [7],

Cov (X, Y) = E [X Y] - E [X] E [Y] = 0

, hence

ρ_{X Y} = 0

■

Caution: the converse is false — see the counterexample below.

Karl Pearson | 1857–1936

Pearson, building directly on Francis Galton's work on heredity, gave the correlation coefficient its modern mathematical definition (the "product-moment correlation") in the 1890s. He founded the world's first university statistics department at University College London (1911) and the journal Biometrika. Pearson formalized many tools — the chi-squared test, the method of moments, the correlation coefficient $r$ named for him — though his reputation is shadowed by his promotion of eugenics. His insistence that statistical relationships could be quantified without invoking causation set the philosophical tone for modern applied statistics.

Building Intuition

Picture a cloud of data points on the $X$ – $Y$ plane. Draw the two mean lines $x = μ_{X}$ and $y = μ_{Y}$ , splitting the plane into four quadrants. Covariance is essentially the average signed area of the rectangles formed by each point's deviations $(X - μ_{X}) (Y - μ_{Y})$ :

A point in the upper-right (both above mean) or lower-left (both below) contributes a positive product.
A point in the upper-left or lower-right contributes a negative product.

If the cloud tilts up-right, positives dominate → positive covariance. If it tilts down-right, negatives dominate → negative. If it's a symmetric blob, contributions cancel → zero.

Correlation then asks: "After stretching/squashing each axis to standard-deviation units, how tight is the tilt?" $ρ = \pm 1$ means the points fall on a perfect line; $ρ = 0$ means no linear tilt at all (but possibly a curved relationship you can't see in $ρ$ ).

Sign of covariance by quadrant: products of mean-deviations

Worked Examples

Discrete joint PMF — full computation

Given. A joint PMF table over

X \in {0, 1}

Y \in {0, 1}

| | $Y = 0$ | $Y = 1$ | |---|---|---| | $X = 0$ | $0.1$ | $0.2$ | | $X = 1$ | $0.3$ | $0.4$ |

Step 1 — marginals.

P (X = 1) = 0.3 + 0.4 = 0.7

P (X = 0) = 0.3

P (Y = 1) = 0.2 + 0.4 = 0.6

P (Y = 0) = 0.4

. Step 2 — means.

E [X] = 1 \cdot 0.7 = 0.7

E [Y] = 1 \cdot 0.6 = 0.6

. Step 3 — $E [X Y]$ . Only the

(1, 1)

cell gives

x y \neq = 0

E [X Y] = 1 \cdot 1 \cdot 0.4 = 0.4

. Step 4 — covariance via [7]:

Cov = 0.4 - 0.7 \cdot 0.6 = 0.4 - 0.42 = - 0.02

. Step 5 — variances.

Var (X) = E [X^{2}] - E [X]^{2} = 0.7 - 0.49 = 0.21

(since

X^{2} = X

for 0/1). Likewise

Var (Y) = 0.6 - 0.36 = 0.24

. Step 6 — correlation via [9]:

ρ = \frac{- 0.02}{0.21 0.24} = \frac{- 0.02}{0.0504} = \frac{- 0.02}{0.2245}

Cov (X, Y) = - 0.02, ρ_{X Y} \approx - 0.089 (dimensionless)

A weak negative linear relationship.

Continuous joint density — covariance via integration

Given.

f_{X Y} (x, y) = x + y

on the unit square

0 \leq x \leq 1, 0 \leq y \leq 1

(zero elsewhere). Verify it's valid, then find

ρ

. Step 1 — validity.

\int_{0}^{1} \int_{0}^{1} (x + y) d x d y = \int_{0}^{1} (\frac{1}{2} + y) d y = \frac{1}{2} + \frac{1}{2} = 1

. ✓ Step 2 — marginal of $X$ .

f_{X} (x) = \int_{0}^{1} (x + y) d y = x + \frac{1}{2}

. Step 3 — $E [X]$ .

\int_{0}^{1} x (x + \frac{1}{2}) d x = \frac{1}{3} + \frac{1}{4} = \frac{7}{12}

. By symmetry

E [Y] = \frac{7}{12}

. Step 4 — $E [X^{2}]$ .

\int_{0}^{1} x^{2} (x + \frac{1}{2}) d x = \frac{1}{4} + \frac{1}{6} = \frac{5}{12}

. So

Var (X) = \frac{5}{12} - (\frac{7}{12})^{2} = \frac{60}{144} - \frac{49}{144} = \frac{11}{144}

. Same for

Y

. Step 5 — $E [X Y]$ via [8]:

\int_{0}^{1} \int_{0}^{1} x y (x + y) d x d y = \int_{0}^{1} \int_{0}^{1} (x^{2} y + x y^{2}) d x d y

. Inner over

x

\frac{1}{3} y + \frac{1}{2} y^{2}

. Outer over

y

\frac{1}{3} \cdot \frac{1}{2} + \frac{1}{2} \cdot \frac{1}{3} = \frac{1}{6} + \frac{1}{6} = \frac{1}{3}

. Step 6 — covariance [7]:

Cov = \frac{1}{3} - (\frac{7}{12})^{2} = \frac{48}{144} - \frac{49}{144} = - \frac{1}{144}

. Step 7 — correlation [9]:

ρ = \frac{- 1/144}{11/144} = - \frac{1}{11}

Cov (X, Y) = - \frac{1}{144} \approx - 0.00694, ρ_{X Y} = - \frac{1}{11} \approx - 0.0909

Common Misconceptions

Correlation ≠ causation.

ρ \neq = 0

tells you two variables move together; it says nothing about why. A lurking confounder, reverse causation, or pure coincidence (spurious correlation) can all produce strong

ρ

. Zero correlation ≠ independence. Correlation captures only the linear component of dependence. A perfectly predictable nonlinear relationship can have

ρ = 0

. Independence ⟹ zero covariance, but NOT the reverse. The implication is one-directional. Correlation is not robust. A single outlier can drag Pearson's

ρ

dramatically; it also assumes the relationship is linear and the second moments exist (correlation is undefined for, e.g., a Cauchy distribution with infinite variance).

Uncorrelated but dependent

Let $X$ be uniform on ${- 1, 0, 1}$ (each probability $\frac{1}{3}$ ) and define $Y = X^{2}$ . Then $Y$ is completely determined by $X$ — maximal dependence.

Compute: $E [X] = 0$ . $E [X Y] = E [X^{3}] = \frac{1}{3} (- 1) + \frac{1}{3} (0) + \frac{1}{3} (1) = 0$ . Therefore $Cov (X, Y) = E [X Y] - E [X] E [Y] = 0 - 0 = 0$ , so $ρ_{X Y} = 0$ .

Yet

X

and

Y

are not independent: e.g.

P (Y = 0 ∣ X = 0) = 1 \neq = P (Y = 0) = \frac{1}{3}

. The covariance is blind to the parabolic relationship because the symmetric positive and negative deviations cancel.

Connections

Joint distributions, covariance, and correlation sit at the hub of probability and statistics:

They generalize Variance and Standard Deviation — variance is $Cov (X, X)$ .
For many variables, covariances assemble into the covariance matrix, central to Principal Component Analysis and the Multivariate Normal Distribution.
They underpin Linear Regression, where the slope equals $Cov (X, Y) / Var (X)$ and $ρ^{2}$ is the coefficient of determination.
The conditional density [4] connects to Conditional Expectation and Bayes' Theorem.
Independence (eq. [5]) relates to Independence of Random Variables and the Law of Large Numbers.

Real-World Applications

Finance — portfolio theory. Markowitz's mean-variance optimization uses the covariance matrix of asset returns; diversification works precisely because low/negative correlations reduce overall portfolio variance (via eq. [11]).
Genetics & heredity. Galton's original motivation: correlation between parents' and offspring's heights ( $ρ \approx 0.5$ ), giving "regression toward the mean."
Signal processing. The autocorrelation function measures how a signal correlates with a time-shifted copy of itself, foundational to filtering and the Wiener–Khinchin theorem.
Machine learning. Feature correlation analysis detects redundant predictors; PCA decorrelates features via eigendecomposition of the covariance matrix.
Climate science. Correlations between sea-surface temperatures and rainfall (e.g. the El Niño–Southern Oscillation index) drive seasonal forecasting.
Medical research. Correlation between risk factors (cholesterol, blood pressure) and disease outcomes guides epidemiological hypotheses (always with the causation caveat).
Quality control / metrology. Covariance propagation quantifies how correlated measurement errors combine in derived quantities.

Historical Development

1846 — Auguste Bravais studies the bivariate normal and "correlated errors," anticipating the correlation formula in an astronomical context.
1877 — Francis Galton introduces "reversion" (later "regression") while studying sweet-pea seed sizes.
1888 — Galton publishes Co-relations and their Measurement, defining the correlation index $r$ geometrically from scatter diagrams.
1892 — Edgeworth develops the mathematics of the multivariate normal correlation.
1896 — Karl Pearson gives the rigorous "product-moment" definition of $r$ (eq. [9]), establishing the modern formula.
1904 — Charles Spearman introduces rank correlation, a nonparametric alternative.
1920 — Pearson's "Notes on the History of Correlation" surveys the field's origins.
1936 — Hotelling generalizes to canonical correlation between two sets of variables.
1950s — Markowitz embeds the covariance matrix at the core of modern portfolio theory, winning a 1990 Nobel Prize.

Subtleties & Limits

Beyond linear correlation. Pearson's

ρ

is fundamentally a linear and second-moment measure. Capturing general dependence rigorously remains active:

Existence. Correlation requires finite variances; heavy-tailed distributions (Cauchy, some financial returns) break it entirely.
Nonlinear dependence measures. Distance correlation (Székely, 2007) equals zero iff variables are independent — a genuine improvement over Pearson's. Mutual information and maximal information coefficient (MIC) attempt to capture arbitrary relationships, but each has estimation and interpretability trade-offs.
Copulas. Sklar's theorem separates marginals from the dependence structure; the "right" scalar summary of a copula is non-unique (Kendall's $τ$ , Spearman's $ρ$ , tail-dependence coefficients all coexist).
High dimensions. Estimating large covariance matrices from limited samples is ill-conditioned; shrinkage and sparse estimation (graphical lasso) are ongoing research areas.
Causal inference. Recovering causal structure from observational correlations (the do-calculus of Pearl) is a deep, still-evolving program.