Joint, Marginal and Conditional Distributions

Almost nothing in physics is measured one number at a time. A tracking detector reports a position and a momentum; a gravitational-wave interferometer reports a strain time-series whose samples are all correlated. To describe several random quantities together we need one object: the joint distribution over a vector of random variables. This single function is the complete probabilistic description — every question you can legitimately ask, individually or in combination, is already answered inside it and extracted by nothing more than integration. Two operations do almost all the work. Marginalisation integrates out the variables we do not care about, collapsing the joint to the law of what remains — "the distribution of $X$ alone?" Conditioning fixes some variables at observed values and renormalises the surviving slice — "given $Y=y$ , the law of $X$ ?" Conditioning is the continuous face of conditional probability, and Bayes' theorem is just the statement that $f_{X\mid Y}$ and $f_{Y\mid X}$ are two views of the same joint density. Dependence is summarised — usefully but incompletely — by the covariance matrix, its diagonal holding variances and its off-diagonal entries measuring how variables move together; rescaled by the standard deviations it becomes the correlation $\rho\in[-1,1]$ . Covariance sees only linear association, so it can vanish while the variables remain tightly, non-linearly bound; genuine independence is the stronger statement that the joint factorises. Only for jointly Gaussian variables do the two coincide — which is exactly why the multivariate normal governs so much of physics.

Key takeaways

The joint distribution of several random variables is the complete probabilistic description; for two continuous variables it is a density $f_{XY}(x,y)\ge 0$ that integrates to one over the plane.
Marginals come from integrating the unwanted variables out: $f_X(x)=\int f_{XY}(x,y)\,\mathrm{d}y$ . Conditionals come from fixing a value and renormalising: $f_{X\mid Y}(x\mid y)=f_{XY}(x,y)/f_Y(y)$ .
Independence is factorisation, $f_{XY}=f_X f_Y$ ; it is equivalent to "every conditional equals the corresponding marginal".
Covariance $\operatorname{Cov}(X,Y)=E[XY]-\mu_X\mu_Y$ and the dimensionless correlation $\rho=\operatorname{Cov}/(\sigma_X\sigma_Y)\in[-1,1]$ quantify only linear dependence.
Independence forces $\operatorname{Cov}=0$ , but zero covariance does not force independence — except for jointly Gaussian variables, where it does.
Dependence propagates: $\operatorname{Var}(aX+bY)=a^2\operatorname{Var}X+b^2\operatorname{Var}Y+2ab\operatorname{Cov}(X,Y)$ , the term the naive "add errors in quadrature" rule wrongly drops.
Key idea: the joint distribution is the master object — marginals come from integrating out variables, conditionals from fixing and renormalising, and covariance and correlation summarise the linear dependence, with independence equivalent to the joint factorising into its marginals.

The core insight

The joint distribution is the complete description; marginals come from integrating out variables, conditionals from fixing and renormalising, and covariance and correlation summarise the (linear) dependence — with independence equivalent to factorisation of the joint into its marginals.

Where this lives today

Joint, marginal, and conditional distributions are the daily machinery of quantitative physics. In precision cosmology the analysis of the Planck cosmic-microwave-background data delivers a high-dimensional joint posterior over cosmological parameters; the headline numbers — the matter density, the Hubble constant — are marginals got by Markov-chain Monte Carlo integrating out every nuisance parameter, and the "banana" contours in a $\Omega_m$ – $\sigma_8$ plane are two-dimensional marginals whose tilt is a correlation. In gravitational-wave astronomy, LIGO and Virgo model detector noise by a large covariance matrix and infer source masses and spins as marginals of a joint posterior. The Kalman filter that steers spacecraft is the bivariate-Gaussian conditioning of Derivation C applied recursively, each measurement updating a Gaussian prior exactly as $\mathcal{N}(\rho y,1-\rho^2)$ prescribes. Principal-component analysis diagonalises $\boldsymbol\Sigma$ to find the uncorrelated (for Gaussians, independent) modes carrying the variance. And every serious particle-physics uncertainty now ships a full covariance matrix, because the correlated-error arithmetic of Worked Example 3 is the difference between a correct and an overconfident result.

The physical picture

Picture the joint density of two variables as a hill of probability over the $(x,y)$ plane, its enclosed volume exactly one — normalisation. Marginalising over $y$ is like shining a lamp down the $y$ -direction and reading the shadow the hill casts on the $x$ -axis: you sum the probability in every $y$ -column, and what remains is the law $f_X(x)$ . Conditioning on $Y=y_0$ is the complementary move: cut the hill along $Y=y_0$ , take the exposed profile, and rescale it to area one again — because a raw slice does not integrate to one, and only a renormalised slice is a legitimate density.

Figure

Marginalise by projecting the joint density hill onto the x-axis (integrate out y); condition by slicing at fixed Y = y0 and renormalising that profile to unit area

Whether the hill tilts is what covariance measures: positive if large $x$ accompanies large $y$ , negative if large $x$ accompanies small $y$ . But a hill can be circular, or symmetric like a saddle or a ring, with no net tilt at all — covariance zero — while still richly structured, so knowing $y$ tells you a great deal about $x$ . That gap is why "uncorrelated" is weaker than "independent". Independence is the special case where the hill is an exact product of a profile in $x$ and a profile in $y$ : every conditional slice is then a rescaled copy of the single marginal.

Figure

Correlation ρ tilts the Gaussian 1σ ellipse: the conditional law of X given Y = y is Normal with mean ρy and variance 1 − ρ², here ρ = 0.8 and 1 − ρ² = 0.36; at ρ = 0 the contour is a circle and X and Y are independent

The correlation coefficient $\rho$ is covariance with the units divided out — a pure number in $[-1,1]$ reporting how tightly the samples hug a straight line, $\rho=\pm1$ exact linear dependence and $\rho=0$ no linear trend. For the Gaussian hills above, $\rho$ fixes the entire dependence structure: the conditional law of $X$ given $Y=y$ is $\mathcal{N}(\rho y,\,1-\rho^2)$ — mean sliding linearly with $y$ , variance shrunk from one to $1-\rho^2$ .

The core equations

The starting point is the joint cumulative distribution function and, when it is smooth, the joint density obtained by differentiating in both arguments — the probability mass per unit area, from which everything else is computed.

P(X\le x,\,Y\le y)=F_{XY}(x,y),\qquad f_{XY}(x,y)=\frac{\partial^2 F_{XY}}{\partial x\,\partial y} \qquad (1)

where: $F_{XY}$ — joint cumulative distribution function; $f_{XY}$ — joint probability density; $x,y$ — values of $X,Y$ . Reading it: the CDF accumulates probability over the quadrant below-and-left of $(x,y)$ ; the mixed derivative recovers the local density. A joint density is admissible exactly when it is non-negative and integrates to one — normalisation in two dimensions.

\iint_{\mathbb{R}^2} f_{XY}(x,y)\,\mathrm{d}x\,\mathrm{d}y = 1,\qquad f_{XY}\ge 0 \qquad (2)

where: $f_{XY}$ — joint density, non-negative and integrating to unit mass. Reading it: the probability hill has volume one, so any legitimate reduction must preserve that. Marginalisation integrates the unwanted variable out — the shadow the hill casts on each axis.

f_X(x)=\int_{-\infty}^{\infty} f_{XY}(x,y)\,\mathrm{d}y,\qquad f_Y(y)=\int_{-\infty}^{\infty} f_{XY}(x,y)\,\mathrm{d}x \qquad (3)

where: $f_X,f_Y$ — marginal densities of $X$ and $Y$ . Reading it: to forget a variable, integrate over it; derived in Derivation A. Conditioning fixes one variable and renormalises the exposed slice by the marginal, giving a proper density in the survivor.

f_{X\mid Y}(x\mid y)=\frac{f_{XY}(x,y)}{f_Y(y)},\qquad f_Y(y)>0 \qquad (4)

where: $f_{X\mid Y}$ — density of $X$ given $Y=y$ ; $f_Y(y)$ — marginal of $Y$ , the normaliser. Reading it: the raw slice $f_{XY}(x,y_0)$ has area $f_Y(y_0)$ , so dividing restores unit area. Bayes' theorem is the identity that both conditionals are the joint divided by a marginal, so one determines the other.

f_{X\mid Y}(x\mid y)=\frac{f_{Y\mid X}(y\mid x)\,f_X(x)}{f_Y(y)} \qquad (5)

where: $f_{Y\mid X}$ — reverse conditional, the likelihood; $f_X(x)$ — prior marginal on $X$ ; $f_Y(y)$ — the evidence. Reading it: the continuous engine of Bayesian inference. The covariance is the mean of the product of the centred variables — one number for how they co-vary about their means.

\operatorname{Cov}(X,Y)=E\!\left[(X-\mu_X)(Y-\mu_Y)\right]=E[XY]-\mu_X\mu_Y \qquad (6)

where: $\mu_X,\mu_Y$ — the means $E[X],E[Y]$ ; $E[XY]$ — mixed second moment. Reading it: positive when both exceed their means together, negative when one exceeds as the other falls short; it captures linear co-movement only. Dividing covariance by the two standard deviations strips the units, giving the correlation that Cauchy–Schwarz confines to $[-1,1]$ .

\rho_{XY}=\frac{\operatorname{Cov}(X,Y)}{\sigma_X\,\sigma_Y}\in[-1,1] \qquad (7)

where: $\sigma_X,\sigma_Y$ — standard deviations; $\rho_{XY}$ — dimensionless linear correlation. Reading it: $\rho=\pm1$ is exact linear dependence, $\rho=0$ no linear trend; see Derivation B. The variance of a linear combination carries a covariance cross-term — what correlated measurement errors add.

\operatorname{Var}(aX+bY)=a^2\operatorname{Var}X+b^2\operatorname{Var}Y+2ab\,\operatorname{Cov}(X,Y) \qquad (8)

where: $a,b$ — real constants. Reading it: only when $\operatorname{Cov}=0$ do variances add in quadrature; otherwise $ab\operatorname{Cov}$ shifts the result. Expectations of any function of the pair come from weighting with the joint density — LOTUS in two variables.

E[g(X,Y)]=\iint g(x,y)\,f_{XY}(x,y)\,\mathrm{d}x\,\mathrm{d}y \qquad (9)

where: $g$ — any measurable function of both variables. Reading it: integrate $g$ against the joint density directly, never needing the law of $g$ itself. For several variables the covariances organise into a symmetric, positive-semidefinite matrix with the variances on its diagonal.

\boldsymbol{\Sigma}=\begin{pmatrix}\sigma_X^2 & \operatorname{Cov}(X,Y)\\ \operatorname{Cov}(X,Y) & \sigma_Y^2\end{pmatrix} \qquad (10)

where: $\boldsymbol{\Sigma}$ — the covariance matrix, symmetric with $\boldsymbol{\Sigma}\succeq0$ . Reading it: for a Gaussian this matrix and the mean vector are the entire distribution — every marginal and conditional follows from it.

Derivation AFrom the joint density to its marginals and conditionals

The goal is to show that the two reduction operations are not definitions imposed from outside but forced consequences of what a joint distribution already is. We start from the joint cumulative distribution function, obtain the marginal by pushing the other variable to infinity and differentiating, and obtain the conditional as the unique renormalisation of a slice that integrates back to one — so marginalisation and conditioning are the only self-consistent answers to "the law of $X$ alone" and "the law of $X$ given $Y=y$ ".

Step 1. Write the marginal CDF of $X$ by allowing $Y$ any value: $F_X(x)=P(X\le x,\,Y<\infty)=F_{XY}(x,\infty)$ . Why: "the law of $X$ alone" places no restriction on $Y$ , which in CDF language sends its upper limit to infinity.

Step 2. Express it as an integral of the joint density and differentiate: $f_X(x)=\dfrac{\mathrm d}{\mathrm dx}\displaystyle\int_{-\infty}^{x}\!\!\int_{-\infty}^{\infty} f_{XY}(u,y)\,\mathrm dy\,\mathrm du$ . Why: the density is the derivative of the CDF, and the double integral exposes the inner $y$ -integral that survives.

Step 3. The outer derivative cancels the outer integral, leaving $f_X(x)=\displaystyle\int_{-\infty}^{\infty} f_{XY}(x,y)\,\mathrm dy$ , equation (3). Why: differentiating an accumulation in $x$ undoes it, so only the integration over the discarded $y$ remains — marginalisation is "integrate out what you drop".

Step 4. Now fix $Y=y$ . The raw slice $f_{XY}(x,y)$ integrates over $x$ to $f_Y(y)$ , not to one, so dividing by that constant gives $f_{X\mid Y}(x\mid y)=f_{XY}(x,y)/f_Y(y)$ , equation (4). Why: a conditional must be a genuine density in $x$ , and there is exactly one rescaling that restores unit area — division by the slice's own total mass $f_Y(y)$ .

Step 5. Rearranging equation (4) as $f_{XY}=f_{X\mid Y}(x\mid y)f_Y(y)=f_{Y\mid X}(y\mid x)f_X(x)$ and equating the two products gives Bayes' theorem, equation (5). Why: the joint factors two ways, and setting them equal is the identity that inverts a forward likelihood into a posterior.

Go deeper · Derivation B · 1 minWhy the correlation coefficient lives in minus one to plus one

Derivation BWhy the correlation coefficient lives in minus one to plus one

The goal is to prove the sharp bound $-1\le\rho_{XY}\le1$ and, in the same stroke, see why independence forces zero correlation while zero correlation does not force independence. The bound is Cauchy–Schwarz applied to the centred variables, got by demanding a certain expected square stay non-negative for every real parameter; the equality condition pins down the extremes, and a symmetric counterexample shows the converse fails — the backbone behind treating $\rho$ as a normalised measure of linear association.

Step 1. Centre the variables as $U=X-\mu_X$ , $V=Y-\mu_Y$ , and consider $q(t)=E[(tU+V)^2]=t^2E[U^2]+2t\,E[UV]+E[V^2]$ for real $t$ . Why: $(tU+V)^2\ge0$ pointwise, so $q(t)\ge0$ for every $t$ — one scalar inequality that encodes the whole bound.

Step 2. A quadratic $at^2+bt+c$ with $a\ge0$ that never goes negative has non-positive discriminant $b^2-4ac\le0$ ; with $a=E[U^2]$ , $b=2E[UV]$ , $c=E[V^2]$ this is $4(E[UV])^2-4E[U^2]E[V^2]\le0$ . Why: a positive discriminant would let the parabola dip below zero, contradicting $q(t)\ge0$ .

Step 3. Simplify to $(E[UV])^2\le E[U^2]E[V^2]$ , i.e. $\operatorname{Cov}^2\le\sigma_X^2\sigma_Y^2$ , so $\rho^2\le1$ and $-1\le\rho\le1$ . Why: this is Cauchy–Schwarz for the inner product $E[UV]$ ; dividing by the variances gives the correlation squared.

Step 4. Equality holds only when $q(t_0)=0$ , i.e. $t_0U+V=0$ almost surely — an exact affine relation $Y=aX+b$ , with the sign of $a$ setting the sign of $\rho=\pm1$ . Why: a non-negative quadratic touches zero only at a double root, and $E[(t_0U+V)^2]=0$ forces the bracket to vanish with probability one.

Step 5. Now suppose $X$ and $Y$ are independent, $f_{XY}=f_Xf_Y$ . By LOTUS $E[XY]=\big(\int x f_X\,\mathrm dx\big)\big(\int y f_Y\,\mathrm dy\big)=\mu_X\mu_Y$ , so $\operatorname{Cov}=0$ . Why: factorisation separates the double integral, so the mixed moment collapses to the product of means — independence always kills covariance.

Step 6. The converse fails: let $X$ be symmetric about zero and $Y=X^2$ ; then $E[X]=E[X^3]=0$ , so $\operatorname{Cov}(X,X^2)=E[X^3]-E[X]E[X^2]=0$ , yet $Y$ is a deterministic function of $X$ . Why: the parabola has no linear trend even though the dependence is total — uncorrelated is strictly weaker than independent.

Go deeper · Derivation C · 1 minConditioning a bivariate Gaussian yields another Gaussian

Derivation CConditioning a bivariate Gaussian yields another Gaussian

The goal is to prove the single most-used fact about correlated Gaussians: if $(X,Y)$ are jointly normal, the conditional law of $X$ given $Y=y$ is again normal, its mean moving linearly with $y$ and its variance reduced below the marginal by the correlation. We take the standardised bivariate normal with correlation $\rho$ , divide by the Gaussian marginal of $Y$ , and complete the square in $x$ ; out drops $\mathcal{N}(\rho y,\,1-\rho^2)$ — the result behind linear regression, Kalman filtering, and the reason uncorrelated upgrades to independent for Gaussians.

Step 1. Start from $f_{XY}(x,y)\propto\exp\!\big[-\tfrac{1}{2(1-\rho^2)}(x^2-2\rho xy+y^2)\big]$ and the marginal $f_Y(y)\propto e^{-y^2/2}$ , and form $f_{X\mid Y}(x\mid y)=f_{XY}/f_Y$ . Why: conditioning is equation (4), so we need the exponent surviving after the $y$ -marginal is divided out.

Step 2. Combine the exponents. The marginal contributes $+\tfrac12 y^2$ , which cancels the $y$ -only pieces of the joint, leaving in $x$ just $-\tfrac{1}{2(1-\rho^2)}(x^2-2\rho xy)$ plus terms independent of $x$ . Why: any factor without $x$ is a constant for the conditional in $x$ and is swept into the normalisation.

Step 3. Complete the square: $x^2-2\rho xy=(x-\rho y)^2-\rho^2 y^2$ , so $f_{X\mid Y}(x\mid y)\propto\exp\!\big[-\tfrac{1}{2(1-\rho^2)}(x-\rho y)^2\big]$ . Why: the exponent is now a perfect square in $x$ ; the leftover $-\rho^2y^2$ has no $x$ and is absorbed by the normaliser.

Step 4. Read off the parameters: $\mathcal{N}(\rho y,\,1-\rho^2)$ , so $E[X\mid Y=y]=\rho y$ and $\operatorname{Var}(X\mid Y=y)=1-\rho^2$ . Why: matching $\exp[-(x-\mu)^2/2\sigma^2]$ identifies $\mu=\rho y$ and $\sigma^2=1-\rho^2$ directly.

Step 5. Set $\rho=0$ : the conditional collapses to $\mathcal{N}(0,1)$ , exactly the marginal of $X$ — knowing $Y$ tells you nothing. Why: for Gaussians zero correlation makes the conditional equal the marginal, i.e. independence — the "uncorrelated becomes independent" gate that fails for general distributions.

Limiting cases

Special structures of the joint distribution collapse the general machinery into one-line statements — each row a place where a full joint law degenerates into something simpler.

Case	Result	Meaning
Independence	$f_{XY}(x,y)=f_X(x)\,f_Y(y)$	Every conditional equals its marginal; covariance and correlation vanish; the joint characteristic function factorises.
Perfect linear dependence	$\rho=\pm 1 \iff Y=aX+b\ \text{a.s.}$	The Cauchy–Schwarz bound is saturated; the probability hill collapses onto a straight line in the plane.
Jointly Gaussian, uncorrelated	$\rho=0 \Rightarrow X\perp Y$	For the multivariate normal only, zero covariance upgrades to full independence; the density factorises.
Discrete joint	$\sum_i\sum_j p_{ij}=1,\ p_{i\cdot}=\sum_j p_{ij}$	Integrals become sums; each marginal mass is a row or column sum of the joint table.
Uncorrelated, non-Gaussian	$\operatorname{Cov}=0$ but $f_{XY}\ne f_Xf_Y$	No linear trend, yet dependence survives in higher moments — e.g. $Y=X^2$ with symmetric $X$ .

Common trapZero covariance does not mean independence

The commonest error with joint distributions is to read $\operatorname{Cov}(X,Y)=0$ as " $X$ and $Y$ are independent". It is not. Independence is the strong, global statement that the joint factorises everywhere, $f_{XY}=f_X f_Y$ — equivalently, that every conditional slice is a rescaled copy of the marginal. Zero covariance is a single weak scalar condition: no linear trend, no net tilt of the hill. A symmetric ring or saddle, or the deterministic parabola $Y=X^2$ with $X$ symmetric about zero, has exactly zero covariance while $Y$ is completely determined by $X$ . The two notions coincide only for jointly Gaussian variables, where the entire dependence structure lives in the covariance matrix so that nulling the off-diagonal really does sever the link. Elsewhere, treat "uncorrelated" and "independent" as different words for different things, and never substitute one for the other in a proof.

Worked examples

1 · A discrete joint table, worked end to end (confidence builder)

Two binary detector flags $X,Y\in\{0,1\}$ fire with joint probabilities $P(0,0)=0.3$ , $P(0,1)=0.1$ , $P(1,0)=0.2$ , $P(1,1)=0.4$ . Find the marginals, a conditional, the covariance and correlation, and decide whether the flags are independent.

Step 1. Marginals are row and column sums. Over $y$ : $P(X=0)=0.4$ , $P(X=1)=0.6$ . Over $x$ : $P(Y=0)=0.5$ , $P(Y=1)=0.5$ . The cells sum to $1.0$ , so it is a valid joint law, and $\mu_X=E[X]=\mathbf{0.6}$ , $\mu_Y=E[Y]=\mathbf{0.5}$ .

Step 2. A renormalised conditional slice: $P(X=1\mid Y=1)=0.4/0.5=\mathbf{0.8}$ , whereas $P(X=1\mid Y=0)=0.2/0.5=\mathbf{0.4}$ . Because the law of $X$ changes when $Y$ flips, the flags are already visibly dependent.

Step 3. Covariance from the mixed moment: only the $(1,1)$ cell contributes, so $E[XY]=0.4$ and $\operatorname{Cov}(X,Y)=0.4-0.6\times0.5=\mathbf{0.1}$ — positive, so the flags tend to fire together.

Step 4. Correlation. For a $0/1$ variable $E[X^2]=E[X]$ , so $\operatorname{Var}X=0.6-0.36=0.24$ , $\sigma_X=0.4899$ ; $\operatorname{Var}Y=0.25$ , $\sigma_Y=0.5$ . Hence $\rho=0.1/(0.4899\times0.5)=\mathbf{0.408}$ , consistent with Step 2.

Step 5. Independence check: independence needs $P(1,1)=0.6\times0.5=0.30$ , but the table has $0.40$ . The joint does not factorise, so $X$ and $Y$ are dependent, agreeing with $\operatorname{Cov}\ne0$ . Marginals 0.4/0.6 and 0.5/0.5, $\operatorname{Cov}=0.1$ , $\rho=0.408$ , dependent.

2 · A continuous joint density on the unit square (fully worked)

Two quantities have joint density $f_{XY}(x,y)=x+y$ on the unit square $0\le x,y\le1$ , zero elsewhere. Find the marginal of $X$ , the conditional $f_{X\mid Y}$ , the covariance, and the correlation.

Step 1. Normalise, then marginalise: $\iint_{[0,1]^2}(x+y)=\int_0^1(\tfrac12+y)\,\mathrm dy=1$ , a legitimate density. Integrating out $y$ gives $f_X(x)=\int_0^1(x+y)\,\mathrm dy=x+\tfrac12$ , and by symmetry $f_Y(y)=y+\tfrac12$ . $f_X(x)=x+\tfrac12$ .

Step 2. Conditional by equation (4): $f_{X\mid Y}(x\mid y)=\dfrac{x+y}{\,y+\tfrac12\,}$ . It depends on $y$ — at $y=0$ it is $2x$ (rising), at $y=1$ nearly flat — so the conditional is not the marginal and the variables are dependent. $f_{X\mid Y}(x\mid y)=(x+y)/(y+\tfrac12)$ .

Step 3. Moments: $E[X]=\int_0^1 x(x+\tfrac12)\,\mathrm dx=\tfrac{7}{12}\approx0.583$ , $E[X^2]=\tfrac14+\tfrac16=\tfrac{5}{12}$ , so $\operatorname{Var}X=\tfrac{5}{12}-(\tfrac{7}{12})^2=\tfrac{11}{144}\approx0.0764=\operatorname{Var}Y$ .

Step 4. Covariance: $E[XY]=\iint xy(x+y)=\tfrac16+\tfrac16=\tfrac13$ , so $\operatorname{Cov}(X,Y)=\tfrac13-(\tfrac{7}{12})^2=\tfrac{48-49}{144}=\mathbf{-\tfrac{1}{144}}\approx-0.00694$ — slightly negative, since under this density a large $x$ makes a large $y$ marginally less likely.

Step 5. Correlation: with $\sigma_X=\sigma_Y=\sqrt{11/144}$ , $\rho=(-1/144)/(11/144)=\mathbf{-\tfrac{1}{11}}\approx-0.0909$ — the small tilt the density carries. $\operatorname{Cov}=-1/144$ , $\rho=-1/11\approx-0.091$ , dependent despite the tiny correlation.

3 · Correlated errors: why quadrature can lie

Two resistors in series, $R=R_1+R_2$ , are each measured with uncertainty $\sigma_1=\sigma_2=2\ \Omega$ . A shared calibration correlates the measurements at $\rho$ . Find the uncertainty in $R$ for $\rho=0,+0.5,-0.5$ .

Step 1. Apply equation (8) with $a=b=1$ : $\operatorname{Var}(R)=\sigma_1^2+\sigma_2^2+2\operatorname{Cov}$ , with $\operatorname{Cov}=\rho\sigma_1\sigma_2=4\rho$ , so $\operatorname{Var}(R)=8+8\rho$ .

Step 2. Uncorrelated ( $\rho=0$ ): $\operatorname{Var}(R)=8$ , $\sigma_R=\sqrt{8}=\mathbf{2.83\ \Omega}$ — the familiar "add in quadrature" answer.

Step 3. Positively correlated ( $\rho=+0.5$ ): $\operatorname{Var}(R)=12$ , $\sigma_R=\mathbf{3.46\ \Omega}$ — the shared calibration inflates the error by $22\%$ . Anti-correlated ( $\rho=-0.5$ ): $\operatorname{Var}(R)=4$ , $\sigma_R=\mathbf{2.00\ \Omega}$ — the errors partly cancel. Quadrature gives $2.83\ \Omega$ regardless; the truth is $3.46$ or $2.00\ \Omega$ depending on the sign of $\rho$ .

Physics of the portrait

The idea that dependence between two measured quantities is itself a number — bounded, dimensionless, computable — arrived late, out of nineteenth-century attempts to quantify heredity, then flowed back to unify the theory of errors Gauss and Laplace had begun.

Origins · 1 mincorrelation coefficient and how this was discovered

correlation coefficient

the dependence number ρ = Cov(X,Y) / σ_X σ_Y, bounded to minus one through plus one · multivariate probability and the theory of errors in the physical sciences

The correlation coefficient is younger than the objects it measures. Joint laws of errors were already in use when Gauss formulated the normal law of observational error in 1809 and Laplace, working on astronomical and geodetic errors, developed the multivariate normal and its characteristic functions — but neither wrote down a normalised measure of dependence. That step came from the study of heredity. Francis Galton, plotting parents' heights against their children's around 1885, noticed both "regression toward the mean" and a reproducible "co-relation", sketching the tilted elliptical scatter that is the bivariate normal seen from above. Karl Pearson made it exact: in 1896 he defined the product-moment coefficient $\rho=\operatorname{Cov}(X,Y)/\sigma_X\sigma_Y$ , proved it lay in $[-1,1]$ , and tied it to the bivariate normal whose contours are Galton's ellipses. The measure-theoretic footing that made a "joint distribution" a precise object — one probability measure on a product space, with marginals and conditionals derived by projection and disintegration — came with Kolmogorov's axioms in 1933. The physicist inherits it all at once: a covariance matrix for detector noise, a correlation reporting how two channels move together, and a conditional Gaussian $\mathcal{N}(\rho y,1-\rho^2)$ at the heart of least-squares fitting, the Kalman filter, and correlated-error propagation.

History

1809

— Gauss formulates the normal law of observational errors

1811

— Laplace develops the multivariate error distribution and characteristic functions

1885

— Galton finds regression and co-relation in parent–child height data

1896

— Pearson defines the product-moment correlation coefficient ρ

1933

— Kolmogorov's axioms put joint distributions on a measure-theoretic basis

Going further

Texts. Stone and Goldbart, Mathematics for Physics, probability chapter · Arfken and Weber, Mathematical Methods for Physicists, probability chapter (multivariate distributions and conditioning) · Hassani, Mathematical Physics, probability and statistics · Byron and Fuller, Mathematics of Classical and Quantum Physics, probability sections · Riley, Hobson and Bence, Mathematical Methods for Physics and Engineering, ch. 30 · Feller, An Introduction to Probability Theory and Its Applications, vol. 1. The worked-example results are: the discrete table gives $\operatorname{Cov}=0.1$ , $\rho=0.408$ ; the density $x+y$ gives $\operatorname{Cov}=-1/144$ , $\rho=-1/11$ ; the correlated sum gives $\sigma_R\in\{2.83,3.46,2.00\}\ \Omega$ for $\rho\in\{0,+0.5,-0.5\}$ .

Prerequisites. Random Variables and Probability Distributions · Conditional Probability and Bayes' Theorem · Expectation, Variance and Moments · Axioms of Probability. Leads to. Multivariate Gaussian Distribution · Central Limit Theorem · Moment-Generating and Characteristic Functions · Laws of Large Numbers and Convergence of Random Variables.

Ask the tutor: Ask me to build a joint distribution of your own — a table or a density — and I will walk you through its marginals, a conditional, and its correlation, or show you exactly why uncorrelated need not mean independent.

What exactly is the difference between a joint, a marginal, and a conditional distribution?

They are three views of one object. The joint $f_{XY}(x,y)$ is the master: it assigns probability to every combination of values and contains everything. A marginal such as $f_X(x)$ is what you get by forgetting the other variable — integrating it out, collapsing the hill onto one axis. A conditional such as $f_{X\mid Y}(x\mid y)$ is what you get by learning the other variable's value — cutting the hill along $Y=y$ and rescaling to unit area. Marginalising discards information; conditioning uses it. Both come from the joint, and neither can recreate the joint alone unless the variables are independent.

Why do I divide by the marginal when I condition, but not when I marginalise?

Because the two operations have opposite jobs. Marginalising sums probability — you add the joint over every value of the discarded variable, and since the joint already integrated to one, the result does too; no rescaling is needed. Conditioning restricts attention to one slice $Y=y$ , and a raw slice encloses area $f_Y(y)$ , generally less than one. Dividing by $f_Y(y)$ is the unique rescaling that restores unit probability. So the marginal appears as a divisor only in conditioning, precisely because only conditioning breaks normalisation.

Does zero correlation ever guarantee independence?

In exactly one situation: when the variables are jointly Gaussian. There the entire dependence structure sits in the covariance matrix, so zeroing the off-diagonal really does factorise the joint into marginals — uncorrelated becomes independent. Outside the Gaussian world this fails spectacularly: take $X$ symmetric about zero and $Y=X^2$ , and the covariance is zero for lack of a linear trend even though $Y$ is completely determined by $X$ . Covariance is blind to everything except linear co-movement, so a zero value rules out linear association only, never dependence in general — unless you already know the variables are Gaussian.

How does Bayes' theorem fall out of these definitions?

It is pure bookkeeping on the joint density. The joint factors two ways — $f_{XY}=f_{X\mid Y}(x\mid y)f_Y(y)$ and equally $f_{XY}=f_{Y\mid X}(y\mid x)f_X(x)$ — because either variable can be the one you condition on. Setting the two products equal and solving for $f_{X\mid Y}$ gives $f_{X\mid Y}(x\mid y)=f_{Y\mid X}(y\mid x)f_X(x)/f_Y(y)$ , Bayes' theorem. The pieces are named: $f_X$ prior, $f_{Y\mid X}$ likelihood, $f_Y$ evidence, $f_{X\mid Y}$ posterior. So Bayes' theorem is not an extra axiom but the statement that a joint density has two equivalent conditional factorisations.

Why does the covariance term matter so much in error propagation?

Because dropping it can make your quoted uncertainty flatly wrong, either way. The variance of a sum (or any function) of correlated inputs carries a cross-term $2ab\operatorname{Cov}(X,Y)$ , and the naive "add in quadrature" rule silently zeroes it — valid only for uncorrelated inputs. Real measurements are often correlated through a shared calibration or systematic. As Worked Example 3 shows, two $2\ \Omega$ uncertainties combine to $2.83\ \Omega$ if independent, but $3.46\ \Omega$ if $\rho=0.5$ or $2.00\ \Omega$ if anti-correlated — nearly two to one. This is why modern experiments publish full covariance matrices, not lists of independent error bars.

What makes the covariance matrix positive semidefinite, and why should I care?

It is positive semidefinite because it is built from expectations of squares: for any weights $\mathbf a$ , $\mathbf a^\top\boldsymbol\Sigma\,\mathbf a=\operatorname{Var}(\mathbf a^\top\mathbf X)\ge0$ , so $\boldsymbol\Sigma\succeq0$ automatically. You care for three reasons. Every eigenvalue is non-negative, so diagonalising $\boldsymbol\Sigma$ yields real variances along uncorrelated principal directions — the basis of principal-component analysis. It is a consistency check: a matrix with a negative eigenvalue cannot be a genuine covariance and signals an error. And for a Gaussian it must be invertible to write the density at all, its inverse (the precision matrix) appearing in the exponent.