Expectation, Variance and Moments

Expectation and variance are the working currency of every precision measurement now being made. Gravitational-wave detectors like LIGO quote their sensitivity as a noise power spectral density — a variance per unit bandwidth — and their celebrated squeezed-light upgrade is nothing but a redistribution of quantum variance, pushing fluctuations below the shot-noise (Poisson) floor in the quadrature that matters, sensitivity the Fano-factor-one limit forbids for ordinary light. Optical atomic clocks quote the Allan variance, a purpose-built second moment that stays sensible when the ordinary variance of a drifting signal would diverge. In statistical mechanics the fluctuation–dissipation theorem makes the variance of energy literally the heat capacity, $\mathrm{Var}(E)=k_BT^2 C_V$, so a measured susceptibility is a measured second moment. And at the frontier of complex systems — turbulence, earthquakes, financial returns — the recurring surprise is the failure of the variance: heavy-tailed, Cauchy- or Lévy-like distributions where the second moment is infinite and the central-limit intuition collapses, forcing physicists back to the cautionary limiting case at the bottom of the table above.

The cleanest way to feel what a moment is is to make the distribution literally massive. Lay a weightless rod along the $x$-axis and glue a lump of mass at each outcome, the lump's weight equal to that outcome's probability. The total mass is one, because probabilities sum to one. Now every statistical descriptor becomes a mechanical one. The expectation is the point where the rod balances — the centre of mass. The variance is the moment of inertia of the same masses about that balance point. This is not an analogy that merely resembles the truth; it is the truth, because the defining formulas $\mu=\sum_i x_i p_i$ and $\sigma^2=\sum_i (x_i-\mu)^2 p_i$ are word-for-word the mechanics formulas for centre of mass and moment of inertia. The first figure builds exactly this rod for a concrete four-outcome distribution, so the mean and the variance can be read off as balance point and spread.

Where the mean tells you the balance point, the variance tells you the leverage — how far the mass sits from that point on average, squared. Two distributions can share a mean yet feel utterly different because their masses are packed tight or flung wide, and that difference is entirely the variance. The standard deviation $\sigma$ restores the units of $X$ and answers "how far from the mean is a typical outcome?" in metres, volts, or joules. The higher moments then refine the silhouette: the third moment reports whether the mass leans left or right of the balance point, the fourth whether the distribution hoards its probability in a sharp central peak with heavy tails or spreads it into broad shoulders. The second figure makes the variance tangible in the way physicists most often use it — for a Gaussian, $\sigma$ converts directly into a betting statement about where outcomes land.

Everything begins with a single definition applied to different functions of $X$. The expectation is the probability-weighted average of the outcomes — a sum for a discrete variable, an integral against the density for a continuous one.

where: $X$ — random variable; $x_i,\,p_i$ — outcome value and its probability mass; $f_X(x)$ — probability density function of $X$; $\mathbb{E}[\cdot]$ — expectation operator. Reading: weight each outcome by its probability and add — the centre of mass of the probability. The single most useful theorem about expectation says you may push any function inside without first finding its distribution. To get $\mathbb{E}[g(X)]$ you integrate $g(x)$ against $f_X$ directly — the "law of the unconscious statistician".

where: $g$ — any measurable function of the variable. Reading: every moment, generating function, and fluctuation is one choice of $g$ away from this line. Choosing $g(x)=x^n$ and $g(x)=(x-\mu)^n$ generates the two families of moments — raw moments about the origin and central moments about the mean.

where: $\mu_n'$ — $n$-th raw moment about the origin; $\mu_n$ — $n$-th central moment about the mean $\mu=\mu_1'$. Reading: $\mu_1'=\mu$ is location, $\mu_2$ is spread, $\mu_3$ is asymmetry, $\mu_4$ is tail weight. The variance is the second central moment, and expanding the square gives the computational shortcut every calculation actually uses.

where: $\mathrm{Var}(X)$ — variance, the second central moment $\mu_2$; $\sigma=\sqrt{\mathrm{Var}(X)}$ — standard deviation, same units as $X$. Reading: subtract the square of the mean from the mean of the square; the difference can never be negative. Expectation is linear without qualification — the property that carries the most weight in all of physics. Constants pull out, sums split, and no independence is required.

where: $a,b,c$ — arbitrary constants; $X,Y$ — random variables, possibly dependent. Reading: you may average term by term even when the variables are hopelessly entangled. Variance is quadratic, not linear, so a constant scale multiplies it by the square while a constant shift leaves it untouched; and when two variables are added, their covariance appears.

where: $a,b$ — constants; $\mathrm{Cov}(X,Y)=\mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y]$ — covariance, zero for uncorrelated variables. Reading: shifts do not change spread, scales square it, and correlations add or subtract a cross term. Normalising the third and fourth central moments by the appropriate power of $\sigma$ produces the dimensionless shape numbers — skewness and excess kurtosis.

where: $\gamma_1$ — skewness, positive for a right-leaning tail; $\gamma_2$ — excess kurtosis, tail-versus-peak weight relative to a Gaussian, the $-3$ set so the Gaussian scores zero. Reading: pure numbers that compare shapes across distributions of any scale. The moment-generating function packages the entire moment sequence into one object whose derivatives at the origin reproduce the moments.

where: $M_X(t)$ — moment-generating function; $t$ — auxiliary real parameter. Reading: differentiate $n$ times, set $t=0$, and read off the $n$-th moment — often the fastest route to it. Finally, the variance alone buys a universal guarantee about how much probability can hide far from the mean — Chebyshev's inequality, true for any distribution with a finite variance, no shape assumed.

where: $k>0$ — number of standard deviations; $\mu,\sigma$ — mean and standard deviation. Reading: at most a fraction $1/k^2$ of the probability can lie beyond $k$ standard deviations, however weird the distribution.

Feeding standard distributions through the moment machinery both fixes intuition and marks the exact place where the machinery breaks.

1 · A discrete distribution end to end (confidence builder)

Take the four-outcome distribution of the first figure: values $x=1,2,3,4$ with probabilities $0.1,0.2,0.4,0.3$. Find the mean, variance, standard deviation, and skewness.

Step 1. Mean from equation (1): $\mu=\sum x_i p_i = 1(0.1)+2(0.2)+3(0.4)+4(0.3)=0.1+0.4+1.2+1.2=\mathbf{2.9}$. This is the balance point of the rod.

Step 2. Second raw moment and variance: $\mathbb{E}[X^2]=1(0.1)+4(0.2)+9(0.4)+16(0.3)=9.3$, so by the shortcut (4), $\mathrm{Var}(X)=9.3-2.9^2=9.3-8.41=\mathbf{0.89}$ and $\sigma=\sqrt{0.89}=\mathbf{0.943}$. Direct check as a moment of inertia: $\sum p_i(x_i-\mu)^2 = 0.361+0.162+0.004+0.363=0.890$ — the two routes agree.

Step 3. Third central moment and skewness: $\mu_3=\sum p_i(x_i-\mu)^3 = 0.1(-1.9)^3+0.2(-0.9)^3+0.4(0.1)^3+0.3(1.1)^3 = -0.6859-0.1458+0.0004+0.3993=-0.432$. Then $\gamma_1=\mu_3/\sigma^3=-0.432/(0.943)^3=-0.432/0.8385=\mathbf{-0.52}$. The negative sign says the distribution leans left — a longer tail toward the small values — matching the eye: the heaviest lump sits at $x=3$, above the mean, with the thin tail reaching down to $x=1$. μ = 2.9, σ = 0.94, γ₁ = −0.52 (left-skewed).

2 · Photon counting and the square-root law

A photodetector collects light obeying Poisson statistics. In a fixed exposure it registers on average $\bar N = 10\,000$ photons. What is the noise, the signal-to-noise ratio, and the fractional uncertainty? How do these change if the exposure is lengthened until $\bar N = 1{,}000{,}000$?

Step 1. For Poisson statistics the variance equals the mean, so the noise is $\sigma_N=\sqrt{\bar N}=\sqrt{10\,000}=\mathbf{100}$ photons. The spread grows, but only as the square root of the count.

Step 2. Signal-to-noise ratio: $\mathrm{SNR}=\bar N/\sigma_N=\bar N/\sqrt{\bar N}=\sqrt{\bar N}=\sqrt{10\,000}=\mathbf{100}$, i.e. a fractional uncertainty of $\sigma_N/\bar N=1/\sqrt{\bar N}=\mathbf{1\%}$.

Step 3. At $\bar N=10^6$: $\sigma_N=\sqrt{10^6}=1000$, $\mathrm{SNR}=\sqrt{10^6}=\mathbf{1000}$, fractional uncertainty $=\mathbf{0.1\%}$. Collecting $100\times$ more light improved precision only $10\times$ — the universal square-root penalty of shot noise, and the reason astronomers integrate for hours. σ = √N‾ ; ten-thousand photons give 1% noise, a million give 0.1%.

3 · What Chebyshev guarantees without knowing the shape

A quantity has mean $\mu$ and standard deviation $\sigma$, but its distribution is completely unknown. Bound the probability that a single measurement lands more than two standard deviations from the mean. Compare with the Gaussian value.

Step 1. Apply Chebyshev's inequality (9) with $k=2$: $P(|X-\mu|\ge 2\sigma)\le 1/k^2 = 1/4 = \mathbf{25\%}$. This holds for every distribution with a finite variance — skewed, bimodal, discrete, whatever.

Step 2. For the special case of a Gaussian the true tail probability is far smaller: $P(|X-\mu|\ge 2\sigma)=1-0.954=\mathbf{4.6\%}$, straight from the 68–95–99.7 rule of the second figure. Chebyshev is loose here because it must survive the worst-case distribution, which piles its mass exactly at $\pm k\sigma$.

Step 3. The value of the bound is that it needs no shape assumption: from a mean and a variance alone it certifies that at most $1/k^2$ of outcomes can be extreme — the rigorous backbone of the weak law of large numbers. Chebyshev caps the two-sigma tail at 25%; a Gaussian actually delivers 4.6%.

The words in this article were forged over three centuries, from Huygens's first reckoning of a fair wager to Pearson's borrowing of "moment" from mechanics. But the figure who turned expectation and variance into rigorous, universal tools of analysis — rather than descriptive summaries — is Pafnuty Chebyshev.

Texts. Stone & Goldbart, Mathematics for Physics, chapter on probability and statistics · Arfken, Weber & Harris, Mathematical Methods for Physicists, 7th ed., Ch. 23 · Riley, Hobson & Bence, Mathematical Methods for Physics and Engineering, 3rd ed., Ch. 30 · Feller, An Introduction to Probability Theory and Its Applications, Vol. 1 · Byron & Fuller, Mathematics of Classical and Quantum Physics, Ch. on probability. Key numerical results collected for reference: variance shortcut $\mathbb{E}[X^2]-\mathbb{E}[X]^2$; discrete example $\mu=2.9$, $\sigma^2=0.89$, $\sigma=0.943$, $\gamma_1=-0.52$; Poisson $\mathbb{E}[X]=\mathrm{Var}(X)=\lambda$, confirmed via $M_X(t)=\exp[\lambda(e^t-1)]$; Gaussian variance integral $\langle u^2\rangle=1$ and fourth-moment result $\mu_4=3\sigma^4$ giving $\gamma_2=0$; Bernoulli mean $p$ and variance $p(1-p)$; uniform variance $(b-a)^2/12$; Chebyshev's bound $1/k^2$ at $k=2$ giving $25\%$ versus the Gaussian $4.6\%$; and the divergence of the Cauchy first moment.

Prerequisites. Random Variables and Probability Distributions · Axioms of Probability · Cumulative Distribution Functions and Change of Variables. Leads to. Moment-Generating and Characteristic Functions · Central Limit Theorem · Laws of Large Numbers and Convergence of Random Variables · Gaussian (Normal) Distribution.

Ask the tutor: Ask me to work any distribution — binomial, exponential, chi-squared — from its density all the way to its skewness and kurtosis, showing where each moment comes from.