Expectation, Variance, and Moment Generating Functions

Expectation, Variance, and Moment Generating Functions are the three foundational tools for summarizing the location, spread, and full distributional shape of a random variable. The expectation

E [X]

is the probability-weighted average (the center of mass), the variance

Var (X)

measures squared dispersion about that center, and the moment generating function

M_{X} (t) = E [e^{tX}]

encodes ALL moments at once and uniquely determines the distribution when it exists. They apply across all of probability, statistics, physics, finance, and engineering. Core ideas trace to Huygens (1657), Chebyshev, and later formalization by Kolmogorov (1933).

For a random variable $X$ , the expectation $E [X]$ is the average value weighted by probability, the variance $Var (X) = E [(X - E [X])^{2}]$ quantifies how far values typically lie from that average, and the moment generating function (MGF) $M_{X} (t) = E [e^{tX}]$ packages the entire sequence of moments $E [X^{n}]$ into a single function whose derivatives at $t = 0$ recover them. These quantities matter because they reduce infinite-dimensional distributional information to interpretable numbers and because sums of independent variables behave simply under them — variances add and MGFs multiply. The surprising fact: two distributions with identical means and variances can look wildly different, yet if their MGFs agree on any interval around zero, the distributions are provably identical. The MGF thus acts as a fingerprint, turning convolution problems into algebra.

The Core Insight

A distribution can be probed at increasing depth: the expectation pins its center, the variance pins its scale, higher moments pin skew and tail weight, and the moment generating function assembles them into a master object. Operations that are hard on distributions — adding independent variables, scaling, shifting — become trivial on these summaries. This is the same philosophy as a Fourier transform: replace a complicated function (the density) by a transform (the MGF) where the hard operation (convolution) becomes a simple one (multiplication).

Formal Statement

Expectation

For a discrete random variable $X$ with probability mass function $p (x) = P (X = x)$ over support $S$ , the expectation is

E [X] = x \in S \sum x p (x),

⟳weight each attainable value by its probability and sum; this is the definition of a probability-weighted average (center of mass).

For a continuous random variable with probability density function $f (x)$ ,

E [X] = \int_{- \infty}^{\infty} x f (x) d x,

⟳replace the sum over discrete masses by an integral over the density;

f (x) d x

is the infinitesimal probability of the interval

[x, x + d x]

Here $x$ ranges over outcomes, $p (x)$ and $f (x)$ are nonnegative and integrate/sum to 1. The expectation exists when $\sum ∣ x ∣ p (x) < \infty$ or $\int ∣ x ∣ f (x) d x < \infty$ (absolute convergence).

Law of the Unconscious Statistician (LOTUS)

For any (measurable) function $g$ ,

E [g (X)] = x \sum g (x) p (x) or E [g (X)] = \int_{- \infty}^{\infty} g (x) f (x) d x .

⟳one need NOT find the distribution of

Y = g (X)

first; weight

g (x)

by the probability of

x

directly, justified by pushing forward the measure of

X

Variance and Standard Deviation

The variance is the expected squared deviation from the mean $μ = E [X]$ :

Var (X) = E [(X - μ)^{2}] = E [X^{2}] - (E [X])^{2} .

⟳expand

(X - μ)^{2} = X^{2} - 2 μ X + μ^{2}

, take expectation termwise using linearity, and substitute

E [X] = μ

The standard deviation $σ = Var (X)$ restores the original units. Variance has units of $X$ squared; standard deviation has the units of $X$ .

Moments and Central Moments

The $n$ -th raw moment is $μ_{n}^{'} = E [X^{n}]$ ; the $n$ -th central moment is $μ_{n} = E [(X - μ)^{n}]$ . Thus $μ_{1}^{'} = E [X]$ and $μ_{2} = Var (X)$ . The skewness and kurtosis are

γ_{1} = \frac{μ _{3}}{σ ^{3}}, γ_{2} = \frac{μ _{4}}{σ ^{4}} - 3.

⟳normalize the third and fourth central moments by powers of

σ

to obtain scale-free shape descriptors; subtracting 3 makes the normal distribution's excess kurtosis zero.

Moment Generating Function

The moment generating function of $X$ is

M_{X} (t) = E [e^{tX}] = ⎩ ⎨ ⎧ \sum_{x} e^{t x} p (x) \int_{- \infty}^{\infty} e^{t x} f (x) d x (discrete) (continuous)

⟳apply LOTUS with

g (x) = e^{t x}

; the MGF exists only for those

t

in an open interval about 0 where the sum/integral converges.

When $M_{X}$ exists in a neighborhood of $t = 0$ , moments are recovered by differentiation:

E [X^{n}] = M_{X}^{(n)} (0) = \frac{d ^{n}}{d t ^{n}} M_{X} (t)_{t = 0} .

⟳expand

e^{tX} = \sum_{n \geq 0} \frac{t ^{n}}{n !} X^{n}

, take expectation termwise (justified by absolute convergence near 0), so

M_{X} (t) = \sum_{n} \frac{t ^{n}}{n !} E [X^{n}]

; the coefficient of

t^{n} / n!

is the

n

-th moment.

Key Properties

For constants $a, b$ and independent variables $X, Y$ :

E [a X + b] = a E [X] + b, Var (a X + b) = a^{2} Var (X),

⟳linearity of expectation gives the first; for variance, the additive constant

b

shifts the mean but not the spread, and the factor

a

pulls out as

a^{2}

because variance is quadratic.

M_{a X + b} (t) = e^{b t} M_{X} (a t), M_{X + Y} (t) = M_{X} (t) M_{Y} (t) if X ⊥ Y .

⟳

E [e^{t (a X + b)}] = e^{b t} E [e^{(a t) X}]

; for the sum, independence factorizes

E [e^{tX} e^{t Y}] = E [e^{tX}] E [e^{t Y}]

Var (X + Y) = Var (X) + Var (Y) + 2 Cov (X, Y),

⟳expand

E [((X + Y) - μ_{X} - μ_{Y})^{2}]

and collect the cross term, which defines covariance; if independent, the covariance vanishes.

Why It Holds / Derivation

Variance Shortcut Formula

We prove $Var (X) = E [X^{2}] - (E [X])^{2}$ from equation [4].

Start from the definition $Var (X) = E [(X - μ)^{2}]$ with $μ = E [X]$ .
Expand the square: $(X - μ)^{2} = X^{2} - 2 μ X + μ^{2}$ .
Apply linearity of expectation: $E [(X - μ)^{2}] = E [X^{2}] - 2 μ E [X] + μ^{2}$ .
Substitute $E [X] = μ$ : the middle term becomes $- 2 μ \cdot μ = - 2 μ^{2}$ .
Combine: $E [X^{2}] - 2 μ^{2} + μ^{2} = E [X^{2}] - μ^{2} = E [X^{2}] - (E [X])^{2}$ . $■$

MGF Generates Moments

We justify equation [7].

Write the power series $e^{tX} = \sum_{n = 0}^{\infty} \frac{t ^{n} X ^{n}}{n !}$ , valid for all real arguments.
Provided $M_{X} (t)$ converges in an interval $(- h, h)$ , we may interchange expectation and summation (dominated convergence), giving $M_{X} (t) = \sum_{n = 0}^{\infty} \frac{t ^{n}}{n !} E [X^{n}]$ .
This is a Taylor series in $t$ ; by uniqueness of power-series coefficients, $E [X^{n}]$ is the coefficient of $t^{n} / n!$ .
Differentiating $n$ times and setting $t = 0$ kills all lower terms and the factorials of higher terms vanish at $t = 0$ , isolating $E [X^{n}] = M_{X}^{(n)} (0)$ . $■$

Mean (center of mass), $\pm\sigma$ spread, and the role of the MGF as a transform

Pafnuty Chebyshev | 1821–1894

The Russian mathematician Pafnuty Chebyshev placed expectation and variance at the heart of rigorous probability. His Chebyshev inequality, $P (∣ X - μ ∣ \geq k σ) \leq 1/ k^{2}$ , derived purely from the existence of a finite variance, gave the first distribution-free tail bound and provided the engine for his proof of the law of large numbers. He trained a school (Markov, Lyapunov) that turned moments into the principal tool of limit theory, paving the road to the central limit theorem via the method of moments.

Building Intuition

Think of a probability distribution as a thin metal rod whose mass is spread along the number line according to $f (x)$ . The expectation is the balance point — the center of mass where the rod would balance on a knife edge. The variance is the rod's moment of inertia about that balance point: it measures how the mass is distributed away from the center, with mass far out penalized quadratically. Standard deviation rescales this back to ordinary distance units, answering "how far, typically, are we from the center?"

The MGF is best understood as a generating gadget. Imagine you want to know all the moments at once. Rather than computing infinitely many integrals, you compute one function $M_{X} (t)$ , and then each moment falls out by differentiation, like extracting coefficients from a polynomial. The exponential $e^{tX}$ acts as a "moment vacuum cleaner": its Taylor expansion $1 + tX + \frac{t ^{2}}{2} X^{2} + \dots$ sucks in every power of $X$ simultaneously, and the parameter $t$ tags each moment with its own power so they can be separated later.

Statistical Mechanics Viewpoint

A physicist immediately recognizes $M_{X} (t) = E [e^{tX}]$ as a partition-function-like object. In statistical mechanics, $Z (β) = \sum_{i} e^{- β E_{i}}$ generates the energy moments: $⟨ E ⟩ = - \partial_{β} ln Z$ and $Var (E) = \partial_{β}^{2} ln Z$ . The same algebra appears here through the cumulant generating function $K_{X} (t) = ln M_{X} (t)$ , whose derivatives at zero yield cumulants: $K^{'} (0) = μ$ and $K^{''} (0) = σ^{2}$ . That energy fluctuations relate to heat capacity is precisely the statement that the second cumulant is the variance — thermodynamics and probability share one generating-function skeleton.

Worked Examples

Exponential Distribution: Mean, Variance, MGF

Given:

X \sim Exponential (λ)

with density

f (x) = λ e^{- λ x}

for

x \geq 0

λ = 2 s^{- 1}

(e.g., inter-arrival time of customers). Step 1 — MGF. For

t < λ

M_{X} (t) = \int_{0}^{\infty} e^{t x} λ e^{- λ x} d x = λ \int_{0}^{\infty} e^{- (λ - t) x} d x = \frac{λ}{λ - t} .

Step 2 — Mean via $M^{'} (0)$ .

M^{'} (t) = \frac{λ}{( λ - t ) ^{2}}

, so

E [X] = M^{'} (0) = \frac{1}{λ}

. Step 3 — Second moment via $M^{''} (0)$ .

M^{''} (t) = \frac{2 λ}{( λ - t ) ^{3}}

, so

E [X^{2}] = M^{''} (0) = \frac{2}{λ ^{2}}

. Step 4 — Variance.

Var (X) = \frac{2}{λ ^{2}} - \frac{1}{λ ^{2}} = \frac{1}{λ ^{2}}

. Numbers ( $λ = 2$ ):

E [X] = 0.5 s

Var (X) = 0.25 s^{2}

σ = 0.5 s

E [X] = \frac{1}{λ} = 0.5 s, Var (X) = \frac{1}{λ ^{2}} = 0.25 s^{2}

Sum of Two Independent Dice: Linearity and Variance Addition

Given: Two fair six-sided dice

X_{1}, X_{2}

, independent. For one die:

E [X_{i}] = \frac{1 + 2 + \dots + 6}{6} = 3.5

. Step 1 — Single-die variance.

E [X_{i}^{2}] = \frac{1 + 4 + 9 + 16 + 25 + 36}{6} = \frac{91}{6} \approx 15.167

. Then

Var (X_{i}) = 15.167 - 3. 5^{2} = 15.167 - 12.25 = 2.9167

. Step 2 — Sum's mean (linearity, no independence needed).

E [X_{1} + X_{2}] = 3.5 + 3.5 = 7

. Step 3 — Sum's variance (independence ⇒ covariance 0).

Var (X_{1} + X_{2}) = 2.9167 + 2.9167 = 5.8333

. Step 4 — Standard deviation.

σ = 5.8333 \approx 2.415

E [X_{1} + X_{2}] = 7, Var (X_{1} + X_{2}) = \frac{35}{6} \approx 5.833, σ \approx 2.42

Note that variances added but standard deviations did not ( $2.415 \neq = 1.708 + 1.708$ ).

Common Misconceptions

Variance does NOT add for the standard deviation, and only adds for variance when variables are uncorrelated. A frequent error is writing

σ_{X + Y} = σ_{X} + σ_{Y}

. Standard deviations add in quadrature:

σ_{X + Y} = σ_{X}^{2} + σ_{Y}^{2}

only when independent. A second trap:

E [g (X)] \neq = g (E [X])

in general — for nonlinear

g

, Jensen's inequality gives

E [g (X)] \geq g (E [X])

when

g

is convex (e.g.,

E [X^{2}] \geq (E [X])^{2}

, which is exactly why variance is nonnegative). Third: the MGF need not exist; if the defining integral diverges for all

t \neq = 0

, no moments can be extracted this way.

A Distribution with No MGF and Even No Mean

The Cauchy distribution with density $f (x) = \frac{1}{π ( 1 + x ^{2} )}$ has $\int_{- \infty}^{\infty} ∣ x ∣ f (x) d x = \infty$ , so $E [X]$ does not exist, the variance is undefined, and $M_{X} (t) = E [e^{tX}]$ diverges for every $t \neq = 0$ . Yet the Cauchy distribution is perfectly well-defined, symmetric, and bell-shaped — a vivid reminder that "looks like a bump" does not guarantee finite moments. Its characteristic function $φ_{X} (t) = E [e^{i tX}] = e^{- ∣ t ∣}$ exists, which is why characteristic functions (always defined) supersede MGFs in rigorous theory.

Connections

Expectation and variance are the first two members of an infinite hierarchy unified by the MGF, which is the real analogue of the Characteristic Function (use $i t$ instead of $t$ ). The logarithm of the MGF gives the Cumulant Generating Function, whose derivatives produce cumulants that add under independence. Variance addition under independence is the seed of the Law of Large Numbers and the Central Limit Theorem, both of which Chebyshev's variance bound and the method of moments help prove. Covariance generalizes variance to the joint setting and underlies Linear Regression and the Multivariate Normal Distribution. The MGF's multiplicative property mirrors the Convolution Theorem — sums of independent variables convolve densities and multiply transforms.

Real-World Applications

Insurance and actuarial science: expected claim size $E [X]$ sets premiums; variance and higher moments size the risk reserve and tail exposure.
Finance (portfolio theory): Markowitz mean–variance optimization picks asset weights to maximize $E [return]$ for a fixed $Var$ ; covariance matrices drive diversification.
Queueing and operations (telecom, call centers): exponential inter-arrival MGFs feed Erlang and M/M/1 models predicting wait times and buffer sizes.
Signal processing and communications: noise power equals variance; MGFs (Chernoff bounds) bound bit-error probabilities in fading channels.
Statistical physics: the partition function is a moment/cumulant generator; energy variance equals heat capacity times $k_{B} T^{2}$ .
Quality control (Six Sigma): process capability indices $C_{p k}$ compare specification limits to $\pm 3 σ$ , directly using mean and standard deviation.
Machine learning: variance–bias decomposition of expected error, and the reparameterization of Gaussians by their first two moments, underlie regression, PCA, and variational inference.

Historical Development

1654 — Pascal and Fermat correspond on the "problem of points," implicitly computing expected values of gambling outcomes.
1657 — Christiaan Huygens publishes De Ratiociniis in Ludo Aleae, the first printed treatise giving expectation ("expectatio") a formal definition.
1713 — Jacob Bernoulli's Ars Conjectandi proves the (weak) law of large numbers, tying long-run averages to expectation.
1733–1812 — De Moivre and Laplace develop the normal approximation, implicitly using mean and variance of binomial sums; Laplace introduces generating functions.
1867 — Chebyshev publishes his inequality, giving the first variance-based tail bound and a clean proof of the law of large numbers.
1900s — Markov and Lyapunov refine the method of moments, using moment convergence to prove central limit theorems.
1920s — R. A. Fisher systematizes moments, cumulants, and the term "variance" (coined 1918) in mathematical statistics.
1933 — Kolmogorov's Grundbegriffe axiomatizes probability via measure theory, defining expectation as a Lebesgue integral and securing all earlier results.
1937 — Cramér and others advance large-deviation theory using MGFs and Chernoff-type exponential bounds.

Subtleties & Limits

The MGF is powerful but fragile: it may fail to exist (heavy-tailed laws like Cauchy, log-normal, and Student- $t$ ), so moment-based identification breaks down. Even when all moments are finite, they need not determine the distribution — the moment problem (Stieltjes, Hamburger) shows the log-normal distribution shares its entire moment sequence with infinitely many distinct distributions, so equal moments do not imply equal laws. The characteristic function repairs existence (it is always defined and uniquely determines the distribution), but inverting it constructively, and characterizing precisely which moment sequences correspond to a unique distribution (the determinacy conditions of Carleman and Krein), remain technically delicate. Generalizing moments and concentration bounds to high-dimensional and dependent data is an active research frontier.

Expectation, Variance, and Moment Generating Functions

The Core Insight

Formal Statement

Expectation

Law of the Unconscious Statistician (LOTUS)

Variance and Standard Deviation

Moments and Central Moments

Moment Generating Function

Key Properties

Why It Holds / Derivation

Variance Shortcut Formula

MGF Generates Moments

Mean (center of mass), $\pm\sigma$ spread, and the role of the MGF as a transform

Pafnuty Chebyshev | 1821–1894

Building Intuition

Statistical Mechanics Viewpoint

Worked Examples

Exponential Distribution: Mean, Variance, MGF

Sum of Two Independent Dice: Linearity and Variance Addition

Common Misconceptions

A Distribution with No MGF and Even No Mean

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also

Before this

Where this leads