Probability Distributions

A probability distribution describes how the total probability (always summing to 1) is spread across all possible outcomes of a random variable. Key result in words: every random quantity has a rule — a table, a function, or a curve — that assigns probabilities to its values, and from that rule we compute means, spreads, and the likelihood of any event. Where it applies: statistics, physics, finance, biology, quality control, machine learning. Who/when: developed from Pascal & Fermat (1654) through Gauss, Poisson, and formalized by Kolmogorov (1933).

A probability distribution is a complete specification of the probabilities of all possible values of a random variable — a numerical outcome of a random experiment. Discrete distributions list probabilities for countable outcomes (like a die roll), while continuous distributions use a density curve where probabilities correspond to areas under the curve. Distributions matter because nearly every uncertain quantity in science and life — heights, errors, radioactive decays, stock returns — is modeled by one, letting us predict, summarize, and make decisions under uncertainty. A surprising fact: the sums of many independent random quantities, no matter their individual shapes, pile up into the same bell-shaped normal curve — a phenomenon called the Central Limit Theorem.

The Core Insight

A probability distribution is one whole unit of certainty (probability = 1) divided up among possibilities. For discrete outcomes you split it into chunks; for continuous outcomes you smear it as a density. Knowing how it is split lets you answer every probability question about that variable — and two simple summary numbers, the mean (center) and variance (spread), capture its essential character.

Formal Statement

Random Variable and Distribution

A random variable $X$ is a function assigning a real number to each outcome of a random experiment. A probability distribution specifies $P (X = x)$ (discrete) or a density $f (x)$ (continuous) for all values $x$ .

Discrete case — the probability mass function (PMF)

p (x) = P (X = x)

satisfies:

p (x) \geq 0 for all x, x \sum p (x) = 1

⟳probabilities cannot be negative, and exactly one of the mutually exclusive outcomes must occur, so they sum to certainty.

Continuous case — the probability density function (PDF)

f (x)

satisfies:

f (x) \geq 0, \int_{- \infty}^{\infty} f (x) d x = 1, P (a \leq X \leq b) = \int_{a}^{b} f (x) d x

⟳total area under the density equals total probability 1; probability of an interval is the area above it.

Here $x$ ranges over possible values, $p (x)$ is a probability (dimensionless), and $f (x)$ has units of probability per unit of x.

Cumulative Distribution Function

The cumulative distribution function (CDF) gives the probability of being at or below a value:

F (x) = P (X \leq x) = ⎩ ⎨ ⎧ t \leq x \sum p (t) \int_{- \infty}^{x} f (t) d t (discrete) (continuous)

⟳accumulate all probability from the smallest values up to

x

; for the continuous case differentiating gives back

F^{'} (x) = f (x)

F

is non-decreasing, with

F (- \infty) = 0

and

F (+ \infty) = 1

Mean and Variance

The expected value (mean, $μ$ ) is the probability-weighted average:

μ = E [X] = x \sum x p (x) or \int_{- \infty}^{\infty} x f (x) d x

⟳average each value by how likely it is — the "balance point" of the distribution.

The variance ( $σ^{2}$ ) measures spread about the mean:

σ^{2} = E [(X - μ)^{2}] = x \sum (x - μ)^{2} p (x) = E [X^{2}] - μ^{2}

⟳average squared distance from the mean; expand

(X - μ)^{2}

and use linearity of expectation to get the shortcut

E [X^{2}] - μ^{2}

The standard deviation is $σ = σ^{2}$ , in the same units as $X$ .

The Normal Distribution

The most important continuous distribution is the normal (Gaussian) distribution with mean $μ$ and standard deviation $σ$ :

f (x) = \frac{1}{σ 2 π} exp (- \frac{( x - μ ) ^{2}}{2 σ ^{2}})

⟳the

1/ (σ 2 π)

factor is the normalizing constant making the area equal 1, via the Gaussian integral

\int e^{- u^{2}} d u = π

About 68% of probability lies within $μ \pm σ$ , 95% within $μ \pm 2 σ$ , and 99.7% within $μ \pm 3 σ$ (the empirical rule).

Why It Holds / Derivation

:::proof[That Variance Equals $E [X^{2}] - μ^{2}$ ] Step 1. Start from the definition: $σ^{2} = E [(X - μ)^{2}]$ .

Step 2. Expand the square inside the expectation:

(X - μ)^{2} = X^{2} - 2 μ X + μ^{2} .

Step 3. Apply linearity of expectation (the expectation of a sum is the sum of expectations, and constants pull out):

E [(X - μ)^{2}] = E [X^{2}] - 2 μ E [X] + μ^{2} .

Step 4. Recognize

E [X] = μ

, so the middle term is

- 2 μ \cdot μ = - 2 μ^{2}

= E [X^{2}] - 2 μ^{2} + μ^{2} = E [X^{2}] - μ^{2} .

Step 5. This is the computational shortcut in equation [5]. It works because

μ

is a constant once the distribution is fixed.

■

Carl Friedrich Gauss | 1777–1855

Gauss derived the normal distribution while studying observational errors in astronomy. Seeking a curve for which the arithmetic mean is the most probable estimate of measurements, he arrived at equation [6] in 1809. The "bell curve" became so tied to his name it is called the Gaussian — though Abraham de Moivre had found it earlier (1733) as an approximation to coin-flip counts. Gauss's broader contributions to least-squares estimation made the normal distribution the cornerstone of statistics and physics.

Building Intuition

Think of probability as one liter of paint you must spread along the number line. A discrete distribution drops the paint in discrete blobs (a heavy blob over "3" means a 3 is likely). A continuous distribution smears it into a thin layer whose thickness is the density — thick where outcomes cluster, thin in the tails. The total paint is always exactly one liter. The mean is where the painted strip would balance on a knife edge; the variance tells you how widely the paint is smeared. For continuous variables, asking "what is the probability $X = 3.000 \dots$ ?" is like asking the area of a line with zero width — it's zero. You can only ask about intervals, which have area.

A normal probability density showing the mean and the 68–95–99.7 empirical rule

Worked Examples

A Fair Six-Sided Die (Discrete)

Given:

X

= result of rolling a fair die,

p (x) = \frac{1}{6}

for

x = 1, \dots, 6

. Find the mean. Using equation [4]:

μ = x = 1 \sum 6 x \cdot \frac{1}{6} = \frac{1}{6} (1 + 2 + 3 + 4 + 5 + 6) = \frac{21}{6} = 3.5.

Find the variance. First

E [X^{2}] = \frac{1}{6} (1 + 4 + 9 + 16 + 25 + 36) = \frac{91}{6} \approx 15.167

. Then by equation [5]:

σ^{2} = 15.167 - (3.5)^{2} = 15.167 - 12.25 = 2.9167.

Standard deviation:

σ = 2.9167 \approx 1.708

μ = 3.5 pips, σ^{2} \approx 2.92 pips^{2}, σ \approx 1.71 pips

Normal Heights (Continuous)

Given: Adult male heights are approximately normal with

μ = 175 cm

and

σ = 7 cm

. Question 1: What fraction are between 168 cm and 182 cm? This is

μ \pm σ

, so by the empirical rule:

P (168 \leq X \leq 182) \approx 68%.

Question 2: What fraction are taller than 189 cm? Note

189 = 175 + 2 σ

. The empirical rule gives 95% within

μ \pm 2 σ

, leaving 5% in the two tails, so 2.5% above:

P (X > 189) \approx 2.5%.

Question 3: Standardize 189 cm to a z-score:

z = \frac{x - μ}{σ} = \frac{189 - 175}{7} = 2.0.

68% within \pm σ; P (X > 189) \approx 2.5%; z = 2.0

Common Misconceptions

"For a continuous variable, $f (x)$ is a probability." It is not — it is a density. A density can exceed 1 (e.g. a narrow uniform distribution on

[0, 0.5]

has

f = 2

). Only areas under the curve are probabilities, and

P (X = c) = 0

for any single exact value in a continuous distribution. Always integrate over an interval. "The mean is the most likely value." False in general. For a skewed distribution the mean, median, and mode differ. The mean is the balance point, the mode is the peak (most likely), and the median splits probability 50/50. They coincide only for symmetric distributions like the normal.

A High Density Is Not High Probability

Consider a uniform distribution on the interval $[0, 0.1]$ . To make the area equal 1, the density must be $f (x) = 10$ for $0 \leq x \leq 0.1$ . So $f = 10 ≫ 1$ , yet no probability exceeds 1: the probability of landing in any sub-interval of width $w$ is just $10 w \leq 1$ . This demonstrates density and probability are genuinely different quantities.

Connections

Probability distributions sit at the heart of statistics and connect outward in many directions. The Binomial Distribution counts successes in fixed independent trials and approximates the Normal Distribution for large $n$ . The Poisson Distribution models rare events in time or space and arises as a limit of the binomial. The Central Limit Theorem explains why the normal appears so universally: sums of independent variables converge to it. The whole framework rests on Probability Axioms (Kolmogorov's rules) and on Expected Value for summarizing distributions. Sampling distributions of statistics underpin Hypothesis Testing and Confidence Intervals.

Real-World Applications

Quality control (Six Sigma): manufacturers model product dimensions as normal and reject items beyond $\pm 6 σ$ , targeting fewer than 3.4 defects per million.
Insurance & actuarial science: the Poisson distribution models claim counts and the exponential models waiting times, setting premiums.
Radioactive decay (physics): the number of decays in a fixed interval follows a Poisson distribution; decay times follow an exponential distribution.
Finance (risk modeling): stock returns are often modeled as log-normal or normal; Value-at-Risk uses distribution tails to bound losses.
Genetics (Mendelian inheritance): offspring genotype counts follow binomial and multinomial distributions.
Polling & elections: the sampling distribution of a poll proportion is approximately normal, giving margins of error.
Machine learning: the Gaussian underlies Naive Bayes classifiers, Gaussian mixture models, and the noise assumptions in linear regression.

Historical Development

1654 — Blaise Pascal and Pierre de Fermat correspond about gambling problems, founding the mathematical theory of probability.
1713 — Jacob Bernoulli's Ars Conjectandi introduces the binomial distribution and the Law of Large Numbers.
1733 — Abraham de Moivre derives the normal curve as an approximation to the binomial for large $n$ .
1809 — Carl Friedrich Gauss formalizes the normal distribution in the theory of measurement errors.
1812 — Pierre-Simon Laplace publishes Théorie analytique des probabilités, sharpening the Central Limit Theorem.
1837 — Siméon Denis Poisson introduces the Poisson distribution for rare events.
1894 — Karl Pearson coins the term "standard deviation" and develops systematic families of distributions.
1908 — William Gosset ("Student") introduces the t-distribution for small samples in brewing quality control.
1933 — Andrey Kolmogorov axiomatizes probability theory, giving distributions a rigorous measure-theoretic foundation.

Subtleties & Limits

Not every distribution is well-behaved. The Cauchy distribution has a perfectly valid bell shape yet no mean and no variance — the defining integrals diverge — so the Law of Large Numbers and the Central Limit Theorem fail for it. Such heavy-tailed distributions challenge naive intuition and appear in real data (financial crashes, earthquake magnitudes), where rare extreme events dominate. Modeling these requires care: the comforting empirical rule and even the existence of an average can break down. Deciding which distribution truly governs real data — and how to detect heavy tails from finite samples — remains an active practical and theoretical concern in statistics.