Probability Theory

Probability theory provides a rigorous language for events whose outcomes are uncertain. Formally, it studies probability spaces $(\Omega ,\mathcal{F},P)$: a set of outcomes $\Omega$, a collection of events $\mathcal{F}$, and a measure $P$ assigning each event a number in $[0,1]$. It matters because nearly every quantitative science must reason under uncertainty, and probability is the only internally consistent calculus for doing so (cf. Cox's theorem). A surprising fact: deterministic-looking averages emerge from chaos — the Law of Large Numbers guarantees that long-run frequencies converge to probabilities, and the Central Limit Theorem shows that sums of many independent influences become Gaussian regardless of their individual shapes.

The Core Insight

Randomness can be tamed by treating probability as a normalized measure on a space of outcomes. Once we accept three simple axioms — non-negativity, normalization, and countable additivity — every intuitive rule of chance (complements, unions, conditioning, independence) follows by deduction, and powerful limit theorems become provable rather than merely observed.

Formal Statement

Probability Space

A probability space is a triple $(\Omega ,\mathcal{F},P)$ where:

• $\Omega$ is the sample space, the set of all possible outcomes $\omega$.
• $\mathcal{F}\subseteq 2^{\Omega }$ is a $\sigma$-algebra (the events): a collection closed under complement and countable union, containing $\Omega$.
• $P:\mathcal{F}\to [0,1]$ is a probability measure satisfying Kolmogorov's axioms.

The axioms are:

Conditional Probability and Independence

For events $A,B$ with $P(B)>0$, the conditional probability of $A$ given $B$ is

Events $A,B$ are independent iff

Random Variable and Expectation

A random variable $X$ is a measurable function $X:\Omega \to \mathbb{R}$ (so $\{\omega :X(\omega )\le x\}\in \mathcal{F}$ for all $x$). Its distribution function is $F_X(x)=P(X\le x)$. The expectation is

The variance is $\mathrm{Var}(X)=\mathbb{E}[(X-\mathbb{E}X{)}^2]=\mathbb{E}[X^2]-(\mathbb{E}X{)}^2$.

Bayes' Theorem

For events $A,B$ with $P(A),P(B)>0$,

Strong Law of Large Numbers

Let $X_1,X_2,\dots$ be i.i.d. with $\mathbb{E}|X_1|<\infty$ and mean $\mu$. Then

Central Limit Theorem

Let $X_1,\dots ,X_n$ be i.i.d. with mean $\mu$ and finite variance ${\sigma }^2>0$. Then

Why It Holds / Derivation

Bayes' Theorem from the Axioms

1. By the definition of conditional probability [4], $P(A\cap B)=P(A\mid B)P(B)$.
2. By symmetry of intersection, $A\cap B=B\cap A$, so also $P(A\cap B)=P(B\mid A)P(A)$.
3. Equate steps 1 and 2: $P(A\mid B)P(B)=P(B\mid A)P(A)$.
4. Divide both sides by $P(B)>0$ to obtain $P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B)}$.
5. For the denominator, suppose $\{A_i\}$ partitions $\Omega$ (disjoint, union $=\Omega$). Then $B={\bigcup }_i(B\cap A_i)$ is a disjoint union, so by additivity [3], $P(B)={\sum }_{i}P(B\cap A_i)={\sum }_{i}P(B\mid A_i)P(A_i)$. $■$

Weak Law of Large Numbers via Chebyshev

1. Assume finite variance ${\sigma }^2$ for simplicity. Let ${\bar{X}}_n=\frac{1}{n}{\sum }_{i=1}^n{X}_i$.
2. By linearity of expectation, $\mathbb{E}[{\bar{X}}_n]=\mu$.
3. By independence, variances add: $\mathrm{Var}({\bar{X}}_n)=\frac{1}{n^2}{\sum }_{i=1}^n\mathrm{Var}(X_i)=\frac{{\sigma }^2}{n}$.
4. Chebyshev's inequality states $P(|Y-\mathbb{E}Y|\ge \epsilon )\le \mathrm{Var}(Y)/{\epsilon }^2$. Apply with $Y={\bar{X}}_n$:

1. As $n\to \infty$ the right side $\to 0$ for every fixed $\epsilon >0$, so ${\bar{X}}_n\stackrel{P}{\to }\mu$ (convergence in probability). $■$

Andrey Kolmogorov | 1903–1987

The Soviet mathematician Andrey Nikolaevich Kolmogorov transformed probability from a collection of intuitive techniques into a rigorous branch of mathematics. In his 1933 monograph Grundbegriffe der Wahrscheinlichkeitsrechnung ("Foundations of the Theory of Probability"), he founded the subject on measure theory, defining probability spaces $(\Omega ,\mathcal{F},P)$ and the three axioms above. This grounded random variables as measurable functions and expectations as Lebesgue integrals, unifying discrete and continuous cases. Kolmogorov also made deep contributions to stochastic processes, turbulence, algorithmic complexity, and the strong law of large numbers.

Building Intuition

Think of a probability measure as distributing one unit of "mass" (or paint) over the sample space. Events are regions; $P(A)$ is the amount of paint that fell on region $A$. The axioms say: paint amounts are never negative [1], the total is exactly 1 [2], and the paint on disjoint regions adds up [3].

The two great limit theorems describe what happens when you average many independent splatters: the average position stabilizes (LLN) and the shape of the fluctuation becomes a bell curve (CLT), no matter what the individual splatter shapes were.

Standard normal density (limit shape from the CLT) with ±1σ region shaded

Worked Examples

Bayes with a Diagnostic Test

1. Total probability of a positive test [7]:

1. Compute: $0.0099+0.0495=0.0594$.
2. Apply Bayes [7]: $P(D\mid +)=\frac{0.99\times 0.01}{0.0594}=\frac{0.0099}{0.0594}$.
3. Evaluate: $\approx 0.1667$.

Expectation and Variance of a Fair Die

1. Expectation [6]: $\mathbb{E}[X]={\sum }_{k=1}^{6}k\cdot \frac{1}{6}=\frac{1+2+3+4+5+6}{6}=\frac{21}{6}=3.5$.
2. Second moment: $\mathbb{E}[X^2]={\sum }_{k=1}^6{k}^2\cdot \frac{1}{6}=\frac{1+4+9+16+25+36}{6}=\frac{91}{6}\approx 15.1667$.
3. Variance: $\mathrm{Var}(X)=\mathbb{E}[X^2]-(\mathbb{E}X{)}^2=\frac{91}{6}-{\left(\frac{7}{2}\right)}^2=15.1667-12.25=2.9167$.
4. Standard deviation: $\sigma =\sqrt{2.9167}\approx 1.708$.

Common Misconceptions

Independence ≠ Mutual Exclusivity

These are opposite concepts. Disjoint (mutually exclusive) events with positive probability are never independent: if $A\cap B=\varnothing$ then $P(A\cap B)=0\ne P(A)P(B)>0$, and learning $B$ occurred makes $A$ impossible. Independence means one event gives no information about the other; mutual exclusivity means one event gives maximal information.

The Gambler's Fallacy

The Law of Large Numbers does not say the universe "balances" short-run streaks. After ten heads in a row, a fair coin's next toss is still $P(H)=1/2$. LLN [8] guarantees the average converges, but it does so by diluting early deviations across an ever-larger $n$, not by compensating with opposite outcomes.

Pairwise Independence Does Not Imply Mutual Independence

Toss two fair coins. Let $A=\{$first is H$\}$, $B=\{$second is H$\}$, $C=\{$the two coins differ$\}$. Each has probability $1/2$, and every pair is independent: e.g. $P(A\cap C)=P(\text{first H, coins differ})=P(\text{HT})=1/4=P(A)P(C)$. But the three are not mutually independent: $P(A\cap B\cap C)=P(\text{first H, second H, they differ})=0$, whereas $P(A)P(B)P(C)=1/8$. Knowing $A$ and $B$ determines $C$ exactly.

Connections

Probability theory sits at a crossroads. It is built on Measure Theory — a probability measure is precisely a measure of total mass 1, and expectation is a Lebesgue integral. It feeds directly into Statistics, where probability models data and we invert the reasoning to estimate parameters. The study of probability over time gives Stochastic Processes (Markov chains, Brownian motion, martingales). Through Information Theory, entropy $H=-\sum p\log p$ quantifies the uncertainty of a distribution. Combinatorics underlies classical equally-likely-outcome probabilities, and the Central Limit Theorem connects probability to the ubiquitous Normal Distribution.

Real-World Applications

1. Medical diagnostics — Bayesian updating [7] converts test sensitivity/specificity and disease prevalence into the clinically relevant posterior probability a patient is ill.
2. Insurance & actuarial science — Expected loss [6] and the Law of Large Numbers [8] let insurers price premiums so that aggregate payouts are predictable across many policyholders.
3. Quantitative finance — The Black–Scholes model treats asset prices as geometric Brownian motion; option prices are expectations under a risk-neutral measure.
4. Machine learning — Naive Bayes classifiers, probabilistic graphical models, and Bayesian neural networks all rest on conditional probability and Bayes' rule.
5. Statistical physics — The Maxwell–Boltzmann distribution and entropy arise from assigning probabilities to microstates; the CLT explains Gaussian fluctuations of macroscopic quantities.
6. Telecommunications & information theory — Shannon's channel coding theorem bounds reliable transmission rates using probabilistic models of noise.
7. Polling & survey sampling — The CLT [9] yields the familiar margin of error $\approx 1/\sqrt{n}$ for estimating population proportions.

Historical Development

• 1654 — Blaise Pascal and Pierre de Fermat correspond over gambling problems (the "problem of points"), launching the mathematical theory of chance.
• 1657 — Christiaan Huygens publishes De Ratiociniis in Ludo Aleae, the first printed treatise on probability, introducing expectation.
• 1713 — Jacob Bernoulli's posthumous Ars Conjectandi states the first Law of Large Numbers (the "golden theorem").
• 1733/1738 — Abraham de Moivre derives the normal approximation to the binomial, an early Central Limit Theorem.
• 1763 — Thomas Bayes' essay (published posthumously by Richard Price) introduces inverse probability, later formalized as Bayes' theorem.
• 1812 — Pierre-Simon Laplace's Théorie analytique des probabilités systematizes the classical theory and generating functions.
• 1900s — Émile Borel, Henri Lebesgue, and others develop measure theory, providing the needed analytic foundation.
• 1933 — Andrey Kolmogorov publishes his axiomatic measure-theoretic foundations, the modern basis of the field.
• 1951–1968 — Itô calculus, martingale theory (Doob), and modern stochastic process theory mature, extending probability into continuous time.

Subtleties & Limits

See Also