Probability Foundations

Probability is the mathematical measure of how likely an event is, expressed as a number between 0 (impossible) and 1 (certain). The core result is Kolmogorov's axioms: all of probability theory follows from three simple rules about non-negativity, total certainty, and additivity of mutually exclusive events. It applies everywhere uncertainty appears — gambling, physics, statistics, machine learning, insurance, and medicine. The modern foundations were laid by Andrey Kolmogorov in 1933, building on three centuries of work since Pascal and Fermat (1654). Probability theory is the branch of mathematics that quantifies uncertainty. Formally, it assigns to each event

A

a number

P (A) \in [0, 1]

obeying consistency rules, where events are subsets of a sample space

Ω

containing all possible outcomes of a random experiment. It matters because nearly every empirical science, financial system, and decision under uncertainty relies on it to make rational predictions from incomplete information. A surprising fact: probability did not have rigorous mathematical foundations until 1933 — for nearly 300 years mathematicians computed probabilities successfully without a precise definition of what probability is. The resolution came from measure theory, the same machinery used to define area and integration.

The Core Insight

Probability is not a property of single events but a consistent system of weights spread across all possible outcomes, summing to 1. Once you fix this distribution of weight over the sample space, the probability of any event is simply the total weight it captures. Everything else — conditional probability, independence, expected value — is bookkeeping built on this single picture of "total certainty = 1, divided up among possibilities."

Formal Statement

Sample Space and Events

A random experiment is any process with an uncertain outcome. The sample space $Ω$ is the set of all possible outcomes. An event is a subset $A \subseteq Ω$ . An elementary outcome $ω \in Ω$ is a single point.

Tossing a coin: $Ω = {H, T}$ .
Rolling a die: $Ω = {1, 2, 3, 4, 5, 6}$ ; the event "even" is $A = {2, 4, 6}$ .
$Ω$ itself is the certain event; $\emptyset$ is the impossible event.

Probability Function — Kolmogorov's Axioms

A probability measure $P$ assigns to each event a real number satisfying three axioms:

P (A) \geq 0 for every event A

⟳This is an axiom (a chosen rule), not a theorem — probabilities cannot be negative because they measure a "size."

P (Ω) = 1

⟳An axiom asserting that something in the sample space is certain to happen; total weight is normalized to 1.

P (i = 1 ⋃ \infty A_{i}) = i = 1 \sum \infty P (A_{i}) when the A_{i} are pairwise disjoint

⟳Axiom of countable additivity — if events cannot overlap, the probability of "any one of them" is the sum of their separate probabilities.

Here $A_{i}$ are events with $A_{i} \cap A_{j} = \emptyset$ for $i \neq = j$ . Symbols: $P$ = probability function, $Ω$ = sample space, $A_{i}$ = events, $⋃$ = union ("or"), $\sum$ = sum.

Consequences of the Axioms

From the three axioms, these follow for all events $A, B$ :

P (\emptyset) = 0

⟳

Ω = Ω \cup \emptyset \cup \emptyset \cup \dots

with disjoint sets; apply axiom [3] and subtract.

P (A^{c}) = 1 - P (A)

⟳

A

and its complement

A^{c}

are disjoint and union to

Ω

; apply [2] and [3].

P (A \cup B) = P (A) + P (B) - P (A \cap B)

⟳Split

A \cup B

into disjoint pieces; the overlap

A \cap B

gets counted twice and must be subtracted once (inclusion–exclusion).

A \subseteq B ⟹ P (A) \leq P (B)

⟳Write

B = A \cup (B ∖ A)

as disjoint union; the extra term is

\geq 0

by [1].

Classical (Equally Likely) Probability

When $Ω$ is finite and every outcome is equally likely:

P (A) = \frac{∣ A ∣}{∣Ω∣} = \frac{number of favorable outcomes}{total number of outcomes}

⟳Each of

∣Ω∣

outcomes carries equal weight

1/∣Ω∣

by symmetry and axiom [2]; summing over

A

's outcomes via [3] gives the ratio.

Conditional Probability and Independence

The probability of $A$ given that $B$ has occurred:

P (A ∣ B) = \frac{P ( A \cap B )}{P ( B )}, P (B) > 0

⟳Restrict the sample space to

B

(which now plays the role of certainty), then re-normalize the weight of

A \cap B

by dividing by

P (B)

Events $A$ and $B$ are independent if knowing one tells nothing about the other:

P (A \cap B) = P (A) P (B)

⟳Substitute [10] into [9]:

P (A ∣ B) = P (A)

, i.e. conditioning on

B

leaves

A

's probability unchanged.

Bayes' Theorem

For events with positive probability:

P (A ∣ B) = \frac{P ( B ∣ A ) P ( A )}{P ( B )}

⟳Both

P (A ∣ B) P (B)

and

P (B ∣ A) P (A)

equal

P (A \cap B)

by [9]; equate and divide by

P (B)

With the law of total probability for a partition $A_{1}, \dots, A_{n}$ of $Ω$ :

P (B) = i = 1 \sum n P (B ∣ A_{i}) P (A_{i})

⟳

B = ⋃_{i} (B \cap A_{i})

is a disjoint union; apply [3], then [9] to each term.

Why It Holds / Derivation

Deriving the Complement Rule (Equation 5)

By definition, $A$ and its complement $A^{c}$ satisfy $A \cap A^{c} = \emptyset$ (they share no outcomes).
Their union is everything: $A \cup A^{c} = Ω$ .
Since they are disjoint, axiom [3] gives $P (A \cup A^{c}) = P (A) + P (A^{c})$ .
But $A \cup A^{c} = Ω$ , so by axiom [2], $P (A) + P (A^{c}) = P (Ω) = 1$ .
Rearranging: $P (A^{c}) = 1 - P (A)$ . $■$

Deriving Inclusion–Exclusion (Equation 6)

Decompose $A \cup B$ into three disjoint pieces: $(A ∖ B)$ , $(A \cap B)$ , and $(B ∖ A)$ .
By axiom [3]: $P (A \cup B) = P (A ∖ B) + P (A \cap B) + P (B ∖ A)$ .
Note $P (A) = P (A ∖ B) + P (A \cap B)$ and $P (B) = P (B ∖ A) + P (A \cap B)$ , since each event splits into its overlapping and non-overlapping parts.
Substitute $P (A ∖ B) = P (A) - P (A \cap B)$ and $P (B ∖ A) = P (B) - P (A \cap B)$ into step 2.
This gives $P (A \cup B) = [P (A) - P (A \cap B)] + P (A \cap B) + [P (B) - P (A \cap B)] = P (A) + P (B) - P (A \cap B)$ . $■$

Andrey Kolmogorov | 1903–1987

The Russian mathematician Andrey Nikolaevich Kolmogorov gave probability its rigorous modern foundation in his 1933 monograph Grundbegriffe der Wahrscheinlichkeitsrechnung ("Foundations of the Theory of Probability"). His insight was to identify probability with a measure — the same abstract notion of "size" used to define length, area, and volume — defined on a $σ$ -algebra of events. This dissolved centuries of paradoxes and made probability a respectable branch of pure mathematics. Kolmogorov also made landmark contributions to turbulence, topology, complexity theory (Kolmogorov complexity), and dynamical systems, making him one of the most versatile mathematicians of the 20th century.

Building Intuition

Picture the sample space $Ω$ as a region of total area exactly $1$ — a unit of "certainty." Each event is a sub-region, and its probability is its area. The axioms become visually obvious:

Areas can't be negative (axiom [1]).
The whole region has area 1 (axiom [2]).
Non-overlapping regions have areas that add (axiom [3]).

Conditional probability $P (A ∣ B)$ means: zoom in so that region $B$ now fills your entire view (becomes the new "100%"), and ask what fraction of that view is occupied by $A$ . Independence means $A$ takes up the same fraction of $B$ as it does of the whole space — knowing you're inside $B$ tells you nothing new about $A$ .

The sample space as area 1, with overlapping events A and B

Worked Examples

Two Dice — Sum and Conditioning

Given: Roll two fair six-sided dice. The sample space has

∣Ω∣ = 6 \times 6 = 36

equally likely outcomes. Question 1: What is

P (sum = 7)

? Steps: The favorable pairs are

(1, 6), (2, 5), (3, 4), (4, 3), (5, 2), (6, 1)

— that's

6

outcomes. By equation [8]:

P (sum = 7) = \frac{6}{36} = \frac{1}{6} .

Question 2: Given that the first die shows a

3

, what is

P (sum = 7 ∣ first = 3)

? Steps: Use equation [9]. Event

B = {first = 3}

has

6

outcomes, so

P (B) = 6/36

. The intersection (first = 3 and sum = 7) is just

(3, 4)

, so

P (A \cap B) = 1/36

. Then:

P (A ∣ B) = \frac{1/36}{6/36} = \frac{1}{6} .

The answers are equal — consistent with independence of the two dice.

Boxed answers:

P (sum = 7) = \frac{1}{6} \approx 0.167, P (sum = 7 ∣ first = 3) = \frac{1}{6} \approx 0.167

Medical Test — Bayes' Theorem in Action

Given: A disease affects

1%

of a population. A test is

99%

accurate both ways:

P (positive ∣ disease) = 0.99

and

P (negative ∣ no disease) = 0.99

(so the false-positive rate is

0.01

). A person tests positive. What is the probability they actually have the disease? Steps: Let

D

= "has disease,"

T

= "tests positive."

Prior: $P (D) = 0.01$ , $P (D^{c}) = 0.99$ .
$P (T ∣ D) = 0.99$ , $P (T ∣ D^{c}) = 0.01$ .

First, total probability of a positive test (equation [12]):

P (T) = (0.99) (0.01) + (0.01) (0.99) = 0.0099 + 0.0099 = 0.0198.

Now apply Bayes' theorem (equation [11]):

P (D ∣ T) = \frac{P ( T ∣ D ) P ( D )}{P ( T )} = \frac{( 0.99 ) ( 0.01 )}{0.0198} = \frac{0.0099}{0.0198} = 0.5.

Boxed answer:

P (disease ∣ positive test) = 0.50

Despite a "99% accurate" test, a positive result means only a 50% chance of disease — because the disease is rare, false positives are as numerous as true positives. This is the famous base-rate fallacy.

Common Misconceptions

The Gambler's Fallacy

Independent events have no memory. After five heads in a row with a fair coin, the probability of heads on the next toss is still exactly $\frac{1}{2}$ — the coin does not "owe" you a tail. By equation [10], past tosses are independent of future ones, so

P (H_{6} ∣ H_{1} \cap \dots \cap H_{5}) = P (H_{6}) = \frac{1}{2} .

The belief that "tails is due" confuses independent trials with self-correcting systems.

Confusing $P(A\mid B)$ with $P(B\mid A)$

These are not equal in general — that's exactly why Bayes' theorem (equation [11]) is needed. The medical example shows $P (positive ∣ disease) = 0.99$ but $P (disease ∣ positive) = 0.50$ . Reversing the conditioning (the "prosecutor's fallacy") has led to real wrongful convictions.

Pairwise Independence ≠ Mutual Independence

Toss two fair coins. Define three events: $A$ = "first is heads," $B$ = "second is heads," $C$ = "the two coins differ." Each pair is independent: $P (A \cap C) = \frac{1}{4} = P (A) P (C)$ , and similarly for the others. But the three are not mutually independent, because $A \cap B$ forces "same," making $C$ impossible: $P (A \cap B \cap C) = 0 \neq = P (A) P (B) P (C) = \frac{1}{8}$ . Pairwise independence does not imply joint independence.

Connections

Probability foundations sit at the crossroads of several areas. The classical formula [8] rests on Combinatorics for counting favorable and total outcomes. The axioms are a special case of Measure Theory, where $P$ is a measure with total mass 1. The transition from discrete to continuous sample spaces requires Calculus and the Probability Density Function, replacing sums with integrals. Bayes' theorem [11] is the engine of Bayesian Inference and modern machine learning. The behavior of repeated experiments connects to Law of Large Numbers and the Central Limit Theorem, which explain why averages stabilize and why the Normal Distribution appears so universally.

Real-World Applications

Insurance & Actuarial Science — Premiums are set using probabilities of death, accident, or loss, with expected value (equation derived from weights) determining fair pricing.
Medical Diagnosis — Bayes' theorem [11] converts test accuracy and disease prevalence into the actual probability a patient is ill, as in the worked example.
Quantum Mechanics — The Born rule interprets the squared wavefunction amplitude $∣ ψ ∣^{2}$ as a probability density; physics is fundamentally probabilistic at small scales.
Machine Learning — Spam filters, recommendation engines, and large language models use conditional probability and Bayesian updating to make predictions.
Genetics — Punnett squares apply classical probability [8] to predict offspring genotype ratios.
Finance & Risk — Option pricing (Black–Scholes) and Value-at-Risk models rest on probability distributions of asset returns.
Cryptography & Communications — Error-correcting codes and secure key exchange rely on probabilities of bit errors and adversary success.

Historical Development

1654 — Blaise Pascal and Pierre de Fermat exchange letters solving the "problem of points" (how to divide stakes in an interrupted game of chance), founding mathematical probability.
1657 — Christiaan Huygens publishes De Ratiociniis in Ludo Aleae, the first printed treatise on probability, introducing expected value.
1713 — Jacob Bernoulli's posthumous Ars Conjectandi proves the first Law of Large Numbers.
1733–1738 — Abraham de Moivre derives the normal curve as an approximation to the binomial distribution.
1763 — Thomas Bayes' theorem on inverse probability is published posthumously by Richard Price.
1812 — Pierre-Simon Laplace's Théorie analytique des probabilités systematizes the classical theory and the principle of equally likely outcomes.
1900 — Hilbert's sixth problem calls for the axiomatization of physics, explicitly including probability.
1933 — Andrey Kolmogorov publishes his measure-theoretic axioms, giving probability rigorous foundations and ending centuries of paradox.
1950s–present — Frequentist vs. Bayesian interpretations are debated; computation and machine learning make Bayesian methods practical and dominant.

Subtleties & Limits

What does probability actually mean? Kolmogorov's axioms tell you how to compute with probabilities, but not what a probability is. Two major interpretations compete:

Frequentist: $P (A)$ is the long-run relative frequency of $A$ in repeated trials. But this struggles with one-time events ("probability it rains tomorrow") and never quite defines the limit rigorously.
Bayesian: $P (A)$ is a degree of rational belief, updated via Bayes' theorem as evidence arrives. But whose belief, and what prior is "correct"?

Deeper subtleties: not every subset of a continuous sample space can be assigned a probability (non-measurable sets, related to the Banach–Tarski paradox), which is why measure theory restricts events to a $σ$ -algebra. And the foundations of inference — how much data justifies a conclusion — remain philosophically contested. Probability is mathematically settled but interpretively open.