Probability & Statistics

Probability measures how likely an event is on a scale from 0 (impossible) to 1 (certain); statistics is the science of collecting, summarising, and drawing conclusions from data. Key result in words: the long-run frequency of an event converges to its probability, and large random samples reliably reveal the structure of the population they come from. Applies to: gambling, insurance, quality control, medicine, polling, machine learning, and any reasoning under uncertainty. Foundations laid by Pascal & Fermat (1654), formalised by Kolmogorov (1933). Probability is a function

P

that assigns to each possible outcome of a random experiment a number between 0 and 1, obeying a few simple axioms, so that the chances of compound events can be computed by arithmetic. Statistics runs the logic in reverse: given observed data, it infers the unknown probabilities or quantities that generated them. The subject matters because almost no real measurement is exact and almost no future event is certain — probability and statistics give a disciplined language for quantifying and managing that uncertainty. A surprising fact is that even perfectly random processes produce strikingly stable averages: flip a fair coin a million times and the proportion of heads will sit astonishingly close to 0.5, a regularity guaranteed by the Law of Large Numbers. Equally surprising is that almost any well-behaved random quantity, when averaged, becomes bell-shaped — the Central Limit Theorem.

The Core Insight

Randomness in the small produces order in the large. A single coin flip is unpredictable, but the average behaviour of many independent repetitions is governed by precise mathematical laws. Probability builds a deductive model of this randomness from axioms; statistics uses observed averages to work backwards to the model's hidden parameters. The bridge between the two is the idea that relative frequency estimates probability, and that estimate improves predictably as the sample grows.

Formal Statement

Sample Space, Events, and Probability

A random experiment has a sample space $Ω$ , the set of all possible outcomes. An event $A$ is any subset of $Ω$ . A probability measure $P$ assigns to each event a number satisfying Kolmogorov's axioms:

P (A) \geq 0 for every event A

⟳probabilities are non-negative by definition — they count "share of certainty."

P (Ω) = 1

⟳something in the sample space must occur, so the total probability is 1.

P (A \cup B) = P (A) + P (B) if A \cap B = \emptyset

⟳for mutually exclusive (disjoint) events, probabilities simply add.

Here $Ω$ is the set of outcomes, $A, B$ are events, $\cup$ is union ("or"), $\cap$ is intersection ("and"), and $\emptyset$ is the empty event (impossible).

Conditional Probability and Independence

The conditional probability of $A$ given that $B$ has occurred is

P (A ∣ B) = \frac{P ( A \cap B )}{P ( B )}, P (B) > 0

⟳restrict the sample space to

B

and renormalise so

P (B ∣ B) = 1

Events $A$ and $B$ are independent when knowing one tells nothing about the other:

P (A \cap B) = P (A) P (B)

⟳substitute independence into [4] to get

P (A ∣ B) = P (A)

Bayes' Theorem

For events $A$ and $B$ with $P (B) > 0$ ,

P (A ∣ B) = \frac{P ( B ∣ A ) P ( A )}{P ( B )}

⟳write

P (A \cap B)

two ways using [4] — as

P (A ∣ B) P (B)

and

P (B ∣ A) P (A)

— then solve.

If $A_{1}, \dots, A_{n}$ partition $Ω$ , the denominator expands by the law of total probability:

P (B) = i = 1 \sum n P (B ∣ A_{i}) P (A_{i})

⟳

B

is the disjoint union of the

B \cap A_{i}

; apply [3] repeatedly and [4].

Random Variables, Expectation, Variance

A random variable $X$ assigns a number to each outcome. Its expected value (mean) and variance are

E [X] = i \sum x_{i} P (X = x_{i}) (discrete), E [X] = \int_{- \infty}^{\infty} x f (x) d x (continuous)

⟳a probability-weighted average of all values

x_{i}

, with

f (x)

the probability density.

Var (X) = E [(X - μ)^{2}] = E [X^{2}] - μ^{2}, μ = E [X]

⟳expand

(X - μ)^{2}

and use linearity of

E

, noting

E [μ^{2}] = μ^{2}

The standard deviation is $σ = Var (X)$ , in the same units as $X$ .

Key Distributions

The binomial distribution counts $k$ successes in $n$ independent trials each with success probability $p$ :

P (X = k) = (k n) p^{k} (1 - p)^{n - k}, k = 0, 1, \dots, n

⟳each specific sequence has probability

p^{k} (1 - p)^{n - k}

; multiply by the

(k n)

orderings.

The normal (Gaussian) distribution with mean $μ$ and standard deviation $σ$ has density

f (x) = \frac{1}{σ 2 π} exp (- \frac{( x - μ ) ^{2}}{2 σ ^{2}})

⟳the unique symmetric density whose log is quadratic; the constant

\frac{1}{σ 2 π}

makes the total area 1.

Why It Holds / Derivation

The Mean and Variance of a Sample Average

Let $X_{1}, \dots, X_{n}$ be independent random variables, each with mean $μ$ and variance $σ^{2}$ , and let $\overset{ˉ}{X} = \frac{1}{n} \sum_{i} X_{i}$ be the sample mean.

Mean of the average. By linearity of expectation, $E [\overset{ˉ}{X}] = \frac{1}{n} \sum_{i} E [X_{i}] = \frac{1}{n} (n μ) = μ$ . The sample mean is an unbiased estimator of $μ$ .

Variance of a sum of independent variables. For independent variables, variances add: $Var (\sum_{i} X_{i}) = \sum_{i} Var (X_{i}) = n σ^{2}$ .

Scaling. Since $Var (aY) = a^{2} Var (Y)$ with $a = 1/ n$ ,

Var (\overset{ˉ}{X}) = \frac{1}{n ^{2}} (n σ^{2}) = \frac{σ ^{2}}{n}

⟳combine additivity of variance for independent terms with the

a^{2}

scaling rule.

Consequence. The standard deviation of $\overset{ˉ}{X}$ is $σ / n$ , the standard error. As $n \to \infty$ it shrinks to 0, so $\overset{ˉ}{X}$ concentrates ever more tightly around $μ$ — this is the quantitative heart of the Law of Large Numbers. ∎

Andrey Kolmogorov | 1903–1987

The Soviet mathematician Andrey Kolmogorov transformed probability from a collection of clever gambling tricks into a rigorous branch of mathematics. In his 1933 monograph Grundbegriffe der Wahrscheinlichkeitsrechnung he proposed that probability is simply a measure — a non-negative function on sets summing to 1 — borrowing the machinery of measure theory developed by Lebesgue. This axiomatic foundation (equations [1]–[3]) settled centuries of philosophical dispute about what probability is by declining to answer it: instead it specified the rules probability must obey, leaving interpretation open. Kolmogorov also made deep contributions to turbulence, dynamical systems, and complexity theory.

Building Intuition

Think of probability as shared-out certainty. You have one unit of belief to distribute over all possible outcomes; equation [2] says it all has to go somewhere, and [1] says you can't assign negative belief. Conditional probability [4] is zooming in: once you learn $B$ happened, you throw away the part of the world where it didn't and rescale what remains so it again sums to 1.

For statistics, picture a giant opaque urn (the population) and a handful of balls you draw out (the sample). You never see the whole urn, but each handful is a blurry photograph of it. The blur is the standard error $σ / n$ from [12]: bigger handfuls give sharper photographs, but only as fast as $n$ — to halve the blur you must quadruple the sample. This square-root law is why national polls of 1,000 people work surprisingly well, yet doubling accuracy is genuinely expensive.

The normal distribution: the 68–95–99.7 rule. About 68% of probability lies within one standard deviation of the mean.

Worked Examples

Disease Testing and Bayes' Theorem

Given: A disease affects 1% of a population. A test is 99% sensitive (detects the disease when present) and 95% specific (correctly negative when absent). A randomly chosen person tests positive. What is the probability they actually have the disease? Steps: Let

D

= has disease,

T

= tests positive.

$P (D) = 0.01$ , $P (\neg D) = 0.99$
$P (T ∣ D) = 0.99$ , $P (T ∣ \neg D) = 1 - 0.95 = 0.05$
Total probability of a positive test [7]:

P (T) = (0.99) (0.01) + (0.05) (0.99) = 0.0099 + 0.0495 = 0.0594

Bayes' theorem [6]:

P (D ∣ T) = \frac{( 0.99 ) ( 0.01 )}{0.0594} = \frac{0.0099}{0.0594} \approx 0.167

Answer:

P (D ∣ T) \approx 0.17 = 17%

— despite the accurate test, most positives are false alarms because the disease is rare. This is the base-rate effect.

Sampling and the Standard Error

Given: Heights of adult women have

μ = 162

cm and

σ = 7

cm. A random sample of

n = 49

women is taken. Within what range does the sample mean

\overset{ˉ}{X}

fall about 95% of the time? Steps:

Standard error from [12]: $\frac{σ}{n} = \frac{7}{49} = \frac{7}{7} = 1.0$ cm.
By the Central Limit Theorem, $\overset{ˉ}{X}$ is approximately normal about $μ = 162$ cm.
The 95% range is roughly $μ \pm 2 (SE) = 162 \pm 2 (1.0)$ .

Answer:

160 cm \leq \overset{ˉ}{X} \leq 164 cm (about 95% of samples)

. Note the individual heights vary by

\pm 14

cm, but the average of 49 varies only by

\pm 2

cm — averaging suppresses noise.

Common Misconceptions

The Gambler's Fallacy

Independent events have no memory. After five heads in a row, a fair coin is still exactly 50% to land heads next — equation [5] guarantees past flips do not change future ones. The belief that tails is "due" confuses the Law of Large Numbers (long-run averaging) with a non-existent self-correcting force. The average drifts toward 0.5 not because future flips compensate for the past, but because past flips become an ever-smaller fraction of the total.

Correlation Is Not Causation

Two variables can move together — high correlation — without one causing the other. Ice-cream sales and drowning deaths correlate strongly, but neither causes the other; both rise with summer temperature, a confounding variable. Statistics can establish association; only a controlled experiment or careful causal reasoning can establish cause.

Probability of "At Least One" Is Not a Simple Sum

A common error: with a 1-in-6 chance per roll, "the probability of at least one six in 6 rolls" is not $6 \times \frac{1}{6} = 1$ . Probabilities of non-exclusive events don't simply add (that violates the disjointness condition in [3]). Compute via the complement instead:

P (at least one six) = 1 - (\frac{5}{6})^{6} \approx 1 - 0.335 = 0.665.

The naïve sum even exceeds 1 for 7 rolls, which is impossible by [1]–[2].

Connections

Probability is the deductive engine; statistics runs it in reverse for inference. The Central Limit Theorem explains why the Normal Distribution appears everywhere, justifying the confidence intervals and hypothesis tests of Statistical Inference. Bayes' Theorem generalises into Bayesian Inference, an entire framework for updating beliefs with data. The expectation and variance of [8]–[9] connect to Combinatorics (for counting outcomes) and to Calculus (integrals define continuous expectations and densities). Variance and the standard error link to Regression Analysis, where the same square-of-distance ideas underpin least-squares fitting.

Real-World Applications

Insurance & actuarial science: premiums are set so that expected payouts [8] plus a margin stay below collected premiums; the Law of Large Numbers makes aggregate losses predictable.
Medical trials & epidemiology: randomised controlled trials use hypothesis tests and $p$ -values to decide whether a drug beats placebo; Bayes' theorem [6] interprets diagnostic test results.
Political polling: the margin of error reported by Gallup and Pew is essentially $\pm 2 σ / n$ from [12].
Quality control (Six Sigma): factories monitor whether measurements stay within $\pm 6 σ$ of target, using control charts built on the normal distribution [11].
Finance & risk (Value at Risk): banks model portfolio returns as random variables to estimate worst-case losses at a chosen probability.
Machine learning: classifiers like naïve Bayes apply [6] directly; nearly all of statistics underlies modern AI's training and evaluation.
Genetics: Mendelian inheritance ratios are binomial [10]; population genetics tracks allele frequencies probabilistically.

Historical Development

1654 — Blaise Pascal and Pierre de Fermat exchange letters solving the "problem of points," founding mathematical probability.
1657 — Christiaan Huygens publishes De ratiociniis in ludo aleae, the first printed treatise on probability, introducing expected value.
1713 — Jacob Bernoulli's posthumous Ars Conjectandi states and proves the first Law of Large Numbers.
1733/1738 — Abraham de Moivre derives the normal curve as an approximation to the binomial, an early Central Limit Theorem.
1763 — Thomas Bayes' essay (published posthumously) introduces what becomes Bayes' theorem.
1809–1812 — Carl Friedrich Gauss and Pierre-Simon Laplace develop least squares and the general central limit theorem; the Gaussian distribution takes Gauss's name.
1900s–1920s — Karl Pearson, Ronald Fisher, and Jerzy Neyman build modern statistics: correlation, the chi-squared test, maximum likelihood, experimental design, and hypothesis testing.
1933 — Andrey Kolmogorov axiomatises probability with measure theory, giving the field its rigorous modern foundation.

Subtleties & Limits

The meaning of probability remains philosophically contested. Frequentists define $P (A)$ as the long-run fraction of times $A$ occurs in repeated trials — but this is awkward for one-off events ("the probability it rains tomorrow"). Bayesians treat probability as a degree of belief, updated by [6] — but then the choice of prior belief is subjective. Both interpretations use the same axioms [1]–[3]; they disagree on what the numbers mean. Deeper still: real data are rarely truly independent or identically distributed, so the clean theorems above are idealisations. Modern statistics wrestles with the replication crisis — many published findings fail to reproduce, partly from misuse of $p$ -values and " $p$ -hacking" — and with making valid inferences from massive, messy, non-random datasets where the classical $n$ guarantees may not apply.