Bayes' Theorem

Suppose a diagnostic test for a rare disease is 99% accurate. A patient tests positive. How worried should they be? Intuition trained on the accuracy figure alone screams "very worried." Yet the honest answer depends crucially on how rare the disease actually is. Getting from "accuracy of the test" to "chance you're actually sick" requires inverting a conditional probability. Bayes' theorem is the precise, provable rule for performing that inversion. The idea matters because most evidence in science, medicine, law, and everyday reasoning arrives in the wrong direction. We observe an effect and want to reason back to a cause. We know $P(\text{evidence} \mid \text{hypothesis})$ — this is what a model or an experiment tells us directly. But we want $P(\text{hypothesis} \mid \text{evidence})$ instead. Bayes' theorem is the bridge between these two. It is also the algebraic engine behind an entire philosophy of statistical inference (Bayesian statistics). In this philosophy, unknown parameters are treated as random quantities. They carry prior beliefs that get updated into posterior beliefs upon seeing data. The big picture is deceptively simple. The theorem is a one-line rearrangement of the definition of conditional probability. Its power comes from combining it with the Law of Total Probability to compute a normalizing constant. There is no new axiom here. It only demands the discipline of writing down every piece explicitly. This means the prior probability of each hypothesis, the likelihood of the observed evidence under each hypothesis, and the resulting posterior probability after the evidence is taken into account. This article states the precise definitions of prior, likelihood, and posterior. It proves the theorem rigorously from the axioms of probability. It works through two asymmetric numerical examples (a diagnostic test and an urn/factory problem). It shows exactly what breaks if the partition hypothesis is dropped. Finally, it surveys where the theorem is used today, from spam filters to forensic DNA evidence to the foundations of Independence of Events.

Key takeaways

Bayes' theorem inverts a conditional probability: it converts $P(B\mid A)$ into $P(A\mid B)$ .
It requires a partition of hypotheses $A_1,\dots,A_n$ (mutually exclusive, exhaustive) with known priors $P(A_i)$ .
The likelihood $P(B\mid A_i)$ measures how probable the observed evidence $B$ is under each hypothesis.
The denominator is exactly the Law of Total Probability applied to $B$ ; it normalizes the posterior to sum to 1.
Base rates dominate: rare hypotheses stay improbable even under fairly strong evidence — this is the resolution of the diagnostic-test paradox.
Bayes' theorem is a theorem, not an assumption — it follows purely from the definition of conditional probability.
Key idea: posterior probability is proportional to prior times likelihood, renormalized so all hypotheses' posteriors sum to one.

Where this lives today

Bayes' theorem is the computational core of Bayesian statistics, in which unknown parameters $\theta$ are assigned a prior distribution $\pi(\theta)$ , data $x$ are modeled via a likelihood $f(x\mid\theta)$ , and inference proceeds by computing the posterior $\pi(\theta\mid x) \propto f(x\mid\theta)\pi(\theta)$ — an infinite-partition version of the same identity proved above. In medical diagnostics and epidemiology, it is the standard tool for converting test sensitivity/specificity and disease prevalence into predictive values, exactly as in Example 1. In forensic science and law, likelihood ratios (the odds-form corollary) are used to weigh DNA or fingerprint evidence, with careful attention paid to avoiding the prosecutor's fallacy. In machine learning, naive Bayes classifiers apply the theorem (with a simplifying conditional-independence assumption, foreshadowing Independence of Events) to spam filtering, document classification, and sentiment analysis. In engineering, Bayesian filters — including the Kalman filter and particle filters — recursively apply the theorem to update state estimates from noisy sensor readings, underlying GPS navigation and robotics. In cryptography and information theory, Bayesian updating models an adversary's evolving beliefs about a secret key given observed ciphertext, informing security proofs. In cognitive science, "Bayesian brain" models propose that perception itself operates as approximate Bayesian inference, combining sensory likelihoods with learned priors.

Ask the tutor: How does Bayes' theorem generalize when the evidence arrives as a continuous stream of observations rather than a single event?

(\Omega, \mathcal{F}, P)

Intuition

Think of the sample space $\Omega$ as sliced into disjoint wedges $A_1, \dots, A_n$ — the possible "causes" or "states of the world." The event $B$ — the observed "evidence" — is some region overlapping several wedges. Before you learn that $B$ happened, your belief about which wedge you're in is given by the prior areas $P(A_i)$ . Once you learn $B$ occurred, you know you are somewhere in the shaded region $B$ , and your belief about which wedge you're in should be re-proportioned according to how much of $B$ sits inside each wedge — that's exactly $P(A_i \cap B) = P(B\mid A_i) P(A_i)$ . Dividing by the total area of $B$ renormalizes these into genuine probabilities. The key perceptual trap the diagnostic-test example exposes is confusing $P(B\mid A)$ with $P(A\mid B)$ . A test can be highly reliable ( $P(\text{positive}\mid\text{sick})$ large) while $P(\text{sick}\mid\text{positive})$ remains small, purely because sick people are rare — the wedge $A$ = "sick" is tiny compared to $\Omega$ , so even a fairly complete overlap of $B$ with $A$ is dwarfed by the small sliver of $B$ that leaks into the enormous wedge $A^c$ = "healthy."

Figure

Partition of

\Omega

into hypotheses

A_1,A_2,A_3

, with evidence event

B

overlapping each

Deriving the theorem

(\Omega,\mathcal F,P)

E, F

Note the proof technique: direct algebraic derivation from the definitions, using the Law of Total Probability only as a lemma to rewrite the denominator — no circularity, since the Law of Total Probability does not itself depend on Bayes' theorem.

A

A

This corollary says: posterior odds = likelihood ratio × prior odds. It is the form most used in sequential updating, since each new piece of evidence simply multiplies the running odds by a new likelihood ratio.

Worked examples

Example 1 — the diagnostic test (asymmetric numbers, two hypotheses). A disease is present in $0.4\%$ of a population: $P(D) = 0.004$ , so $P(D^c) = 0.996$ . A test has sensitivity $P(+\mid D) = 0.97$ (true positive rate) and false-positive rate $P(+\mid D^c) = 0.05$ . A random person tests positive. Find $P(D \mid +)$ . By Bayes' theorem with partition $\{D, D^c\}$ and evidence $B = \{+\}$ :

P(D \mid +) = \frac{P(+\mid D) P(D)}{P(+\mid D) P(D) + P(+\mid D^c) P(D^c)}.

Compute the numerator:

P(+\mid D) P(D) = 0.97 \times 0.004 = 0.00388.

Compute the other joint term:

P(+\mid D^c) P(D^c) = 0.05 \times 0.996 = 0.0498.

Sum (this is $P(+)$ via the Law of Total Probability):

P(+) = 0.00388 + 0.0498 = 0.05368.

Therefore

P(D \mid +) = \frac{0.00388}{0.05368} \approx 0.07227.

So even with a $97\%$ -sensitive test and only a $5\%$ false-positive rate, a positive result implies only about a 7.2% chance of actually having the disease — because the disease is rare, the $5\%$ false-positive rate applied to the huge healthy population ( $0.0498$ ) swamps the true positives ( $0.00388$ ). This is the classic base-rate phenomenon, and it is not a paradox once the denominator is examined term by term. Checking via odds form: prior odds of disease are $0.004/0.996 \approx 0.004016$ ; likelihood ratio is $0.97/0.05 = 19.4$ ; posterior odds $\approx 0.004016 \times 19.4 \approx 0.07791$ , giving $P(D\mid+) = 0.07791/(1+0.07791) \approx 0.07227$ — matching, as it must, confirming internal consistency of the two forms.

Example 2 — three factories, asymmetric priors and likelihoods. A company buys light bulbs from three factories, $A_1, A_2, A_3$ , supplying $50\%$ , $30\%$ , $20\%$ of stock respectively:

P(A_1) = 0.50, \quad P(A_2) = 0.30, \quad P(A_3) = 0.20.

Their defect rates are $P(D\mid A_1) = 0.02$ , $P(D\mid A_2) = 0.05$ , $P(D\mid A_3) = 0.08$ . A bulb is drawn at random and found defective ( $D$ ). Find $P(A_3 \mid D)$ , the probability it came from the highest-defect-rate factory, and compare to its prior $P(A_3) = 0.20$ . Compute each joint probability $P(D \cap A_j) = P(D\mid A_j) P(A_j)$ :

P(D \cap A_1) = 0.02 \times 0.50 = 0.010,

P(D \cap A_2) = 0.05 \times 0.30 = 0.015,

P(D \cap A_3) = 0.08 \times 0.20 = 0.016.

By the Law of Total Probability,

P(D) = 0.010 + 0.015 + 0.016 = 0.041.

By Bayes' theorem,

P(A_3 \mid D) = \frac{0.016}{0.041} = \frac{16}{41} \approx 0.3902.

So observing a defect nearly doubles our belief that the bulb came from factory 3, moving it from a $20\%$ prior share to a $\approx 39.0\%$ posterior share — even though factory 3 supplies the fewest bulbs, its higher defect rate makes it disproportionately likely to be the source once a defect is observed. For completeness,

P(A_1 \mid D) = \frac{0.010}{0.041} = \frac{10}{41} \approx 0.2439, \qquad P(A_2 \mid D) = \frac{0.015}{0.041} = \frac{15}{41} \approx 0.3659,

and indeed $10/41 + 15/41 + 16/41 = 41/41 = 1$ , confirming the posteriors form a valid probability distribution over the partition, as guaranteed by the theorem's denominator being exactly $P(D)$ .

\{A_i\}

Scope, edge cases & common traps

Situation	What happens	Correct handling
$\{A_i\}$ not a partition	Denominator $\ne P(B)$ ; formula invalid (see counterexample)	Choose or verify a genuine partition before applying the theorem
Some $P(A_i) = 0$	Conditioning $P(A_i \mid B)$ undefined for that hypothesis; term still contributes 0 to the sum if $P(A_i)=0$	Drop hypotheses with zero prior, or restrict to positive-prior atoms
$P(B) = 0$	$P(A_i \mid B)$ undefined (division by zero)	Bayes' theorem does not apply to zero-probability evidence; needs limiting/density arguments (continuous case)
Continuous parameter $\theta$	Sums become integrals: $f(\theta\mid B) = f(B\mid\theta)\pi(\theta)/\int f(B\mid\theta')\pi(\theta')\,d\theta'$	Same theorem, densities replace probability masses
Confusing $P(B\mid A)$ with $P(A\mid B)$	The "prosecutor's fallacy" — treating likelihood as posterior	Always renormalize by $P(B)$ , weighting by the prior

Common trapThe classic error: swapping the direction of conditioning

The single most common mistake — sometimes called the prosecutor's fallacy in legal contexts — is asserting $P(A\mid B) = P(B\mid A)$ or otherwise treating the likelihood as though it were already the posterior. In the diagnostic test example, $P(+\mid D) = 0.97$ is not $P(D\mid +)$ ; the latter, correctly computed via Bayes' theorem, was only about $0.072$ . Whenever a "the test is $X\%$ accurate, therefore $P(\text{hypothesis}\mid\text{evidence}) = X\%$ " argument appears, check whether the base rate (prior) has been accounted for at all — if not, the argument is invalid regardless of how accurate the test genuinely is.

The mathematician behind it

Origins · 1 minThomas Bayes and how this was discovered

Thomas Bayes

c. 1701–1761 · Presbyterian minister and mathematician, Tunbridge Wells, England

Thomas Bayes was an English Nonconformist minister with a serious sideline in mathematics, elected a Fellow of the Royal Society in 1742 largely on the strength of a defense of Newton's calculus against philosophical attacks. The result now bearing his name appeared only posthumously: his friend Richard Price found the manuscript among Bayes' papers after his death, edited it, and presented it to the Royal Society in 1763 as "An Essay towards solving a Problem in the Doctrine of Chances." Bayes' original essay addressed a specific inverse-probability problem — given the number of successes in Bernoulli trials, what can be inferred about the underlying success probability — using a billiard-table thought experiment to justify a uniform prior. The general algebraic form used throughout probability today, and the modern emphasis on priors and posteriors as a general framework for inference, was substantially developed later by Pierre-Simon Laplace, working independently and more systematically.

History

- c. 1701 — Thomas Bayes is born (exact date and birthplace uncertain), son of a Presbyterian minister in London.

- 1742 — Bayes is elected Fellow of the Royal Society.

- 1761 — Bayes dies in Tunbridge Wells; his mathematical papers pass to his family.

- 1763 — Richard Price posthumously communicates Bayes' essay "An Essay towards solving a Problem in the Doctrine of Chances" to the Royal Society.

- 1774–1812 — Pierre-Simon Laplace independently derives and generalizes inverse probability, embedding it in the broader theory now called Bayesian inference.

Is Bayes' theorem an assumption of Bayesian statistics, or a proven fact?

It is a proven mathematical fact, true in any probability space satisfying the Law of Total Probability hypotheses (a valid partition with positive prior probabilities). What is philosophically distinctive about Bayesian statistics is not the theorem itself — frequentist and Bayesian statisticians agree it is a theorem — but the willingness to assign prior probability distributions to unknown parameters or hypotheses, treating them as random quantities with degrees of belief. The theorem then supplies the mechanically correct way to update those beliefs given data. So the controversy in statistical philosophy is about the interpretation and use of priors, never about whether the algebraic identity holds.

Why does a positive result on an accurate test still leave a low probability of disease?

Because the theorem weighs the likelihood against the prior, and rare diseases have tiny priors. Even a small false-positive rate applied to the enormous healthy population can generate more false positives in absolute terms than the small diseased population generates true positives. Concretely, in Example 1 the false positives contributed $0.0498$ to $P(+)$ while true positives contributed only $0.00388$ — the healthy group is 249 times larger, so its 5% false-positive rate dwarfs the sick group's 97% true-positive rate in absolute counts. This is exactly the base-rate effect Bayes' theorem quantifies precisely.

What happens if two candidate hypotheses overlap instead of being disjoint?

The theorem's denominator is derived from the Law of Total Probability, which strictly requires a partition — pairwise disjoint and exhaustive events. If the "hypotheses" overlap, the sum $\sum_j P(B\mid A_j)P(A_j)$ double-counts the overlapping probability mass and no longer equals $P(B)$ , so the formula's output is simply the wrong number, as shown explicitly in the counterexample above. The fix is always to first construct a genuine partition — for instance, by refining the events into disjoint atoms — before applying the theorem.

Does Bayes' theorem work with continuous random variables and probability densities?

Yes, with sums replaced by integrals. If $\theta$ has prior density $\pi(\theta)$ and data $x$ has conditional density $f(x\mid\theta)$ , the posterior density is

\pi(\theta \mid x) = \frac{f(x\mid\theta)\,\pi(\theta)}{\int f(x\mid\theta')\,\pi(\theta')\,d\theta'},

which is the continuous analogue of the discrete formula, with the denominator again playing the role of a normalizing constant guaranteeing $\int \pi(\theta\mid x)\,d\theta = 1$ . The derivation is essentially identical, using density-based conditional probability and an integral form of the Law of Total Probability.

How does sequential updating with multiple pieces of evidence work?

The odds-form corollary makes this transparent: each new independent piece of evidence multiplies the running posterior odds by its own likelihood ratio. If evidence $B_1$ updates prior odds to posterior odds $O_1$ , and a second piece of evidence $B_2$ is conditionally independent of $B_1$ given each hypothesis, then $O_2 = O_1 \times \frac{P(B_2\mid A)}{P(B_2\mid A^c)}$ . This is exactly how naive Bayes classifiers and sequential Bayesian filters accumulate evidence, and it foreshadows the formal treatment of Independence of Events, since the simple multiplicative update relies on a conditional independence assumption between pieces of evidence.

Can Bayes' theorem give a posterior probability of exactly 0 or 1?

Yes, in degenerate cases. If $P(B \mid A_i) = 0$ for some hypothesis $A_i$ — the evidence is literally impossible under that hypothesis — then its posterior becomes exactly $0$ , regardless of how strong its prior was: one contradicting observation can eliminate a hypothesis entirely. Conversely if every other hypothesis has $P(B\mid A_j) = 0$ while $P(B\mid A_i) > 0$ , then $P(A_i\mid B) = 1$ . This is the mathematical expression of Sherlock Holmes' principle: once impossible alternatives are excluded, whatever remains, however improbable a priori, must be the truth — provided its likelihood for the evidence is strictly positive.