Bayes Theorem

Bayes' Theorem is an exact identity of probability theory that inverts conditional probabilities: it converts the "likelihood" of evidence given a hypothesis into the "posterior" probability of the hypothesis given the evidence. It matters because most real reasoning runs backward from observed data to hidden causes — we see symptoms and want diseases, see data and want parameters. The theorem is mathematically trivial (a one-line consequence of the definition of conditional probability), yet it underpins an entire philosophy of inference, Bayesianism, that treats probability as degree of belief. The surprising fact: a test that is "99% accurate" can still mean a positive result is more likely a false alarm than a true detection when the underlying condition is rare — a consequence purely of the prior, and a constant source of public misjudgment.

The Core Insight

Evidence does not tell you $P(\text{hypothesis}\mid \text{evidence})$ directly — it gives you $P(\text{evidence}\mid \text{hypothesis})$. Bayes' Theorem is the bridge that flips this direction, and the flip requires you to bring in the prior $P(\text{hypothesis})$ — your belief before the evidence. The posterior is proportional to likelihood times prior. Forgetting the prior (the base rate) is the single most common error in probabilistic reasoning.

Formal Statement

Conditional Probability

For events $A$ and $B$ in a probability space with $P(B)>0$, the conditional probability of $A$ given $B$ is

Here $P(A\cap B)$ is the probability that both $A$ and $B$ occur, and $P(B)$ is the (positive) probability of the conditioning event.

Bayes' Theorem

Let $A$ and $B$ be events with $P(A)>0$ and $P(B)>0$. Then

The terms are named:

• $P(A\mid B)$ — the posterior: belief in $A$ after observing $B$.
• $P(B\mid A)$ — the likelihood: probability of evidence $B$ if $A$ holds.
• $P(A)$ — the prior: belief in $A$ before observing $B$.
• $P(B)$ — the evidence (or marginal likelihood): total probability of observing $B$.

Law of Total Probability and Expanded Form

If $\{A_1,A_2,\dots ,A_n\}$ is a partition of the sample space (mutually exclusive, exhaustive, each with positive probability), the denominator can be expanded:

Substituting into Equation [2] gives the form used in practice:

For the common binary case ($A$ and its complement $A^c$):

Odds Form

Dividing the posterior for $A$ by that for $A^c$, the evidence $P(B)$ cancels, yielding the elegant odds form:

The likelihood ratio (or Bayes factor) is the multiplicative strength of the evidence — how many times more often $B$ occurs under $A$ than under $A^c$.

Why It Holds / Derivation

Derivation of Bayes' Theorem

We derive Equation [2] from the definition of conditional probability.

This is Equation [2]. $■$

The entire content of the theorem is the symmetry of the joint probability viewed through two different conditionings. The expanded forms [4]–[6] are obtained by rewriting $P(B)$ using the law of total probability or by taking ratios.

Thomas Bayes | c. 1701–1761

Thomas Bayes was an English Presbyterian minister and mathematician, a Fellow of the Royal Society. He never published his theorem in his lifetime; it appeared in "An Essay towards solving a Problem in the Doctrine of Chances", communicated to the Royal Society in 1763 by his friend Richard Price, two years after Bayes' death. Bayes' original problem was inverse: given a number of successes and failures in independent trials with unknown success probability $p$, what can be said about $p$? He effectively solved a special case of inference for a binomial parameter using a uniform prior. The general formulation and its systematic use in inference are largely due to Pierre-Simon Laplace, who rediscovered and vastly extended the idea independently. Much of what we call "Bayesian" is really Laplacian.

Building Intuition

Think of probability as area. The whole sample space is a unit square. Event $A$ carves out a region; event $B$ carves out another; they overlap in $A\cap B$.

When you learn that $B$ happened, you erase everything outside $B$ — your world shrinks to the $B$ region. The question "how likely is $A$ now?" becomes: what fraction of the surviving $B$-region also belongs to $A$? That fraction is $P(A\cap B)/P(B)$ — exactly Equation [1].

Bayes' Theorem is just the observation that you can compute that overlap two ways. You might naturally know "of all the $A$-cases, what fraction are also $B$" (the likelihood) plus "how big is $A$ to begin with" (the prior). Bayes assembles these into the overlap and divides by the size of $B$ to flip the conditioning.

The crucial intuition is base rates: even if $B$ almost always accompanies $A$ (high likelihood), if $A$ is itself tiny (low prior), then most of region $B$ may come from the complement $A^c$. A big slice of a rare event can be dwarfed by a tiny slice of a common one.

Updating belief: prior region, evidence narrows the world, posterior is the surviving overlap

Worked Examples

Medical Test with a Rare Disease

• Sensitivity (true positive rate): $P(+\mid D)=0.99$
• False positive rate: $P(+\mid D^c)=0.05$ (so specificity $=0.95$).

A randomly screened person tests positive. Find $P(D\mid +)$.

Despite a "99% accurate" test, a positive result means only about a 2% chance of actually having the disease — because false positives from the huge healthy population (0.04995) swamp the true positives (0.00099). This is the base-rate fallacy.

Two Factories and a Defective Part — Odds Form

Let $S_2$ = "from Supplier 2", $S_1=S_2^c$ = "from Supplier 1", $B$ = "defective".

Though Supplier 2 makes only 30% of bolts, its higher defect rate makes it the more likely source of any given defective bolt.

Common Misconceptions

The Prosecutor's Fallacy

Do not confuse the likelihood $P(B\mid A)$ with the posterior $P(A\mid B)$ — they are generally very different. In a courtroom, "the probability of this DNA match if the defendant is innocent is one in a million" ($P(\text{match}\mid \text{innocent})=10^{-6}$) is not the same as "the probability the defendant is innocent given the match." With a database of millions of people, many innocent matches can occur. Equating these two reversed conditionals is the prosecutor's fallacy, and it has led to real wrongful convictions. Bayes' Theorem [2] shows the conversion requires the prior $P(A)$ — which the fallacy silently ignores.

High Accuracy ≠ High Posterior

A common belief: "If a test is 99% accurate and I test positive, I'm 99% likely to have the condition." The medical example above is a direct counterexample: with $P(+\mid D)=0.99$ and a low base rate $P(D)=0.001$, the posterior is only 1.94%. The same "99% accurate" test gives very different posteriors depending entirely on the prior:

| Prior $P(D)$ | Posterior $P(D\mid +)$ | |---|---| | 0.001 | 1.9% | | 0.01 | 16.7% | | 0.10 | 68.7% | | 0.50 | 95.2% |

Accuracy is a property of the test; the posterior is a property of the test plus the population. There is no posterior without a prior.

Connections

Bayes' Theorem sits at the hub of probability and inference:

• It is an immediate corollary of Conditional Probability and the Law of Total Probability.
• It generalizes from events to densities, giving the foundation of Bayesian Inference, where one updates a Prior Distribution into a Posterior Distribution using a Likelihood Function.
• In its repeated/sequential form it is the engine of Bayesian Updating and recursive estimators like the Kalman Filter and Hidden Markov Models.
• The Naive Bayes Classifier applies it with a conditional-independence assumption for classification.
• It contrasts philosophically with Frequentist Inference and connects to Maximum Likelihood Estimation (the MAP estimate reduces to the MLE under a flat prior).
• The odds form links it to Information Theory via the log-likelihood ratio as evidence in bits.

Real-World Applications

1. Medical diagnostics & screening — interpreting mammograms, HIV tests, and COVID-19 antigen tests by combining test sensitivity/specificity with disease prevalence (base rate), exactly as in the worked example.
2. Spam filtering — the Naive Bayes classifier scores emails by combining per-word likelihoods $P(\text{word}\mid \text{spam})$ with a spam prior; pioneered in tools like SpamAssassin.
3. Search and rescue — the U.S. Coast Guard's SAROPS system maintains a posterior probability map of a missing vessel's location, updated as each search sector is cleared (a Bayesian "fail to find" update). The recovery of the wreck of Air France Flight 447 (2011) used Bayesian search theory.
4. Forensic science & law — likelihood ratios for DNA, fingerprint, and trace evidence are combined with case priors; misuse produces the prosecutor's fallacy.
5. Machine learning — Bayesian networks, Gaussian processes, variational inference, and Bayesian neural networks all rest on posterior computation; Markov Chain Monte Carlo (MCMC) samples high-dimensional posteriors.
6. Spacecraft & robotics navigation — Kalman and particle filters fuse noisy sensor readings into a posterior estimate of position and velocity; used in GPS, self-driving cars, and the Apollo guidance computer.
7. Spam/fraud detection and A/B testing — Bayesian methods update fraud probabilities per transaction and give continuously interpretable test results without fixed sample sizes.

Historical Development

• 1763 — Richard Price posthumously presents Bayes' Essay towards solving a Problem in the Doctrine of Chances to the Royal Society, containing the first version of the theorem for inferring a binomial parameter.
• 1774 — Pierre-Simon Laplace, independently, formulates the general principle of inverse probability and applies it broadly (population statistics, astronomy, jurisprudence).
• 1812 — Laplace's Théorie analytique des probabilités systematizes "inverse probability," the dominant inferential framework of the 19th century.
• 1920s–1930s — R. A. Fisher, Jerzy Neyman, and Egon Pearson develop frequentist methods (significance tests, confidence intervals), pushing Bayesian methods to the margins amid criticism of subjective priors.
• 1939–1961 — Harold Jeffreys revives objective Bayesianism in Theory of Probability, introducing non-informative ("Jeffreys") priors.
• 1950s–1960s — Bruno de Finetti and L. J. Savage ground subjective probability and Bayesianism in axioms of rational decision-making (coherence, exchangeability).
• 1953 / 1970 — The Metropolis and Hastings algorithms (MCMC) are developed, eventually making high-dimensional posteriors computable.
• 1980s–1990s — Cheap computing and the Gibbs sampler (Geman & Geman, 1984; Gelfand & Smith, 1990) trigger a Bayesian renaissance across statistics and machine learning.
• 2000s–present — Probabilistic programming (Stan, PyMC), Bayesian deep learning, and Bayesian optimization make the theorem a standard workhorse of modern data science.

Subtleties & Limits

The Problem of the Prior

Bayes' Theorem is mathematically uncontroversial, but its use in inference raises deep, partly unresolved issues:

• Choosing the prior. Where does $P(A)$ come from when no frequency data exist? "Objective" or "non-informative" priors (uniform, Jeffreys, reference priors) attempt to encode ignorance, but a prior uniform in one parameterization is non-uniform in another — there is no universally canonical ignorance prior. For continuous infinite-dimensional spaces, ignorance can even be incoherent.
• Improper priors. Common non-informative priors (e.g. $p(\mu )\propto 1$ over all reals) do not integrate to 1; they sometimes yield proper posteriors and sometimes yield paradoxes or nonexistent posteriors.
• Sensitivity and robustness. With small data, conclusions can depend heavily on the prior; quantifying and bounding this sensitivity (robust Bayes) is an active area.
• Computation. Exact posteriors are intractable for most realistic models; MCMC and variational methods give approximations whose convergence and error are not always guaranteed.
• Calibration vs. coherence. Bayesian updates are internally coherent but need not be well-calibrated against reality if the model or prior is misspecified — the theorem cannot rescue a wrong model.

These are not flaws in the theorem (an exact identity) but open questions about how to deploy it as a theory of inference.

See Also