Bayes’ Theorem
Bayesian probability is a powerful framework for reasoning under uncertainty. Unlike classical probability, which treats probabilities as fixed frequencies based on equally likely outcomes, Bayesian probability treats probability as a measure of belief or confidence in an event — a belief that can and should be updated whenever new evidence arrives. This makes it especially useful in real-world situations where information accumulates over time. At the heart of Bayesian probability is Bayes' Theorem, a mathematical formula derived from the basic rules of conditional probability. It tells you exactly how to revise your probability estimate for a hypothesis H after observing some evidence E. You start with a 'prior' probability — your best guess before seeing the evidence — and the theorem produces a 'posterior' probability that incorporates the new information. This process of updating beliefs is called Bayesian inference. Bayes' Theorem has applications across virtually every quantitative field: medical diagnosis (how likely is a patient to have a disease given a positive test?), spam filtering, machine learning, forensic science, and even weather forecasting. Understanding it gives you a rigorous tool for thinking clearly about evidence and uncertainty, skills that are valuable far beyond mathematics class.
- Recall Conditional Probability
- Write the Joint Probability Two Ways
- Solve for P(A | B)
- Expand the Denominator Using the Law of Total Probability
- Identify the Three Key Ingredients
Recall Conditional Probability
The conditional probability of event A given event B has already occurred is defined as P(A | B) = P(A ∩ B) / P(B), provided P(B) > 0. This captures how knowing B changes the likelihood of A.
Write the Joint Probability Two Ways
The joint probability P(A ∩ B) can be expressed using either variable as the condition: P(A ∩ B) = P(A | B) · P(B) and also P(A ∩ B) = P(B | A) · P(A). Since both expressions equal P(A ∩ B), we can set them equal: P(A | B) · P(B) = P(B | A) · P(A).
Solve for P(A | B)
Dividing both sides by P(B) gives us Bayes' Theorem: P(A | B) = [P(B | A) · P(A)] / P(B). In Bayesian language, A is the hypothesis H and B is the evidence E, so we write P(H | E) = [P(E | H) · P(H)] / P(E).
Expand the Denominator Using the Law of Total Probability
Often P(E) is not directly known. If H and its complement H̄ (not-H) are mutually exclusive and exhaustive, then by the Law of Total Probability: P(E) = P(E | H) · P(H) + P(E | H̄) · P(H̄). Substituting this into the denominator gives the full form of Bayes' Theorem, which allows calculation even when the overall probability of the evidence is not immediately obvious.
Identify the Three Key Ingredients
To apply Bayes' Theorem you need three things: (1) the Prior P(H) — your initial probability for the hypothesis before seeing evidence; (2) the Likelihood P(E | H) — how probable the evidence would be if the hypothesis were true; and (3) the Marginal P(E) — the overall probability of the evidence under all scenarios. The result, P(H | E), is called the Posterior — your updated belief after incorporating the evidence.
The Key Equation
- P(H \mid E) — Posterior probability (dimensionless [0, 1]): The updated probability of hypothesis H after observing evidence E.
- P(E \mid H) — Likelihood (dimensionless [0, 1]): The probability of observing evidence E assuming hypothesis H is true.
- P(H) — Prior probability (dimensionless [0, 1]): The initial probability of hypothesis H before any evidence is considered.
- P(\bar{H}) — Prior of complement (dimensionless [0, 1]): The probability that H is false; equals 1 − P(H).
- P(E \mid \bar{H}) — False positive rate (likelihood under not-H) (dimensionless [0, 1]): The probability of observing evidence E even when hypothesis H is false.
Worked Example
Problem. A medical test for a rare disease correctly identifies a sick patient 99% of the time (sensitivity = 0.99). However, it also gives a false positive 2% of the time for healthy patients. The disease affects 0.5% of the population. A patient tests positive. What is the probability they actually have the disease?
Answer. Approximately 19.9% — the patient has roughly a 1-in-5 chance of actually having the disease despite testing positive.
Origins · 1 minThomas Bayes and how this was discovered
Thomas Bayes was an English statistician, philosopher, and Presbyterian minister. He formulated the theorem that bears his name as a way to reason about inverse probability — given observed outcomes, what can we infer about unknown causes? His key essay, 'An Essay towards solving a Problem in the Doctrine of Chances,' was published posthumously in 1763 by his friend Richard Price, who recognized its significance and submitted it to the Royal Society. Bayes framed the problem geometrically using a billiard table thought experiment, and his ideas were later extended and popularized by French mathematician Pierre-Simon Laplace. Bayes never published his theorem himself during his lifetime. It was found among his papers after his death and submitted for publication by a friend. For over a century it remained a mathematical curiosity before becoming one of the most important ideas in modern statistics and artificial intelligence.
Related Concepts
- Conditional Probability — Bayes' Theorem is derived directly from the definition of conditional probability. Mastering P(A | B) is the essential prerequisite for understanding Bayes' Theorem.
- Law of Total Probability — The denominator in Bayes' Theorem is calculated using the Law of Total Probability, which partitions the sample space into mutually exclusive and exhaustive events.
- Independence and the Multiplication Rule — Understanding when events are independent (or not) is crucial for correctly identifying when Bayes' Theorem must be used instead of simpler multiplication rules.
- Sample Spaces and Events — Probability theory requires a well-defined sample space. Understanding events and their complements underpins the logic of prior and posterior probabilities.