Bayes theorem

A doctor tells a patient that a test for a rare disease has come back positive, and the test is "99% accurate." How worried should the patient actually be? The instinct is to say "99% worried" — but that instinct is almost always wrong, and the reason is a piece of mathematics that runs conditional probability backwards. Every conditional probability statement so far in this chapter has answered a forward question: given the cause, what is the chance of the effect? A bag is chosen, then a ball is drawn — what is the chance the ball is red? Bayes' theorem flips that question around. Given that the effect has already happened — the ball drawn is red, the test came back positive — what is the chance it arose from one particular cause among several possible ones? It takes the total probability theorem, which adds up the contributions of every possible route to an outcome, and asks what fraction of that total came from one specific route. This is the capstone result of the Probability chapter, sitting directly on top of conditional probability, the multiplication theorem, and the total probability theorem.

The core insight

Bayes' theorem finds the probability of a particular cause given that an effect has been observed, by dividing that cause's contribution to the effect by the total probability of the effect summed over all possible causes.

The core idea

Setting up the ingredients. Bayes' theorem needs a partition of the sample space $S$ into events $E_1, E_2, \dots, E_n$ — mutually exclusive events whose union is all of $S$ , each with $P(E_i) > 0$ — together with any event $A$ with $P(A) > 0$ that can occur alongside one of these $E_i$ . Think of the $E_i$ as the possible "causes" or "routes" (which bag was picked, which machine made the item, whether the patient has the disease) and $A$ as the observed "effect" (the ball is red, the item is defective, the test is positive).

The formula. For each $i$ ,

P(E_i\mid A) = \frac{P(E_i)\,P(A\mid E_i)}{\displaystyle\sum_{j=1}^{n} P(E_j)\,P(A\mid E_j)}.

where: $P(E_i)$ is the prior probability of cause $E_i$ (its probability before any evidence is known), $P(A\mid E_i)$ is the likelihood of seeing the effect $A$ if $E_i$ is true, and $P(E_i\mid A)$ is the posterior probability — the updated probability of $E_i$ once it is known that $A$ has occurred. The denominator is nothing but $P(A)$ , computed exactly as in the total probability theorem: $P(A) = \sum_{j=1}^n P(E_j)P(A\mid E_j)$ .

Reading the formula as a fraction of a whole. The numerator $P(E_i)P(A\mid E_i)$ is just one term of the sum sitting in the denominator. So Bayes' theorem is really asking: of all the probability mass that flows into $A$ from every possible cause, what share came specifically from cause $E_i$ ? This "share of the total" reading is worth holding onto — it explains every worked example below.

Priors update into posteriors. Before the ball is drawn, $P(E_1)$ (probability Bag I was chosen) is just $\tfrac12$ — a guess based only on the setup, before any evidence. After the ball is drawn and seen to be red, that guess is revised into $P(E_1\mid A)$ , a sharper number that accounts for the new evidence. This updating process — prior in, evidence applied, posterior out — is the entire content of the theorem, and it is exactly the same logic used far beyond this syllabus, in areas like medical screening, spam filtering, and machine learning.

Figure

Two branches labelled cause one and cause two feeding into a shared effect node, with the Bayes formula picking out the fraction of the total contributed by cause one

Derivation Aof Bayes' theorem from conditional probability and the total probability theorem

Show that $P(E_i\mid A)$ , the probability that cause $E_i$ occurred given that effect $A$ has already been observed, must equal the ratio of $E_i$ 's own contribution to $A$ over the combined contribution of every possible cause to $A$ , using nothing but the definition of conditional probability, the multiplication theorem, and the total probability theorem already established for this partition.

Step 1. Start from the definition of conditional probability applied to $E_i$ given $A$ :

P(E_i\mid A) = \frac{P(A \cap E_i)}{P(A)}, \qquad P(A) > 0.

This is the standard definition — nothing new is assumed yet, only that both $A$ and $E_i$ are events in the same sample space with $P(A) > 0$ .

Step 2. Rewrite the numerator $P(A \cap E_i)$ using the multiplication theorem, $P(A \cap E_i) = P(E_i)P(A\mid E_i)$ , which holds because $A \cap E_i = E_i \cap A$ and $P(E_i) > 0$ :

P(E_i\mid A) = \frac{P(E_i)\,P(A\mid E_i)}{P(A)}.

At this stage the formula already resembles Bayes' theorem, but the denominator $P(A)$ is not yet expressed in terms of the partition.

Step 3. Expand $P(A)$ using the total probability theorem. Since $E_1, \dots, E_n$ partition $S$ , the event $A$ decomposes as the disjoint union $A = (A\cap E_1) \cup (A \cap E_2) \cup \dots \cup (A\cap E_n)$ , so

P(A) = \sum_{j=1}^n P(A \cap E_j) = \sum_{j=1}^n P(E_j)\,P(A\mid E_j).

Step 4. Substitute this expanded form of $P(A)$ from Step 3 into the denominator of the expression from Step 2:

P(E_i\mid A) = \frac{P(E_i)\,P(A\mid E_i)}{\displaystyle\sum_{j=1}^{n} P(E_j)\,P(A\mid E_j)}.

This is exactly Bayes' theorem, and it holds for every $i = 1, 2, \dots, n$ , so it can be applied to any one of the causes once the priors $P(E_i)$ and likelihoods $P(A\mid E_i)$ are known.

Worked example

Bag I contains 4 red and 4 black balls, Bag II contains 2 red and 6 black balls. One bag is chosen at random and a ball is drawn from it, and the ball turns out to be red. Find the probability that the ball was drawn from Bag I.

Step 1. Identify the partition and the priors. Let $E_1$ = "Bag I chosen," $E_2$ = "Bag II chosen." Since the bag is chosen at random from two bags, $P(E_1) = P(E_2) = \dfrac12$ .

Step 2. Identify the likelihoods. Let $A$ = "ball drawn is red." Bag I has 4 red out of 8 balls, and Bag II has 2 red out of 8 balls, so

P(A\mid E_1) = \frac12, \qquad P(A\mid E_2) = \frac14.

Step 3. Find $P(A)$ using the total probability theorem.

P(A) = P(E_1)P(A\mid E_1) + P(E_2)P(A\mid E_2) = \frac12\times\frac12 + \frac12\times\frac14 = \frac14+\frac18 = \frac38.

Step 4. Apply Bayes' theorem for $E_1$ .

P(E_1\mid A) = \frac{P(E_1)P(A\mid E_1)}{P(A)} = \frac{\tfrac12\times\tfrac12}{\tfrac38} = \frac{1/4}{3/8} = \frac14\times\frac83 = \boxed{\frac23}.

So, given that the drawn ball is red, the probability it came from Bag I is $\mathbf{\dfrac23}$ . This is sensible: Bag I has a higher proportion of red balls than Bag II, so observing a red ball should shift belief toward Bag I over its plain prior of $\tfrac12$ .

Competitive stretch

A diagnostic test is 99% accurate both ways: $P(A\mid E_1) = 0.99$ for a diseased person testing positive, and a false positive rate of $0.01$ for a healthy person, where only $0.5\%$ of the population actually has the disease ( $P(E_1) = 0.005$ , $P(E_2) = 0.995$ ). Then $P(A) = (0.005)(0.99) + (0.995)(0.01) = 0.00495 + 0.00995 = 0.01490$ , and Bayes' theorem gives

P(E_1\mid A) = \frac{0.00495}{0.01490} \approx 0.332.

Even with a "99% accurate" test, a person who tests positive has only about a 33.2% chance of actually having the disease — because the disease is rare, the false positives drawn from the large healthy population outnumber the true positives from the tiny diseased population. Compare this against Bag I's example: there, equal priors ( $\tfrac12$ vs $\tfrac12$ ) combined with very different likelihoods still moved the posterior a lot; here, a low prior ( $0.005$ ) pulls a seemingly strong likelihood ( $0.99$ ) down to roughly one-third. The prior always matters as much as the likelihood.

Common traps

Mistaking the test's "accuracy" for the final posterior. A high $P(A\mid E_1)$ does not make $P(E_1\mid A)$ equally high — a small prior $P(E_1)$ can pull the posterior down sharply, as in the diagnostic-test stretch example above.
Forgetting to compute $P(A)$ via the total probability theorem first. The denominator of Bayes' theorem is rarely given directly; it must be built by summing $P(E_j)P(A\mid E_j)$ over every $j$ before the formula can be applied.
Dropping partition events from the denominator sum. With three or more possible causes, every single $E_j$ must appear in the sum — not just the one whose posterior is being computed.
Swapping prior and posterior. $P(E_i)$ is the probability before evidence $A$ is known; $P(E_i\mid A)$ is the probability after $A$ is known. Writing one where the other belongs silently breaks the whole calculation.

Ask the tutor: Want to see why a 99% accurate test can still leave a patient only about 33% likely to have the disease, or need help setting up the priors and likelihoods for your own Bayes' theorem word problem?

Why is Bayes theorem called reversing the conditional probability

Ordinary conditional probability, like $P(A\mid E_i)$ , answers a forward question: given the cause, what is the chance of the effect. Bayes' theorem answers the reverse question: given that the effect has already happened, what is the chance it came from a particular cause. It literally swaps which event sits after the "given" bar, which is why textbooks describe it as running conditional probability backwards.

Do I need to memorise the total probability theorem separately from Bayes theorem

Yes, and they are closely linked, not identical. The total probability theorem computes only the denominator, $P(A) = \sum_j P(E_j)P(A\mid E_j)$ . Bayes' theorem reuses that exact sum as its denominator and adds the specific numerator $P(E_i)P(A\mid E_i)$ on top, to isolate one cause's share. In practice, solving a Bayes' theorem problem almost always means computing the total probability theorem's answer first, then dividing.

What happens if there are three or more possible causes instead of two

Nothing changes structurally — the formula still reads $P(E_i\mid A) = \dfrac{P(E_i)P(A\mid E_i)}{\sum_j P(E_j)P(A\mid E_j)}$ , only now the sum in the denominator runs over three, four, or however many partition events exist. The main danger is arithmetic: every single $E_j$ term must be included in that sum, or the denominator will be too small and every posterior will come out inflated.

Bayes theorem

The core idea

Worked example

Competitive stretch

Common traps

Before this

Where this leads