Conditional Probability, Bayes' Theorem, Independence

Conditional probability rescales the probability of an event $A$ to the new sample space defined by a conditioning event $B$, dividing the overlap $P(A\cap B)$ by $P(B)$. It matters because almost all real inference is conditional: we never observe a system in total ignorance but always relative to some evidence. Bayes' theorem is the algebraic engine that reverses the direction of conditioning, turning $P(\text{evidence}\mid \text{hypothesis})$ — which models give us — into $P(\text{hypothesis}\mid \text{evidence})$ — which we actually want. Independence captures informational irrelevance and underpins everything from product measures to the law of large numbers. The surprising fact most students stumble over: a test that is 99% accurate for a rare disease can still mean a positive result is more likely false than true — the base rate dominates.

The Core Insight

Conditioning on $B$ is nothing more than renormalizing the probability measure to live inside $B$. The whole machinery — Bayes, independence, the chain rule, total probability — follows mechanically from the single defining ratio $P(A\mid B)=P(A\cap B)/P(B)$. Bayes' theorem is just this definition written twice and equated through the shared joint probability $P(A\cap B)$.

Formal Statement

Conditional Probability

Let $(\Omega ,\mathcal{F},P)$ be a probability space and let $A,B\in \mathcal{F}$ with $P(B)>0$. The conditional probability of $A$ given $B$ is

Symbols: $\Omega$ is the sample space; $\mathcal{F}$ the $\sigma$-algebra of events; $P$ a probability measure; $A\cap B$ the joint event "both $A$ and $B$"; $P(B)$ the prior weight of the conditioning event, required positive so the ratio is defined.

The Multiplication (Chain) Rule

Rearranging [1] gives the joint probability as a product:

Generalized to $n$ events:

Independence

Events $A$ and $B$ are independent iff

A family $\{A_i{\}}_{i\in I}$ is mutually independent iff for every finite subset $S\subseteq I$,

Law of Total Probability

If $\{B_1,\dots ,B_n\}$ is a partition of $\Omega$ (disjoint, exhaustive, each $P(B_k)>0$), then for any event $A$,

Bayes' Theorem

For events $A,B$ with $P(A),P(B)>0$,

With a partition $\{B_1,\dots ,B_n\}$, expanding the denominator via [6]:

Here $P(B_k)$ is the prior, $P(A\mid B_k)$ the likelihood, $P(B_k\mid A)$ the posterior, and the denominator the evidence (marginal likelihood / normalizing constant).

Why It Holds / Derivation

Bayes' Theorem from the Multiplication Rule

1. By definition [1], assuming $P(A)>0$ and $P(B)>0$:

1. The joint $P(A\cap B)$ is symmetric — it is the same number in both expressions. Solve the second for it:

1. Substitute into the first:

Law of Total Probability

1. Since $\{B_k\}$ partitions $\Omega$, the sets $A\cap B_k$ are pairwise disjoint and their union is $A$:

1. By (countable) additivity of $P$ on disjoint sets:

1. Apply the multiplication rule [2] to each term: $P(A\cap B_k)=P(A\mid B_k)P(B_k)$, yielding [6]. $■$

Thomas Bayes | c. 1701–1761

A Presbyterian minister and Fellow of the Royal Society, Bayes left an unpublished essay solving the "inverse probability" problem: given an observed number of successes, what can be said about the underlying chance? His friend Richard Price edited and read it posthumously to the Royal Society in 1763 ("An Essay towards solving a Problem in the Doctrine of Chances"). Bayes treated only the uniform-prior case for a binomial parameter; the general theorem and its systematic use were developed independently and far more powerfully by Pierre-Simon Laplace from 1774 onward. The modern measure-theoretic foundation came from Kolmogorov in 1933.

Building Intuition

Picture probability as area on a unit-area dartboard $\Omega$. Event $B$ is a region; conditioning on $B$ means throwing away the rest of the board and asking what fraction of the remaining region $B$ also lies in $A$. That is why we divide by $P(B)$: we are recomputing area as a fraction of the new, smaller universe.

Conditional probability as renormalized area: $P(A\mid B)$ is the shaded overlap as a fraction of region $B$

Worked Examples

Medical Test and the Base-Rate Effect

1. Total probability of a positive test [6]:

1. Bayes' theorem [7]:

Mutual vs. Pairwise Independence

1. Check pairwise. $P(A\cap B)=P(HH)=\frac{1}{4}=P(A)P(B).$ ✓

1. Check mutual independence via [5] for $S=\{A,B,C\}$:

1. Since $\frac{1}{4}\ne \frac{1}{8}$, the triple product rule fails.

Common Misconceptions

Six Frequent Errors

1. Confusing the direction: $P(A\mid B)\ne P(B\mid A)$ in general (the "prosecutor's fallacy"). They differ by the factor $P(B)/P(A)$.
2. Ignoring the base rate: treating $P(D\mid +)$ as if it equals the test's sensitivity $P(+\mid D)$.
3. Confusing independent with mutually exclusive: if $A,B$ are disjoint with positive probability, they are dependent, since $P(A\mid B)=0\ne P(A)$.
4. Assuming pairwise independence implies mutual independence (see counterexample above).
5. Conditioning on a null event: $P(\cdot \mid B)$ is undefined when $P(B)=0$ (needs regular conditional probability / measure theory to repair).
6. Forgetting that independence is relative to the measure $P$: the same events can be independent under one $P$ and dependent under another.

Mutually Exclusive Is the Opposite of Independent

Roll one fair die. Let $A=\{$roll is $\le 3\}$ and $B=\{$roll is $\ge 5\}$. These are mutually exclusive: $A\cap B=\varnothing$, so $P(A\cap B)=0$. Yet $P(A)P(B)=\frac{1}{2}\cdot \frac{1}{3}=\frac{1}{6}\ne 0$. Knowing $B$ occurred makes $A$ impossible — maximal dependence. Students who hear "$A$ and $B$ can't both happen" and conclude "they don't influence each other" have it exactly backwards.

Connections

Conditional probability is the gateway to nearly all of probability theory. It specializes to Conditional Expectation when we condition random variables rather than events, and to the Conditional Density $f_{X\mid Y}(x\mid y)=f_{X,Y}(x,y)/f_Y(y)$ in the continuous case. Independence of events generalizes to independence of Random Variables (factorization of joint distributions) and underlies the Law of Large Numbers and Central Limit Theorem, both of which assume i.i.d. samples. Bayes' theorem is the foundation of Bayesian Inference, the engine of Markov Chains (via conditional transition rules), and the backbone of Naive Bayes Classifiers. The chain rule [3] structures Probabilistic Graphical Models and Bayesian networks.

Real-World Applications

1. Medical diagnostics: computing post-test disease probability from sensitivity, specificity, and prevalence — exactly the base-rate example. Used in screening guidelines (mammography, HIV testing).
2. Spam filtering: the Naive Bayes classifier assigns $P(\text{spam}\mid \text{words})$ by treating word occurrences as conditionally independent given the class.
3. Forensic science & DNA matching: likelihood ratios update the probability of a suspect's guilt; misuse causes the "prosecutor's fallacy" in real trials (e.g., the Sally Clark case).
4. Search and rescue: the U.S. Coast Guard's SAROPS system uses Bayesian updating of a probability map as search areas are cleared (famously located the 1968 lost submarine USS Scorpion via Bayesian search by John Craven).
5. Machine learning: Bayesian inference, Kalman filters (recursive Bayes for linear-Gaussian systems), and Hidden Markov Models all rest on conditional/Bayesian updating.
6. Insurance and credit scoring: updating risk estimates conditional on policyholder or applicant features.
7. A/B testing and clinical trials: Bayesian sequential analysis updates the posterior on a treatment effect as data accrue.

Historical Development

• 1654 — Pascal and Fermat correspond on the problem of points, implicitly using conditional reasoning about unfinished games.
• 1713 — Jacob Bernoulli's Ars Conjectandi (posthumous) states the law of large numbers, framing how evidence accumulates.
• 1763 — Richard Price reads Thomas Bayes' essay to the Royal Society, solving inverse probability for the binomial with a uniform prior.
• 1774 — Laplace independently formulates the general principle of inverse probability ("Mémoire sur la probabilité des causes").
• 1812 — Laplace's Théorie analytique des probabilités systematizes Bayesian methods and applies them to astronomy and demography.
• 1900s–1920s — Frequentists (Fisher, Neyman, Pearson) challenge "inverse probability," sidelining Bayesian methods for decades.
• 1933 — Kolmogorov's Grundbegriffe der Wahrscheinlichkeitsrechnung gives the axiomatic, measure-theoretic definition of conditional probability and independence.
• 1950s–60s — Revival of Bayesian statistics through Jeffreys, Savage, and de Finetti (subjective probability and exchangeability).
• 1990s–present — Computational power (MCMC, Gibbs sampling) makes Bayesian inference practical, fueling modern statistics and machine learning.

Subtleties & Limits

See Also