Conditional Probability and Independence

Conditional probability measures the chance of an event

A

given that another event

B

has occurred, written

P (A ∣ B) = P (A \cap B) / P (B)

. Two events are independent when knowing one tells you nothing about the other, i.e.

P (A \cap B) = P (A) P (B)

. Used everywhere from medical testing to gambling to machine learning. Formalized by Bayes, Laplace, and Kolmogorov between the 1760s and 1933.

Conditional probability refines our beliefs about an event in light of partial information: once we know that $B$ has happened, the sample space effectively shrinks to $B$ , and we re-weight the chance of $A$ accordingly. This idea is central to all of probability because real reasoning almost always proceeds under some condition or given some evidence. Independence is the special, clean case where conditioning changes nothing — the two events live in separate informational worlds. The surprising fact is that independence is a numerical coincidence about probabilities, not a statement about physical causation: events can be statistically independent while being causally linked, and dependent while being physically unrelated. Mastering these two notions unlocks Bayes' theorem, the law of total probability, and the entire machinery of statistical inference.

The Core Insight

Conditioning is renormalization. When you learn that

B

occurred, you discard all outcomes outside

B

and rescale the probabilities of what remains so they sum to 1 again. The event

A

is then judged only by the portion of it that lives inside

B

. Independence is the dividing line: if the conditional probability

P (A ∣ B)

equals the unconditional

P (A)

, the information in

B

is irrelevant to

A

Formal Statement

Conditional Probability

Let $(Ω, F, P)$ be a probability space and let $A, B \subseteq Ω$ be events with $P (B) > 0$ . The conditional probability of $A$ given $B$ is

P (A ∣ B) = \frac{P ( A \cap B )}{P ( B )} .

⟳Restrict the sample space to

B

and renormalize by dividing by

P (B)

so that

P (B ∣ B) = 1

Symbols:

$Ω$ — the sample space (all possible outcomes).
$A, B$ — events (subsets of $Ω$ ).
$A \cap B$ — the event that both $A$ and $B$ occur.
$P (\cdot)$ — the probability measure, assigning numbers in $[0, 1]$ .
$P (A ∣ B)$ — probability of $A$ assuming $B$ is known to have occurred.

The quantity is undefined when $P (B) = 0$ .

Independence of Two Events

Events $A$ and $B$ are independent if and only if

P (A \cap B) = P (A) P (B) .

⟳Substitute

P (A ∣ B) = P (A)

into equation [1] and clear the denominator; the symmetric product form drops the requirement

P (B) > 0

Equivalently, when $P (B) > 0$ , independence means $P (A ∣ B) = P (A)$ , and when $P (A) > 0$ , it means $P (B ∣ A) = P (B)$ .

Multiplication Rule and Chain Rule

Rearranging [1] gives the multiplication rule:

P (A \cap B) = P (B) P (A ∣ B) = P (A) P (B ∣ A) .

⟳Multiply both sides of equation [1] by

P (B)

; the second form follows by symmetry of

A \cap B

For a sequence of events $A_{1}, \dots, A_{n}$ this generalizes to the chain rule:

P (A_{1} \cap \dots \cap A_{n}) = P (A_{1}) P (A_{2} ∣ A_{1}) P (A_{3} ∣ A_{1} \cap A_{2}) \dots P (A_{n} ∣ A_{1} \cap \dots \cap A_{n - 1}) .

⟳Apply equation [3] repeatedly, conditioning each new event on the intersection of all preceding ones.

Law of Total Probability and Bayes' Theorem

If $B_{1}, \dots, B_{k}$ partition $Ω$ (mutually exclusive, exhaustive, each of positive probability), then for any event $A$ :

P (A) = i = 1 \sum k P (B_{i}) P (A ∣ B_{i}) .

⟳Write

A = ⋃_{i} (A \cap B_{i})

as a disjoint union, apply additivity, then equation [3] to each term.

Combining [3] and [5] yields Bayes' theorem:

P (B_{j} ∣ A) = \frac{P ( B _{j} ) P ( A ∣ B _{j} )}{\sum _{i = 1}^{k} P ( B _{i} ) P ( A ∣ B _{i} )} .

⟳Write

P (B_{j} ∣ A) = P (A \cap B_{j}) / P (A)

, expand the numerator with equation [3] and the denominator with equation [5].

Why It Holds / Derivation

The Conditional Measure Is a Genuine Probability

We verify that for fixed $B$ with $P (B) > 0$ , the map $Q (A) := P (A ∣ B)$ satisfies Kolmogorov's axioms.

Non-negativity. Since $P (A \cap B) \geq 0$ and $P (B) > 0$ , the ratio $Q (A) = P (A \cap B) / P (B) \geq 0$ .

Normalization. Take $A = Ω$ . Then $Ω \cap B = B$ , so $Q (Ω) = P (B) / P (B) = 1$ .

Countable additivity. Let $A_{1}, A_{2}, \dots$ be pairwise disjoint. The sets $A_{n} \cap B$ are also pairwise disjoint, so

Q (n ⋃ A_{n}) = \frac{P ( ⋃ _{n} ( A _{n} \cap B ) )}{P ( B )} = \frac{\sum _{n} P ( A _{n} \cap B )}{P ( B )} = n \sum Q (A_{n}) .

Conclusion. $Q$ is a probability measure on $Ω$ concentrated on $B$ . Therefore conditioning is internally consistent: all theorems of probability apply unchanged inside the conditioned world. $■$

Equivalence of the Two Definitions of Independence

Suppose $P (B) > 0$ .

If $P (A \cap B) = P (A) P (B)$ (equation [2]), divide both sides by $P (B)$ : $P (A \cap B) / P (B) = P (A)$ , i.e. $P (A ∣ B) = P (A)$ .

Conversely, if $P (A ∣ B) = P (A)$ , multiply by $P (B)$ : $P (A \cap B) = P (A) P (B)$ .

The product form [2] is preferred because it is symmetric in $A$ and $B$ and remains meaningful even when $P (A)$ or $P (B)$ equals zero — in which case both sides are $0$ and independence holds trivially. $■$

Thomas Bayes | c. 1701–1761

An English Presbyterian minister and amateur mathematician, Bayes never published his famous result during his lifetime. His essay An Essay towards Solving a Problem in the Doctrine of Chances was communicated to the Royal Society by Richard Price in 1763, two years after Bayes' death. It addressed an inverse problem: given observed successes, infer the underlying probability — the seed of Bayesian Inference. Pierre-Simon Laplace independently rediscovered and vastly generalized the idea, applying it to celestial mechanics, demography, and jurisprudence, and it is largely through Laplace that "Bayesian" reasoning entered scientific practice.

Building Intuition

Picture the sample space as a unit square of area 1. An event is a region; its probability is its area. Now learn that $B$ occurred: erase everything outside $B$ and blow up the region $B$ until it again has area 1. The new area of $A$ inside this magnified picture is exactly $P (A ∣ B)$ — the fraction of $B$ that overlaps $A$ . Independence means $A$ slices through $B$ in the same proportion that it slices through the whole square. If $A$ takes up 30% of $Ω$ , then under independence it also takes up 30% of $B$ , 30% of $B^{c}$ , and 30% of every other region — its "density" is uniform with respect to $B$ . Dependence simply means $A$ is concentrated more in some parts of $Ω$ than others, so knowing where you are (inside $B$ or not) shifts the odds.

Conditioning as renormalization: the overlap of A with B becomes the whole new sample space

Worked Examples

Medical Test — Bayes in Action

Given: A disease affects 1% of a population:

P (D) = 0.01

. A test has sensitivity

P (+ ∣ D) = 0.99

and false-positive rate

P (+ ∣ D^{c}) = 0.05

. A person tests positive. What is

P (D ∣ +)

? Step 1 — Total probability of a positive (eq. [5]):

P (+) = P (D) P (+ ∣ D) + P (D^{c}) P (+ ∣ D^{c}) = 0.01 (0.99) + 0.99 (0.05) = 0.0099 + 0.0495 = 0.0594.

Step 2 — Apply Bayes (eq. [6]):

P (D ∣ +) = \frac{P ( D ) P ( + ∣ D )}{P ( + )} = \frac{0.0099}{0.0594} .

Step 3 — Compute:

0.0099/0.0594 \approx 0.1667

P (D ∣ +) \approx 0.167 = 16.7%

Despite a "99% accurate" test, a positive result means only about a 1-in-6 chance of disease, because healthy people vastly outnumber sick ones — the base-rate dominates.

Drawing Cards Without Replacement — Chain Rule

Given: Draw two cards from a standard 52-card deck without replacement. Find the probability both are aces, then check independence. Step 1 — First ace:

P (A_{1}) = \frac{4}{52} = \frac{1}{13}

. Step 2 — Second ace given first (eq. [3]):

P (A_{2} ∣ A_{1}) = \frac{3}{51} = \frac{1}{17}

. Step 3 — Multiply (chain rule, eq. [4]):

P (A_{1} \cap A_{2}) = \frac{1}{13} \cdot \frac{1}{17} = \frac{1}{221} \approx 0.00452.

P (both aces) = \frac{1}{221} \approx 0.452%

Independence check: If the draws were independent we'd need

P (A_{1} \cap A_{2}) = P (A_{1}) P (A_{2}) = (\frac{1}{13})^{2} = \frac{1}{169} \neq = \frac{1}{221}

. Since

\frac{1}{221} < \frac{1}{169}

, the events are dependent — removing one ace makes a second slightly less likely.

Common Misconceptions

Independence ≠ Mutual Exclusivity

These are opposite ideas, frequently confused. Mutually exclusive events cannot both happen: $P (A \cap B) = 0$ . Independent events satisfy $P (A \cap B) = P (A) P (B)$ . If $A$ and $B$ both have positive probability and are mutually exclusive, then $P (A \cap B) = 0 \neq = P (A) P (B) > 0$ , so they are actually strongly dependent — knowing $A$ occurred guarantees $B$ did not. Never assume "they don't overlap" means "they're independent."

The Conditioning Bar Is Not Symmetric

P (A ∣ B)

and

P (B ∣ A)

are generally different numbers. Confusing them is the prosecutor's fallacy: the probability of innocence given the evidence is not the probability of the evidence given innocence. In the medical example,

P (+ ∣ D) = 0.99

but

P (D ∣ +) = 0.167

— a factor-of-6 difference. Bayes' theorem (eq. [6]) is precisely the tool that converts one direction into the other.

Pairwise Independence Does Not Imply Mutual Independence

Toss two fair coins. Define:

$A$ = first coin is Heads,
$B$ = second coin is Heads,
$C$ = the two coins match (both H or both T).

Each has probability $\frac{1}{2}$ . Check pairs: $P (A \cap C) = P (HH) = \frac{1}{4} = P (A) P (C)$ , and similarly for the other two pairs — so the events are pairwise independent. But $P (A \cap B \cap C) = P (HH) = \frac{1}{4}$ , while $P (A) P (B) P (C) = \frac{1}{8}$ . Since $\frac{1}{4} \neq = \frac{1}{8}$ , the three are not mutually independent: knowing $A$ and $B$ determines $C$ completely. Mutual independence requires the product rule to hold for every subset of events, not just pairs.

Connections

Conditional probability is the gateway to the rest of probability theory. The renormalization viewpoint extends to Conditional Expectation, where we condition a random variable rather than just an event. Independence of events generalizes to Independence of Random Variables and underlies the Law of Large Numbers and the Central Limit Theorem, both of which assume independent (or weakly dependent) samples. Bayes' theorem [eq. 6] is the engine of Bayesian Inference, while the law of total probability [eq. 5] is the discrete cousin of marginalization in Joint Probability Distributions. The notion of conditioning also seeds Markov Chains, where the future depends on the present but not the past.

Real-World Applications

Medical diagnostics. Interpreting test results requires Bayes' theorem to combine sensitivity, specificity, and disease prevalence (base rate), as in the worked example.
Spam filtering. Naïve Bayes classifiers compute $P (spam ∣ words)$ by treating word occurrences as conditionally independent given the class.
Insurance and actuarial science. Premiums depend on conditional risk: $P (claim ∣ age, history, region)$ .
Search and rescue / tracking. Bayesian updating refines the probable location of a lost object as new search data arrives; used to find the wreck of Air France 447 (2011).
Genetics and pedigree analysis. The probability a child carries a recessive allele is computed by conditioning on parental genotypes.
Machine learning. Hidden Markov models, Bayesian networks, and probabilistic graphical models are built entirely from conditional probabilities and (conditional) independence assumptions.
Reliability engineering. System failure probability uses the chain rule over component failures, accounting for dependent failure modes.

Historical Development

1654 — Blaise Pascal and Pierre de Fermat correspond about gambling problems, founding mathematical probability and implicitly using conditional reasoning.
1657 — Christiaan Huygens publishes De ratiociniis in ludo aleae, the first printed probability treatise, introducing expectation.
1713 — Jacob Bernoulli's posthumous Ars Conjectandi proves the first law of large numbers, framing long-run frequency.
1763 — Thomas Bayes' essay, communicated by Richard Price, gives the inverse-probability rule that bears his name.
1774–1812 — Pierre-Simon Laplace develops and generalizes Bayesian methods, stating the principle of inverse probability and applying it across the sciences in Théorie analytique des probabilités.
1837 — Siméon Denis Poisson refines the law of large numbers and studies dependent trials.
1900s — Andrey Markov introduces dependent sequences (Markov chains), demonstrating rich behavior beyond independence.
1933 — Andrey Kolmogorov's Grundbegriffe der Wahrscheinlichkeitsrechnung axiomatizes probability measure-theoretically, giving conditional probability and independence their modern rigorous definitions.
1950s–present — Conditional independence becomes the structural backbone of Bayesian networks (Judea Pearl, 1980s) and modern statistical learning.

Subtleties & Limits

The definition $P (A ∣ B) = P (A \cap B) / P (B)$ breaks down when $P (B) = 0$ — yet conditioning on probability-zero events is unavoidable when the conditioning variable is continuous (e.g. "given that $X = 3.0$ exactly"). Resolving this requires measure-theoretic conditional expectation and the Radon–Nikodym theorem, where the conditional probability becomes a function defined only up to sets of measure zero. The Borel–Kolmogorov paradox shows that conditioning on the same zero-probability event ("a point on a sphere") can yield different answers depending on which limiting family of positive-probability sets you use — so naïve continuous conditioning is genuinely ambiguous. Philosophically, debates persist between frequentist and Bayesian interpretations of what $P (A ∣ B)$ even means: a long-run frequency, or a rational degree of belief. These foundational questions remain live in statistics and the philosophy of probability.