The Coalescent: An Introduction
Take ten copies of a single gene, sampled from ten unrelated humans, and sequence them. Somewhere in the human past, every one of those ten lineages traces back to a single ancestral copy. It was one molecule of DNA that existed in one person, at some point. All ten are descended from it. When did that ancestor live? How long, on average, do we expect to wait until two random lineages merge into their shared ancestor? These are not idle questions. They are exactly the questions that let biologists date the origin of HIV subtypes. They also help reconstruct human migration out of Africa. And they distinguish a gene under natural selection from one drifting neutrally. The coalescent is the mathematical framework built to answer them. Rather than modeling a population moving forward in time — the traditional approach of population genetics, tracking allele frequencies generation after generation — the coalescent runs the film backward. It asks not "what will happen to these genes?" but "where did these genes come from?" This reversal is not just a change in perspective. It is a profound simplification. Forward-time models must simulate an entire population of possibly millions of individuals. They must do so across countless generations. Backward-time coalescent models need only track the sample actually in hand. That might be ten sequences, a hundred genomes, whatever was collected. They ask how their ancestries merge, or coalesce, as one travels into the past. This matters because nearly all of modern population genomics is genealogical inference in disguise. A paper might report an "effective population size of 10,000." It might say "this mutation arose 5,000 years ago." Or it might note "this gene shows a signature of a selective sweep." Those numbers usually come from fitting a coalescent model to observed patterns of genetic variation. The coalescent connects Effective Population Size and Genetic Drift to the branching tree structure that generated observable DNA sequences. These concepts are usually described forward in time. That branching structure actually generated the DNA sequences we can measure today. It is the bridge between an abstract population-genetic parameter and an actual, testable, quantitative prediction about a gene tree. This article builds the coalescent from first principles. It covers the simple urn-model logic behind it, and the exact probability distributions it generates. It also covers the effect of adding mutation. And it covers the classical population-genetic quantities, like Watterson's estimator and Tajima's D, that emerge from it. It closes with how the framework is empirically validated. It also shows how the framework sets the stage for detecting selection in real sequence data.
- The coalescent models gene genealogies backward in time, asking when and how sampled lineages share common ancestors — the reverse of classical forward-in-time population genetics.
- Under the standard neutral (Wright-Fisher) model, the waiting time for two lineages to coalesce is geometric, approximated by an exponential distribution with rate 1 in units of N generations.
- With n sampled lineages, the total genealogy is built from n−1 coalescent events; the expected time to the most recent common ancestor (MRCA) converges to 2(1 − 1/n) in units of N generations, approaching 2N generations as n grows.
- Adding mutation (rate θ = 4Nμ) turns a genealogy into predicted sequence variation: this underlies estimators like Watterson's θ and Tajima's D, used to detect deviations from neutrality.
- The coalescent was formalized by John Kingman (1982), unifying earlier population-genetic diffusion theory into an elegant, tractable stochastic process.
- It is now the computational backbone of phylogeography, viral epidemiology (e.g., dating HIV/SARS-CoV-2 origins), human demographic history inference, and genome-wide scans for selection — foreshadowing Detecting Selection in Sequences.
Instead of tracking allele frequencies forward through generations, the coalescent traces sampled gene lineages backward until they merge into shared ancestors — turning population size, drift, and demographic history into precise, testable predictions about the shape and depth of a gene tree.
The core idea
Classical population genetics, as built by Fisher, Wright, and Haldane, is a forward-in-time enterprise: start with an allele frequency in generation 0, apply mutation, selection, and drift, and ask what the frequency distribution looks like generations later. This works well for predicting the fate of a population, but it is computationally wasteful when the real goal is to explain a dataset already in hand — a handful of DNA sequences sampled today. Simulating an entire population of size N (often thousands to millions) for thousands of generations, just to look at the ten sequences a researcher actually sequenced, throws away enormous amounts of irrelevant information about the other N − 10 individuals. The coalescent, developed rigorously by John Kingman in 1982 (with important precursors in work by Motoo Kimura, Warren Ewens, and others on gene-genealogy theory in the late 1970s), inverts the direction of the analysis. It is a stochastic process describing the genealogy of a sample — the tree of ancestor-descendant relationships connecting the sampled gene copies — traced backward from the present into the past. Two lineages coalesce when we trace them back to the single ancestral DNA copy from which both descended. Because a sample of n genes must ultimately trace back through a series of pairwise mergers to one ancestral lineage (the sample's most recent common ancestor, or MRCA), the coalescent describes a random, branching tree with exactly n − 1 internal (coalescence) nodes. The deep insight is that under the standard neutral model, this tree — its branch lengths and topology — has an exactly calculable probability distribution that depends on only one parameter: the effective population size, N (or, for diploids, often written as 2N or 4N depending on ploidy and mating system). Once that genealogical tree is specified, adding neutral mutations along its branches (via a Poisson process, per The Neutral Theory of Molecular Evolution) generates a precise prediction for the amount and pattern of genetic variation expected in the sample. This makes the coalescent simultaneously (1) a null model against which real data can be tested for deviations (selection, population growth, population structure) and (2) an inference engine for estimating population-genetic parameters directly from sequence data.
Mechanism
Step 1 — the Wright-Fisher ancestry process. Consider the classic Wright-Fisher model: a population of N diploid individuals (2N gene copies), non-overlapping generations, random mating, no selection. Each gene copy in generation t chooses its parent copy uniformly at random from the 2N copies in generation t − 1. Now flip the arrow: instead of asking who a copy's descendants will be, ask, for two randomly sampled gene copies in the present generation, what is the probability that they had the same parent copy one generation back? Since each copy picks a parent uniformly among 2N choices, the probability the second copy picked the same parent as the first is exactly 1/(2N). This is the fundamental building block: a single-generation coalescence probability of 1/(2N).
Step 2 — waiting time to coalescence for a pair. The probability that two lineages do not coalesce in one generation is 1 − 1/(2N). The number of generations until they do coalesce follows a geometric distribution with success probability 1/(2N):
For large N, this geometric distribution is extremely well approximated by an exponential distribution. Rescaling time in units of 2N generations (defining τ = t/2N), the waiting time becomes Exponential(1): mean 1, i.e., a pair of lineages takes on average 2N generations to coalesce. This exponential approximation is the mathematical heart of Kingman's coalescent, and it becomes exact in the limit N → ∞ with time rescaled — the same diffusion-limit logic used elsewhere in population genetics for drift.
Step 3 — many lineages: the rate of coalescence increases as lineages merge. Now sample n gene copies (n ≪ N). Looking one generation back, any pair among the n lineages might coalesce. There are C(n,2) = n(*n−1)/2 possible pairs, each with per-generation coalescence probability ≈1/(2N*) (ignoring the negligible chance of triple mergers). So the total rate of some coalescence event, when there are k lineages remaining, is approximately:
In rescaled time units of 2N generations, the waiting time T_k until the next coalescence (from k lineages down to k−1) is exponentially distributed with rate C(k,2):
This is the key quantitative fact: coalescence is fast when many lineages remain (lots of pairs available to merge) and slow when few remain — critically, the last two lineages (k = 2) have the longest expected waiting time of all, E[T₂] = 1 (in units of 2N generations), which dominates the total depth of the tree.
Step 4 — building the full genealogy. Starting from n sampled lineages, the process proceeds: wait T_n (rate C(n,2)) until the first coalescence, drops to n−1 lineages; wait T_{n-1} (rate C(n-1,2)); and so on down to T₂ (rate C(2,2)=1), after which a single lineage remains — the MRCA of the whole sample. The total tree height (time back to the sample's MRCA) is:
in units of 2N generations. Notice this converges to 2 (i.e., 4N generations in the original scale, or often written 2N for haploid/2N for diploid conventions depending on the source) as n → ∞ — adding more samples yields rapidly diminishing returns in how much deeper the tree gets, because the earliest coalescences (with many lineages) happen almost instantly, while the final pairwise merger dominates the total time regardless of sample size.
Step 5 — adding mutation. Mutations are then sprinkled onto the genealogy's branches as a Poisson process with rate μ per site per generation. The key composite parameter is:
(for diploids; 4Nμ is the expected number of new mutations per generation scaled appropriately so that θ equals the expected pairwise nucleotide diversity). Because branch lengths in the coalescent are proportional to N, and mutation rate is μ, only the product *N*μ is estimable from data — a fundamental identifiability constraint that recurs throughout molecular evolution.
The evidence
The coalescent is a piece of mathematics, so its "proof" is not a single wet-lab experiment but a derivation validated by simulation, consistency with known population-genetic results, and repeated empirical confirmation against real sequence data. Three lines of evidence matter.
1. Kingman's derivation and its consistency with diffusion theory (1982). John Kingman showed that the discrete-generation Wright-Fisher ancestral process, in the limit of large N with time measured in units of N (or 2N) generations, converges to a continuous-time Markov process — now called the n-coalescent — with exactly the exponential waiting times and pairwise-merging structure described above. Crucially, this backward-time process reproduces every classical forward-time result already established by Fisher, Wright, Malécot, and Kimura using diffusion approximations: for instance, it reproduces Kimura's diffusion formula for the probability of fixation of a neutral allele, and it reproduces Watterson's (1975) formula for the expected number of segregating sites in a sample under neutrality. That two independently derived mathematical routes — forward diffusion equations and backward coalescent genealogies — arrive at identical answers is a strong internal consistency check, analogous to two independent proofs of the same theorem.
2. Watterson's estimator and Tajima's test as empirical validation tools. Once the coalescent gives exact predictions for how much genetic variation should exist under neutrality, those predictions can be checked against real DNA polymorphism data. Two classical statistics did exactly this:
- Watterson's θ (1975), based on the number of segregating (polymorphic) sites S in a sample of n sequences:
This estimator is derived directly from coalescent expectations of branch lengths — the expected total tree length (sum of all branches) is a_n in coalescent time units, so S/*a_n estimates θ = 4N*μ.
- Tajima's D (1989), comparing Watterson's θ to the average pairwise nucleotide diversity π (mean number of differences between all pairs of sequences). Under strict neutrality with constant population size, both statistics estimate the same θ, so D ≈ 0. Deviations — D significantly negative (excess rare variants, consistent with a recent selective sweep or population expansion) or significantly positive (excess intermediate-frequency variants, consistent with balancing selection or population bottleneck) — are measured relative to the coalescent's null expectation. This comparison is only meaningful because the coalescent supplies an exact, quantitative null distribution against which real data can be tested — this is precisely why the framework was needed at all, and it is the direct ancestor of modern genome scans for selection.
3. Direct empirical confirmation using pedigreed and experimentally evolved populations. Because coalescent predictions are quantitative (not just qualitative), they have been tested directly. In laboratory populations of Drosophila and E. coli with known, controlled population sizes and generation numbers, sequenced lineages show coalescence patterns (mean time to common ancestor, variance in that time, star-like vs. deep tree shapes under population expansion vs. constant size) matching coalescent predictions within the expected statistical variance. In human mitochondrial DNA and Y-chromosome studies, the distribution of pairwise sequence differences among global samples closely matches the star-shaped, shallow coalescent trees predicted for a population that expanded rapidly from a small ancestral size — a pattern (mismatch distribution) explicitly derived from coalescent theory by Rogers and Harpending (1992), and now a standard tool for reconstructing demographic history (e.g., dating the human population expansion out of Africa to roughly 60,000–100,000 years ago, consistent with archaeological and fossil evidence). The fact that independently derived demographic dates — from archaeology, from fossil morphology, and from coalescent-based genetic dating — converge is the strongest form of validation available for a historical, unrepeatable process.
Worked examples
Example 1 — expected coalescence time and total tree length for a modest human sample. Suppose researchers sample n = 6 gene copies from a population with effective size N = 10,000 (diploid, so 2N = 20,000 gene copies), and the mutation rate per site per generation is μ = 1 × 10⁻⁸. Step 1: θ = 4Nμ = 4 × 10,000 × 10⁻⁸ = 4 × 10⁻⁴ per site. Step 2: Expected time to MRCA for n = 6, in units of 2N generations:
In actual generations: 1.667 × 2N = 1.667 × 20,000 = 33,340 generations. Step 3: Expected total tree length (sum of all branches, needed for expected number of segregating sites), in coalescent units:
(this sum, 2a_{n-1} with a_{n-1} = Σ1/i for i=1 to n−1, is exactly Watterson's a_n term for n − 1 doubled — consistent with the formula above). Converting to generations: 4.567 × 20,000 = 91,340 generations of total branch length across the tree. Step 4: Expected number of segregating sites at a locus of length 1,000 bp:
More directly, using Watterson's formula with a_n = Σ_{i=1}^{5} 1/i = 2.2833 for n=6:
Cleaner: per-site expected segregating sites = θ × a_n = 4×10⁻⁴ × 2.2833 = 9.13×10⁻⁴. Over 1,000 bp: E[S] ≈ 0.91 segregating sites — i.e., roughly a 1-in-1 chance of seeing a single polymorphic site in this short sequence across six sampled humans, a realistic order of magnitude for human nuclear loci of this size and diversity level.
Example 2 — reading a genealogy for signs of selection (a Tajima's D scenario). A researcher sequences a 2,000 bp region from n = 20 individuals of an endangered fish population and finds:
- Number of segregating sites, S = 18
- Average pairwise nucleotide differences (π, summed over the locus) = 4.2
First compute Watterson's θ estimate:
Compare to π = 4.2. Since π < θ_W, Tajima's D will be negative. Biological interpretation using the coalescent's null expectations: a negative D means an excess of low-frequency (rare) variants relative to the neutral coalescent expectation for a constant-sized population. Two coalescent-based explanations compete: 1. Recent population expansion or bottleneck-then-expansion — a sudden increase in population size compresses the early (deep, "few-lineage") part of the tree and produces a burst of new mutations on many short terminal branches, generating exactly this rare-variant excess genome-wide. 2. A recent selective sweep at or near this locus — positive selection drives one haplotype to fixation, dragging linked neutral variation with it (genetic hitchhiking), and the population "regrows" its diversity afterward almost entirely through new, rare mutations on short branches — locally at this locus, not genome-wide. The key diagnostic the coalescent provides: demographic explanations (bottleneck/expansion) should produce negative D genome-wide, at every locus, because they affect the shared ancestral genealogy of the whole population. A selective sweep produces negative D only at and very near the selected locus, decaying with genetic distance (a signature revisited in Detecting Selection in Sequences). If this fish population shows negative D at only this one locus while genome-wide average D is near zero, the evidence points to local positive selection, not a demographic bottleneck — precisely the kind of inference the coalescent null model was built to enable.
Common misconceptions & traps
| Misconception | Why it's wrong |
|---|---|
| "The coalescent predicts what will happen to a population in the future." | It is purely retrospective — it describes ancestry of a sample already collected, tracing backward; it says nothing about future allele frequencies. |
| "Coalescent time units of 'N' generations mean the same thing in every paper." | Conventions differ: haploid models use N, diploid autosomal use 2N, mitochondrial/Y-chromosome (haploid, uniparental) use N/2 relative to autosomal N_e. Always check ploidy and inheritance mode. |
| "A star-shaped gene tree (all lineages coalescing near the root) means the sample is 'primitive' or 'ancestral'." | It usually indicates rapid recent population growth or a strong selective sweep — a demographic/selective signature, not an inherent property of "primitiveness." |
| "θ = 4Nμ lets you solve for N and μ separately from sequence data alone." | Only the product Nμ is identifiable from polymorphism data at a single locus; N and μ are confounded unless independent information (fossil calibration, pedigree mutation rate) is supplied. |
| "The MRCA of a gene is the same individual as the 'most recent common ancestor of the whole population' (e.g., a real historical person)." | Different genes have different genealogies and different MRCAs at different times (deep coalescence and incomplete lineage sorting); no single ancestor carries "the" MRCA for every locus simultaneously. |
The scientist behind it
Origins · 1 minJohn Frank Charles Kingman and how this was discovered
Kingman built his career in probability theory — queuing theory, Markov chains, and stochastic processes — well before turning to genetics. Trained as a mathematician (not a biologist), he brought the machinery of continuous-time Markov chain theory to a question population geneticists had been circling for years: as sample size and population size grow, what does the genealogy of a random sample actually look like? His 1982 papers did not invent a new biological model — the underlying Wright-Fisher reproduction model was already decades old, formalized by Ronald Fisher and Sewall Wright in the 1930s — but they identified the precise mathematical limit (the "n-coalescent") that this model converges to, and derived its exact, tractable properties. Kingman's contribution was recognized as one of the most important unifications in theoretical population genetics of the 20th century, providing the formal backbone that Warren Ewens, Richard Hudson, Fumio Tajima, and Bob Griffiths then built into practical statistical tools over the following decade.
Where this matters today
Viral epidemiology and outbreak tracing. Coalescent methods (implemented in software like BEAST) are routinely applied to viral genome sequences — HIV, influenza, Ebola, SARS-CoV-2 — to estimate when an outbreak's common ancestor existed, effectively dating the origin of an epidemic and reconstructing its transmission history and growth rate directly from sampled viral genomes, often faster than traditional epidemiological contact tracing.
Human demographic history. Coalescent-based methods (PSMC, MSMC, and related pairwise/multi-sequence approaches) reconstruct changes in effective population size over tens of thousands of years from a handful of whole genomes, revealing bottlenecks associated with the out-of-Africa migration and subsequent regional expansions, cross-validated against archaeological and fossil records.
Conservation genetics. Estimating effective population size and recent bottleneck history in endangered species (cheetahs, Tasmanian devils, mountain gorillas) uses coalescent inference from limited genomic samples, informing management decisions about genetic rescue and captive breeding programs.
Detecting natural selection. Departures from coalescent null expectations (Tajima's D and related statistics, extended haplotype homozygosity, site-frequency-spectrum-based tests) are the standard genome-wide toolkit for flagging genes under recent positive or balancing selection — the direct throughline to Detecting Selection in Sequences.
Forensic and medical genetics. Coalescent models underlie likelihood calculations for kinship testing, disease-mapping via linkage disequilibrium, and estimating the age of disease-causing mutations from the extent of shared haplotype around them.
Agricultural breeding. Coalescent-based estimates of effective population size and historical bottlenecks in crop and livestock breeds inform strategies to maintain genetic diversity in breeding programs and detect regions under artificial selection during domestication.
Ask the tutor: How does adding recombination change the single-tree coalescent into the ancestral recombination graph, and why does that matter for detecting selection?
Why does the coalescent use effective population size rather than census population size?
Effective population size (N_e) captures the rate of genetic drift, which depends on variance in reproductive success, sex ratio, and population size fluctuations — not simply the number of living individuals. Coalescence rates in the model are derived directly from the probability that two lineages share a parent, which depends on N_e, not census size. In real populations, N_e is often far smaller than the census count (sometimes by one or two orders of magnitude) because a minority of individuals contribute disproportionately to the next generation. Using census size instead of N_e would badly mis-predict coalescence times and genetic diversity, which is why every coalescent formula presented here uses N meaning effective, not census, population size, consistent with its usage in Effective Population Size.
Does the coalescent assume no recombination?
The basic single-locus coalescent described here assumes no recombination within the locus, so the entire sampled region shares one genealogical tree. Real genomes recombine, so different physical positions along a chromosome can have different genealogies — a phenomenon captured by the "coalescent with recombination" (the ancestral recombination graph), an extension developed by Richard Hudson and others in the 1980s-90s. This extension is more complex (genealogies change along the sequence at recombination breakpoints) but builds directly on the same pairwise-coalescence logic, just applied locally between recombination events.
How is the coalescent related to genetic drift?
Genetic drift is the forward-in-time description of random sampling variation in allele frequencies across generations, driven by the same finite-population reproduction process (Wright-Fisher). The coalescent is the backward-in-time description of the identical process — literally the same underlying random mating and reproduction, viewed as ancestry rather than descent. They are two faces of one process: drift describes how allele frequencies wander forward, while coalescence describes how lineages converge backward; both are governed by the same effective population size parameter, and results derived in one framework (e.g., fixation probabilities from drift, or expected heterozygosity from coalescence) are mutually consistent.
What happens to coalescent predictions if a population recently experienced explosive growth?
Rapid population growth compresses the coalescent tree's deep branches: because the coalescence rate C(k,2)/N is inversely proportional to N, a much larger N in the recent past (larger relative to the deep past) means lineages coalesce more slowly early on (many lineages, huge population) but then rapidly, once few lineages remain, if the population was much smaller further back — producing a characteristic "star-like" tree with many short terminal branches and few deep splits. This generates an excess of rare (low-frequency) mutations relative to the constant-size neutral expectation, exactly the pattern captured by a negative Tajima's D and used to infer historical population expansions, including the human out-of-Africa expansion.
Can the coalescent be applied to asexual organisms or bacteria?
Yes, with modification. Bacteria and other asexual/clonal organisms still have genealogies — every cell division is an ancestor-descendant relationship — so the same backward-in-time logic applies, though the relevant "population" often involves complications like frequent horizontal gene transfer (which acts like recombination, giving different genes different trees) and strong selection sweeps that periodically purge diversity across the whole genome ("periodic selection"). Bacterial population genetics has adapted coalescent theory extensively, though effective population sizes inferred this way can be dramatically smaller than census bacterial numbers due to these dynamics.
Why is the expected time to the MRCA bounded even as sample size n grows very large?
Because E[T_MRCA] = 2(1 − 1/n) approaches 2 (in units of 2N generations) as n → ∞, adding more samples contributes almost nothing to tree depth once n is even moderately large (n=20 already gives 1.9, versus the limit of 2.0). This happens because with many lineages present, coalescence events happen very fast (rate scales as k², so with k=100 lineages, mergers happen roughly 100× faster than with k=10); nearly all of the "waiting" accumulates in the final few mergers, especially the last pair. Practically, this means sequencing more individuals mostly reveals more of the recent branching structure (rare variants, population-specific splits) rather than pushing the estimated root of the tree much further into the past.
How does the coalescent relate to phylogenetic trees built from species data?
Both are tree structures relating lineages to common ancestors, but they operate at different scales and with different underlying processes. Species-level phylogenies describe divergence between reproductively isolated lineages (speciation events), typically over millions of years, and are largely deterministic once a split has occurred. Coalescent gene trees describe the stochastic genealogy of individual gene copies within a population or species, over generations to tens of thousands of years, governed by random genetic drift. A key complication is that gene trees embedded within species trees can disagree with the species tree itself due to incomplete lineage sorting — ancestral genetic variation that hasn't yet fully coalesced by the time a new species split occurs — a phenomenon the coalescent framework is essential for correctly modeling in phylogenomics.