Frequency-Dependent Selection

In the simplest models of natural selection, each genotype has a fixed selection coefficient: AA might have relative fitness 1, Aa fitness 1 − hs, aa fitness 1 − s, regardless of how many AA, Aa, or aa individuals exist. Frequency-dependent selection relaxes this assumption. Fitness is written as a function of genotype (or phenotype) frequency itself, so w(A) = f(p), where p is the frequency of allele or morph A in the population. Because fitness now depends on the very quantity that selection is changing, the population is chasing a moving target — and this feedback loop is what makes the phenomenon behave so differently from directional or classical balancing selection. Two qualitatively different feedback signs are possible, and the sign determines the entire evolutionary outcome:

• Negative frequency-dependent selection (NFDS): fitness declines as a morph becomes more common (rare types are favored). This creates a stable internal equilibrium: if a morph's frequency rises above the equilibrium, its fitness falls below that of alternatives, pushing frequency back down; if it falls below equilibrium, fitness rises, pushing it back up. NFDS is a genuine variation-maintaining force, alongside Balancing Selection and Heterozygote Advantage, but it does not require heterozygotes to be superior — it works even among purely homozygous morphs, and even with many alleles or many discrete phenotypes at once.

• Positive frequency-dependent selection (PFDS): fitness rises as a morph becomes more common. This creates an unstable internal equilibrium (if one exists at all): any perturbation away from a 50:50-type balance is self-reinforcing, and the population is driven to fixation of whichever type happens to be more common, or is captured by whichever of several possible stable "peaks" it happens to be near. PFDS erodes variation and can produce founder-effect-like divergence between populations that started with different initial frequencies — a kind of frequency-driven priority effect.

Crucially, frequency dependence is a statement about the shape of the fitness function, not about a particular biological mechanism. The same mathematical signature — fitness as a decreasing or increasing function of frequency — arises from utterly different biological causes: predators forming search images for common prey morphs, pathogens specializing on common host genotypes, self-incompatibility systems in plants, mate-choice rules, and game-theoretic payoff structures in animal contests. This generality is why the concept generalizes naturally into The Evolution of Cooperation, where the payoff to "cooperate" or "defect" strategies depends on the frequency of each strategy in the population — game theory is frequency-dependent selection dressed in behavioral clothing.

Step 1: Fitness is redefined as a function, not a constant. Let p be the frequency of morph A, and (1 − p) the frequency of morph B. Instead of fixed selection coefficients, define: ``` w_A(p) = 1 − s·p w_B(p) = 1 − s·(1 − p) ``` Here w_A falls as A becomes common (large p), and w_B falls as B becomes common (large 1 − p). Each morph does best while rare. This is the canonical form of negative frequency-dependent selection with a single parameter s controlling the strength of the rare-type advantage.

Step 2: Find the internal equilibrium. Set w_A(p̂) = w_B(p̂) and solve: ``` 1 − s·p̂ = 1 − s·(1 − p̂) p̂ = 1 − p̂ p̂ = 0.5 ``` With this symmetric fitness function the equilibrium is 50:50. Asymmetric functions (different rare-type advantages, or different baseline fitnesses for each morph) shift p̂ away from 0.5 but the qualitative behavior — a single stable interior equilibrium — persists as long as each morph's fitness declines in its own frequency.

Step 3: Confirm stability. Perturb p slightly above p̂. Now w_A(p) < w_B(p) because A is more common than its equilibrium value, so A's frequency falls back toward p̂. Perturb p below p̂ and the reverse happens. The equilibrium is a stable node — selection is self-correcting, the hallmark of NFDS. Contrast this with the w_A(p) = 1 + s·p case (fitness rising with frequency): the same algebra gives an internal equilibrium, but perturbations move away from it — an unstable equilibrium, the hallmark of PFDS.

Step 4: Ground the abstraction in real biological mechanisms. Three recurring causal engines generate NFDS in nature: 1. Predator search images (apostatic selection). Predators learn to recognize and preferentially attack the most common prey phenotype in their recent experience, because building a "search image" for a common form is cognitively efficient. Rare morphs are effectively camouflaged by their own rarity — predators simply aren't looking for them. As the rare morph increases in frequency, predators begin learning it too, and its advantage erodes. This is textbook NFDS acting on prey coloration and mimicry. 2. Host-parasite specificity (matching-allele / Red Queen dynamics). Parasites evolve infectivity genotypes matched to the most common host genotypes, because that is where the transmission payoff is largest. Common host genotypes accumulate specialist parasites and suffer higher mortality/infection; rare host genotypes escape recognition. This produces oscillating, frequency-dependent selection cycles rather than a fixed equilibrium — a dynamic (rather than static) version of NFDS, foreshadowing coevolutionary "Red Queen" dynamics. 3. Self-incompatibility and mate-limitation systems. In many flowering plants, a genotype at the S-locus cannot be fertilized by pollen carrying the same S-allele. A rare S-allele finds compatible mates (i.e., is not blocked) almost everywhere, while a common allele frequently meets pollen of the same type and fails to fertilize. This gives rare alleles a direct, mechanistic transmission advantage, maintaining dozens of alleles segregating simultaneously at a single locus — a striking case of NFDS acting through molecular self/non-self recognition rather than survivorship.

Self-incompatibility alleles: the case that made molecular sense of frequency dependence. Many plant families (e.g., Solanaceae, Rosaceae) use gametophytic self-incompatibility: a pollen grain carrying S-allele i cannot fertilize a stigma that also carries S-allele i. Population geneticists, beginning with theoretical work by Ronald Fisher and later empirical surveys (notably by Ae Emerson and, more thoroughly, later molecular surveys of Physalis and Solanum populations), showed two hallmark predictions of NFDS are met: (1) natural populations carry an extraordinarily large number of segregating S-alleles — often 30–50 or more at a single locus in a single population, far beyond what drift or mutation-selection balance alone would sustain, and (2) each allele's frequency is inversely related to its mating success, exactly as the compatibility mechanism predicts. Because a rare allele is carried by pollen that only rarely meets a matching stigma, it fertilizes more ovules per capita than a common allele — a direct, measurable mechanistic instantiation of w(p) declining in p. This is one of the cleanest cases in evolutionary biology where the causal mechanism of frequency dependence (molecular self/non-self recognition) is fully known, not merely inferred from population patterns.

Batesian mimicry and apostatic predation: field and lab tests. Batesian mimics (palatable species resembling a toxic model) benefit from being rare relative to their model, because predators generalize their aversion from the common toxic model to the rare mimic; if the mimic becomes too common relative to the model, predators encounter enough palatable "mimics" to unlearn the association, and mimicry breaks down. This logic was tested experimentally by A. M. Brower (1960s) using blue jays presented with palatable and unpalatable butterfly models at different relative frequencies: attack rates on mimics rose sharply once mimics exceeded a threshold frequency relative to models, confirming the frequency-dependent nature of protection. A parallel and classic demonstration of apostatic selection comes from Bryan Clarke's and colleagues' work with wild birds foraging on artificially colored bait (cryptic pastry "prey" of different colors scattered on soil): birds disproportionately attacked whichever color morph was locally most common, sparing rare morphs — direct field evidence that predators generate negative frequency-dependent selection on prey polymorphism through search-image formation, not merely through differential camouflage against a fixed background.

The land-snail Cepaea nemoralis polymorphism. Cepaea shells vary in ground color (yellow, pink, brown) and banding pattern, and this variation is stably maintained across huge geographic ranges. Cain and Sheppard's long-running field studies (1950s onward) combined with later experimental work demonstrated that thrush predation is apostatic: thrushes ("song thrush," Turdus philomelos) use "anvils" (stones) to smash snails and leave broken shells in telltale piles, letting researchers directly measure which morphs were disproportionately eaten. Predation consistently fell hardest on whichever morph was locally most common, matching background frequency more than absolute crypsis — direct field evidence for NFDS acting through predator search image, contributing to the maintenance of Cepaea's famous polymorphism alongside habitat-correlated (background-matching) selection.

Host-parasite matching-allele systems: Daphnia and its microparasites. Experimental coevolution studies (notably by Curt Lively, Dieter Ebert and colleagues, working with Daphnia magna and its bacterial parasite Pasteuria ramosa) directly test the Red Queen prediction that parasites track and specialize on the currently common host genotype. In "time-shift" experiments, parasites collected from the same population but from a later time point infect contemporary hosts more successfully than parasites from earlier time points or hosts from later time points — showing that parasite infectivity genotypes chase whichever host genotype was common a generation or two earlier. This produces the dynamic (cycling, not static) version of NFDS: host genotype frequencies oscillate as parasites repeatedly catch up to whichever host type is momentarily common, then are counter-selected as that type becomes rare again.

Sex ratio: Fisher's frequency-dependent argument. R. A. Fisher's classic (1930) argument for the near-universal 1:1 primary sex ratio in outbreeding species is itself a frequency-dependent selection argument, even though it is rarely labeled that way in introductory courses. If one sex becomes rare, each individual of that rare sex has, on average, more mates and thus higher reproductive value per capita (because total reproduction must be split equally between the sexes at the population level, by simple accounting — every offspring has exactly one mother and one father). Parents genetically disposed to produce more of the rare sex leave more grandchildren, so selection pushes production of the rare sex up until the sex ratio returns to parity. This is a beautiful, quantitative illustration that frequency dependence is not confined to morph coloration or self-incompatibility loci; it governs one of the most universal ratios in biology.

Example 1 — quantifying the S-allele mating advantage. Suppose a self-incompatible plant population segregates three S-alleles at frequencies S₁ = 0.60, S₂ = 0.30, S₃ = 0.10. Under gametophytic self-incompatibility, a pollen grain carrying allele Sᵢ fails to fertilize any ovule on a plant that also carries Sᵢ. Assume random mating and that each plant is heterozygous (diploid, two different alleles). The probability that a randomly chosen stigma carries a copy of Sᵢ is approximately 2·pᵢ (for pᵢ small; more precisely, 1 minus the probability neither of the stigma's two alleles is Sᵢ). Compatibility probability for pollen carrying S₁ (frequency 0.60): roughly 1 − 2(0.60) + (0.60)² ≈ 1 − 1.20 + 0.36 = 0.16, i.e., only about 16% of stigmas are compatible. For the rare allele S₃ (frequency 0.10): 1 − 2(0.10) + (0.10)² ≈ 1 − 0.20 + 0.01 = 0.81, i.e., about 81% of stigmas are compatible. The rare allele S₃ therefore has roughly 5× the per-pollen-grain fertilization success of the common allele S₁ (0.81 vs 0.16) purely from the frequency-dependent compatibility mechanism — a direct, calculable rare-allele advantage that will push S₃'s frequency up and S₁'s frequency down each generation, until the population approaches an equilibrium where these transmission advantages are equalized across alleles (which, at equilibrium, requires many alleles at fairly even, low individual frequencies — consistent with the 30-50 allele polymorphisms actually observed).

Example 2 — a batesian mimicry frequency threshold. A population of palatable butterflies mimics a toxic model species. Suppose predator aversion learning means predators attack any individual matching the model's appearance with probability a = 0.05 (low, reflecting strong learned avoidance) when mimics make up less than 20% of all model-patterned individuals encountered, but attack probability rises linearly to a = 0.40 once mimics exceed 50% of model-patterned individuals (because predators experience enough palatable "false alarms" to partially unlearn avoidance). If the toxic model's absolute density in a habitat is 800 individuals and the mimic population grows from 50 to 500 individuals over several generations (density-independent growth in the mimic, e.g., due to a good breeding season), calculate mimic proportion at each stage: at 50 mimics, proportion = 50/(800+50) ≈ 5.9% (well under 20%, so attack probability stays low, ≈0.05 — strong protection). At 500 mimics, proportion = 500/(800+500) ≈ 38.5% (between the 20% and 50% thresholds, so attack probability has risen substantially, perhaps to ≈0.25 by linear interpolation) — protection is eroding fast. This is exactly the frequency-dependent collapse of mimetic protection predicted by theory and confirmed in Brower-style experiments: the advantage of being a mimic depends on the mimic's frequency relative to the model, not on some fixed "mimicry fitness," and a mimic lineage that overshoots the model's toxic reputation undermines its own protection.

The mathematical seed of frequency-dependent selection theory is usually credited to Ronald Fisher, whose 1930 sex-ratio argument in The Genetical Theory of Natural Selection is the earliest rigorous, quantitative frequency-dependent selection argument in evolutionary biology, even though Fisher did not use that exact terminology. The empirical and conceptual development of frequency dependence as a distinct, named force in the maintenance of polymorphism owes enormously to Bryan Clarke, a British evolutionary geneticist who spent decades studying Cepaea snails and other polymorphic mollusks, and who coined and championed the term "apostatic selection" for predator-driven rare-morph advantage. Clarke's field experiments — scattering colored, palatable artificial "prey" at controlled relative frequencies and recording bird attack rates — provided some of the most direct, quantitative demonstrations that predators generate genuine negative frequency-dependent selection in nature, rather than selection merely tracking fixed background camouflage.

Frequency-dependent selection is not a historical curiosity — it underlies active research and applied work across biology and medicine. In infectious disease and vaccine biology, pathogen antigenic diversity (e.g., in Plasmodium falciparum malaria surface antigens, or influenza hemagglutinin) is maintained partly by NFDS: host immune systems more efficiently recognize and clear common pathogen strains, giving rare antigenic variants a transmission advantage, which explains persistent, high antigenic diversity despite strong immune selection — a central consideration in designing multivalent vaccines that must anticipate which rare variants will rise. In agriculture and plant breeding, understanding self-incompatibility S-locus dynamics is essential for breeding self-incompatible crop and orchard species (many fruit trees, some brassicas) where compatible pollinizer varieties must be deliberately chosen based on allele identity, directly exploiting the same mechanism responsible for maintaining allelic diversity in wild populations. In conservation biology, recognizing that some polymorphisms are maintained by NFDS rather than drift helps conservationists avoid mistakenly "purifying" a population toward one morph, which could destroy an actively maintained, adaptive polymorphism (relevant to self-incompatibility allele diversity in small, reintroduced plant populations, where losing S-allele diversity can cause severe mate-limitation and reduced seed set). In behavioral ecology and social evolution, frequency-dependent payoff structures are the mathematical backbone of evolutionary game theory, directly foreshadowing how cooperative and cheating strategies are analyzed in The Evolution of Cooperation, where an individual's best strategy depends on the mix of strategies already present in the population — hawk-dove games, producer-scrounger foraging polymorphisms, and alternative mating tactics (e.g., "sneaker" versus territorial males in many fish) are all frequency-dependent selection operating on behavior rather than morphology.

Ask the tutor: How would the equilibrium frequency change if the two morphs in the symmetric model had different baseline fitnesses instead of a perfectly symmetric rare-type advantage?