The Multiple-Testing Problem
A geneticist scans 20,000 genes on a microarray. She looks for the handful that differ between cancerous and healthy tissue. Every single gene is truly unaffected by the disease. Nothing is biologically real here. Using the standard significance threshold of α = 0.05, how many genes will nonetheless flash "significant" by chance alone? The answer is roughly 1,000 genes. This is a false-discovery catastrophe hiding behind a familiar, trusted number. This is the multiple-testing problem. It is arguably the single most important statistical trap in modern genomics. The problem matters because Hypothesis Testing was built for a world of one question at a time. Does this drug work? Does this fertilizer increase yield? Does this mutation matter? Genomic technologies shattered that world. Microarrays, RNA-sequencing, and genome-wide association studies (GWAS) routinely test thousands to millions of hypotheses simultaneously. One test per gene, one per SNP, one per metabolite. Each individual test still carries its accustomed 5% chance of a false alarm. But run enough tests, and false alarms become not the exception but the expectation. They arrive in droves even when biology has nothing to say. The big picture is a distinction between error rates for one test and error rates for a whole experiment. A p-value of 0.05 promises that a single test will falsely reject a true null hypothesis only 5% of the time. It promises nothing at all about what happens when you perform that test 20,000 times over. Without correction, "statistically significant" results from high-throughput biology are frequently statistical noise dressed up as discovery. This is a lesson learned the hard way. It came through retracted papers, failed replications, and wasted follow-up experiments chasing genes that were never really different. This article builds the mechanism behind the inflation. It quantifies the inflation precisely. It walks through the logic, including a famous demonstration, that exposed the danger. It works two realistic numerical examples — a microarray screen and a GWAS scan. This makes the scale of the problem concrete. It closes by pointing toward the two standard fixes. These are The Bonferroni Correction and The False Discovery Rate. It does not yet solve the problem. That is the next piece of the puzzle.
- Each hypothesis test carries a fixed Type I error rate, usually α = 0.05, meaning a 5% chance of a false positive per test.
- Running many tests compounds this risk: the chance of at least one false positive across the whole experiment (the family-wise error rate) grows rapidly with the number of tests.
- With 20,000 independent tests at α = 0.05, the expected number of false positives is about 1,000, even if nothing is truly different.
- Genomic technologies (microarrays, RNA-seq, GWAS) inherently generate thousands to millions of simultaneous tests, making this problem unavoidable, not optional.
- The famous "dead salmon" fMRI demonstration and simulation studies show random noise alone produces "significant" hits when correction is skipped.
- Solutions reshape how significance is judged across the whole experiment — previewed here, developed fully in The Bonferroni Correction and The False Discovery Rate.
When you test thousands of hypotheses at once, the 5%-per-test error rate guarantees hundreds of false positives — significance thresholds built for single tests do not simply scale up safely to genomic data.
The core idea
A hypothesis test asks whether an observed effect is likely to have arisen by chance under a null hypothesis (H₀) of "no real difference." The threshold α, conventionally 0.05, defines the Type I error rate: the probability of rejecting a true H₀ — a false positive — in a single test. This is a per-comparison error rate (PCER): it describes one test's behavior in isolation, and it is the number most students first learn to interpret. The multiple-testing problem arises because biological experiments rarely stop at one comparison. A microarray experiment tests thousands of genes for differential expression; a GWAS tests hundreds of thousands to millions of single-nucleotide polymorphisms (SNPs) for association with a trait; an RNA-seq differential-expression analysis tests every annotated transcript. Each test, considered alone, has a 5% false-positive rate. But the family of tests, considered together, has a much higher probability that at least one test will falsely reject its null hypothesis — even when every null hypothesis in the family is true. This experiment-wide quantity is the family-wise error rate (FWER):
For m independent tests each run at significance level α, assuming all null hypotheses are true, the probability of avoiding a false positive on any single test is (1 − α). The probability of avoiding false positives on all m tests is (1 − α)^m, so:
This function climbs toward 1 astonishingly fast. At α = 0.05, a single test carries 5% risk; ten independent tests already carry roughly 40% risk of at least one false alarm; a hundred tests carry over 99% risk. Genomic experiments do not run ten or a hundred tests — they run tens of thousands to millions, guaranteeing hundreds or thousands of spurious "hits" if the α = 0.05 threshold is applied naively to each one.
Mechanism — how the inflation actually happens
Step 1: Define the null universe. Imagine a microarray experiment comparing tumor versus normal tissue across m = 20,000 genes, where — for the sake of illustrating the mechanism — none of the genes actually differ in expression. Every one of the 20,000 null hypotheses ("this gene's expression does not differ between groups") is true.
Step 2: Each test still has sampling noise. Gene expression measurements are noisy: biological variability between samples, measurement error in the microarray probes, and small sample sizes (often just 3–10 replicates per group) all generate random fluctuations in the observed group means. Even when the true difference is zero, sample means will differ from each other by chance.
Step 3: The t-test (or equivalent) is applied per gene. For each gene, a two-sample t-test (or moderated variants like those in limma) computes a test statistic and a p-value describing how extreme the observed difference is, assuming H₀ is true.
Step 4: The p-value distribution under a true null is uniform. This is the mathematical heart of the problem. If H₀ is true, the p-value for that test is a random variable uniformly distributed between 0 and 1 — not concentrated near 1, not concentrated near 0, but spread evenly. That means for any true null hypothesis, there is exactly a 5% chance its p-value happens to fall below 0.05, by definition of how p-values are constructed. This is true no matter how many other tests are running alongside it.
Step 5: Multiply by the number of tests. Since each of the 20,000 true nulls independently has a 5% chance of producing p < 0.05, the expected number of false positives is:
One thousand genes will appear "significantly differentially expressed" purely from sampling noise, with zero biological signal behind any of them. A researcher unaware of this mechanism sees a list of 1,000 "hits," writes a paper, and sends the top candidates to the lab bench — where most fail to replicate, because they were never real.
Step 6: FWER versus expected count — two related but distinct quantities. The expected-count calculation (Step 5) tells you the average number of false positives. The FWER calculation tells you the probability of getting at least one. Both grow with m; both are driven by the same uniform-null mechanism; both are reasons the uncorrected α = 0.05 threshold is inappropriate once m is large.
Step 7: Independence assumptions can bend, not break, the logic. In real genomic data, tests are not fully independent — nearby SNPs are correlated through linkage disequilibrium, and co-regulated genes move together. Correlation among tests can reduce the effective number of independent tests below the raw count m, slightly tempering (but never eliminating) the inflation. This is one reason correction methods differ in how conservatively they treat correlated versus independent test families.
The evidence
The mechanism above is not speculative — it has been demonstrated directly, both mathematically and experimentally, and the consequences of ignoring it have been documented in embarrassing, widely cited cases.
The dead salmon fMRI study (Bennett, Baird, Miller & Wolford, 2009). The most vivid demonstration of the multiple-testing problem comes not from genomics originally but from neuroimaging, and it has become a standard teaching example precisely because it is so visceral. Researchers placed a dead Atlantic salmon in an fMRI scanner and showed it photographs of human faces expressing emotions, asking the (dead) salmon to identify the emotion. Standard fMRI analysis divides the brain into tens of thousands of voxels (3-D pixels) and runs a separate statistical test on each voxel to detect activated regions. Analyzed without correction for the roughly 8,000 voxels tested, the dead salmon's brain showed statistically significant "activity" in a cluster of voxels — a false-positive blob generated purely by measurement noise, because a dead fish's brain performs no cognitive processing whatsoever. The point was not a joke about salmon; it was a controlled demonstration that thousands of simultaneous tests, uncorrected, manufacture "significant" results out of pure instrument noise. The same logic applies identically to a microarray chip with 20,000 genes or a GWAS chip with a million SNPs.
Simulation and permutation studies in genomics. The mechanism has been independently confirmed via simulation studies routinely performed by biostatisticians: generate purely random expression data (no true group differences by construction), run tens of thousands of t-tests, and count how many produce p < 0.05. The count matches the theoretical prediction (m × α) almost exactly, confirming the uniform-null p-value distribution is not a theoretical abstraction but an empirically verified property of real test statistics under resampling and permutation (a method pioneered in spirit by R.A. Fisher's permutation tests and formalized for genomics by researchers such as Bradley Efron and John Storey in the early 2000s).
Early GWAS false-positive epidemics. Before genome-wide significance thresholds were standardized, early candidate-gene association studies in the 1990s and early 2000s produced a notorious replication crisis: hundreds of published "disease gene" associations failed to replicate in independent cohorts. Retrospective analysis showed that inadequate correction for the number of genetic markers and phenotypes tested was a dominant driver of the irreproducible findings, motivating the field-wide adoption of stringent genome-wide significance thresholds (commonly α ≈ 5 × 10⁻⁸) discussed further under The Bonferroni Correction. Together, these lines of evidence — a controlled absurdity (the salmon), mathematical derivation (the uniform p-value distribution), simulation confirmation, and a documented real-world replication crisis in early GWAS — converge on the same conclusion: uncorrected multiple testing manufactures false discoveries at a rate proportional to the number of tests, regardless of the underlying biology.
Worked examples
Example 1 — A microarray screen with a mix of true and null genes. A researcher profiles 15,000 genes comparing drug-treated versus untreated cell cultures, using α = 0.05 per gene. Suppose, realistically, that 300 genes are truly affected by the drug (true positives possible) and 14,700 genes are truly unaffected (true nulls). Assume the test has 80% power to detect the true effects (so 80% of the 300 true positives will be correctly flagged). Expected false positives among the 14,700 true nulls:
Expected true positives correctly detected among the 300 real effects:
Total genes flagged as "significant": 735 + 240 = 975. Now compute the false discovery proportion — the fraction of flagged genes that are actually false alarms:
Roughly 75% of the "significant" gene list is noise, even though the test itself has excellent 80% power and a conventional 5% per-test error rate. This single calculation is the crux of why uncorrected multiple testing is dangerous: the raw significance threshold says nothing about the composition of your final hit list once the number of true nulls vastly outnumbers true effects — a very common scenario in genomics, where most genes are not involved in any one condition.
Example 2 — A GWAS scan with realistic asymmetry. A genome-wide association study tests 500,000 SNPs for association with a complex disease (a modest chip density by modern standards, where 1–5 million SNPs or imputed variants are now common). Suppose the true biology involves exactly 12 SNPs with real, detectable effects, and the study has 60% power to detect each one at α = 0.05 (power in GWAS is often limited by small effect sizes). All other 499,988 SNPs are true nulls. Expected false positives from the null SNPs at uncorrected α = 0.05:
Expected true positives detected:
Total "significant" SNPs: roughly 25,006. Of these, the false discovery proportion is:
Essentially every single "significant" SNP in this naive analysis is a false positive — the 7 real signals are buried under a landslide of 25,000 noise hits, utterly undetectable by eye in the resulting list. This is precisely why GWAS analysis abandoned α = 0.05 decades ago in favor of a stringent genome-wide significance threshold near 5 × 10⁻⁸, derived from Bonferroni-style reasoning applied to roughly one million independent tests (accounting for linkage disequilibrium among SNPs) — the mechanics of which are developed fully in The Bonferroni Correction.
Common misconceptions & traps
| Misconception | Why it's wrong |
|---|---|
| "A p-value of 0.05 means only a 5% chance of any false positive in my whole study." | α = 0.05 governs one test in isolation; across many tests the chance of at least one false positive (FWER) climbs toward 1, not 5%. |
| "If I ran 10,000 tests and got 50 significant hits, that's about the number expected by chance, so my result is unreliable everywhere." | The right question is the composition of the hit list (false discovery proportion), not just whether the count looks "chance-like" — some of those 50 may be real; correction methods estimate how many. |
| "Multiple testing only matters for GWAS; a small experiment with 20 genes is fine at α = 0.05." | FWER already exceeds 50% by around 14 independent tests at α = 0.05 — the problem bites long before genome-wide scales. |
| "A lower p-value threshold always means fewer real discoveries get missed." | Tightening α to control false positives (as correction requires) necessarily reduces power, trading false positives for false negatives — a real cost, not a free lunch. |
| "Correcting for multiple testing is just a bureaucratic formality journals impose." | It reflects the actual sampling mathematics of the uniform null p-value distribution — skipping it changes what "significant" means at a fundamental level. |
The scientist behind it
The multiple-testing problem does not trace to a single discoverer in the way The Bonferroni Correction traces to Carlo Emilio Bonferroni, but its modern genomic urgency owes enormously to statistician John D. Storey, who reframed the problem for high-throughput biology at the turn of the millennium. Working alongside Robert Tibshirani at Stanford in the early 2000s, Storey developed the q-value and formalized estimation of the False Discovery Rate specifically for genomic-scale datasets — recognizing that classical FWER control, built for small numbers of tests, was far too conservative once test counts reached the tens of thousands, discarding true discoveries to guard against an error rate genomics could tolerate differently. Storey's work built on the foundational 1995 paper by Yoav Benjamini and Yosef Hochberg, who first formalized the False Discovery Rate as a distinct, less conservative alternative to family-wise error control — a paper that, decades later, is among the most-cited statistics papers in history precisely because genomics, neuroimaging, and large-scale biology needed exactly this tool.
Origins · 1 minJohn D. Storey and how this was discovered
Storey's methodological contributions, alongside Benjamini and Hochberg's earlier framework, transformed multiple-testing correction from a niche statistical concern into a mandatory step in nearly every high-throughput biological analysis pipeline — microarray, RNA-seq, GWAS, and proteomic screens alike.
Where this matters today
Clinical genomics and diagnostics. Panels that screen hundreds of genes for cancer-predisposing mutations, or exome/genome sequencing in rare-disease diagnosis, must correct for the number of variants examined per patient — otherwise clinicians risk flagging incidental, non-causal variants as clinically significant, prompting unnecessary anxiety or intervention.
Genome-wide association studies for disease risk. Every major GWAS — for diabetes, schizophrenia, height, cardiovascular disease — applies a stringent genome-wide significance threshold (commonly 5 × 10⁻⁸) precisely because millions of SNPs are tested per study; without it, the "hits" reported to the public and to drug developers would be dominated by noise.
Drug discovery and high-throughput screening. Pharmaceutical screens testing thousands of candidate compounds against a molecular target face the identical mathematics — controlling false discovery is essential before committing resources to expensive downstream validation of "hit" compounds.
Neuroimaging. fMRI and EEG studies routinely test tens of thousands of voxels or time-frequency points; correction methods (cluster-based thresholding, FDR control) are now mandatory in reputable neuroimaging journals, a direct legacy of the dead-salmon demonstration.
Agricultural and ecological genomics. Marker-assisted breeding programs scan thousands of genetic markers across crop or livestock genomes for association with yield, disease resistance, or drought tolerance; multiple-testing correction determines which markers are trustworthy enough to guide selective breeding decisions.
Public health surveillance. Screening large administrative or epidemiological datasets for associations between exposures (pollutants, dietary factors) and health outcomes across many subgroups and endpoints requires the same discipline, or spurious "risk factors" proliferate in the literature and public discourse.
Ask the tutor: How does the Benjamini-Hochberg procedure actually calculate a false discovery rate–adjusted p-value (q-value) from a ranked list of p-values?
Why doesn't a p-value of 0.05 already "build in" protection against running many tests?
Because a p-value is defined and calculated for a single comparison: it answers "how surprising is this specific result, assuming H₀ is true for this specific test?" It has no knowledge of how many other tests you also ran. The Type I error rate α = 0.05 is a promise about the long-run behavior of repeating that one test many times, not about the behavior of a panel of different tests run once each. Extending that single-test guarantee to a family of thousands of tests requires an explicit correction step — that is the entire content of the multiple-testing problem, and why techniques like Bonferroni correction and False Discovery Rate control exist as separate tools layered on top of ordinary hypothesis testing.
Is the multiple-testing problem only a concern in genomics, or does it appear elsewhere in biology?
It appears anywhere many hypotheses are tested from the same or related datasets: neuroimaging (thousands of brain voxels), ecological studies screening many species-environment correlations, clinical trials examining many secondary endpoints or patient subgroups, and even simple experiments where researchers test several outcome variables without pre-registering which one matters most. Genomics simply makes the problem unavoidable and extreme because test counts routinely reach the tens of thousands to millions, but the identical logic — and the identical need for correction — applies at much smaller scales, including a lab experiment measuring ten different biochemical markers in the same set of animals.
If correcting for multiple testing reduces statistical power, why not just skip correction and accept more false positives to catch more true ones?
Because false positives in genomics are expensive and misleading in ways that compound: each "significant" gene or SNP typically triggers costly, time-consuming follow-up experiments (functional validation, replication cohorts, animal models). Publishing false discoveries erodes trust in a field and wastes the resources of other laboratories that attempt to build on the finding. The trade-off is real — correction does sacrifice some true positives — but the field has collectively decided, especially post the early-GWAS replication crisis, that controlling false discovery is worth that cost. Modern approaches like the False Discovery Rate are specifically designed to strike a more favorable balance than the most conservative corrections, retaining more true positives than blunt approaches while still controlling the error rate meaningfully.
What is the difference between the family-wise error rate and the false discovery rate, at a conceptual level?
The family-wise error rate (FWER) controls the probability of making even one false positive across the entire set of tests — an extremely strict standard well-suited to situations where any single false positive is unacceptable. The false discovery rate (FDR) instead controls the expected proportion of false positives among the tests you call significant — tolerating some false positives as long as they remain a controlled fraction of your discoveries. FDR control is generally far less conservative than FWER control, which is why it became the dominant approach in genomics: it preserves far more true discoveries at large test counts. Both concepts respond directly to the multiple-testing problem described here, and both are developed in full in their own dedicated treatments.
Does the multiple-testing problem mean p-values themselves are flawed or untrustworthy?
No — the p-value calculation itself remains mathematically correct for the single test it was computed for. The problem is not a flaw in the p-value but a mismatch between what a p-value promises (control of error for one comparison) and how researchers were tempted to use it (as if it controlled error across an entire list of thousands of comparisons). The uniform distribution of p-values under a true null is a mathematically guaranteed, well-verified property, not a weakness — it is precisely this guaranteed property that lets statisticians calculate exactly how many false positives to expect and design correction procedures accordingly.
How does correlation between tests, such as linkage disequilibrium between nearby SNPs, affect the severity of the multiple-testing problem?
Correlated tests are not fully independent, so the "effective number" of independent hypotheses is smaller than the raw count of tests performed. In a GWAS, SNPs that are physically close on a chromosome are often inherited together (linkage disequilibrium), meaning their test statistics are correlated rather than independent draws. This slightly reduces the severity of the inflation compared to the fully independent case used in the simplest calculations, but it does not eliminate the problem — genome-wide thresholds like 5 × 10⁻⁸ are specifically calibrated using the effective number of independent tests in the human genome (historically estimated near one million), not the raw SNP count, to account for this correlation structure.
What practical warning sign tells a researcher they are facing a multiple-testing problem in their own data?
Any time you compute more than one p-value from related outcome measures, related genes, related brain regions, or related genetic markers in the same study — and then select the "significant" ones from that whole set — you are in multiple-testing territory. A useful gut check: if you ran the identical analysis on data you know is pure random noise (no real biological signal at all) and would still expect to see "significant" results purely by chance because you tested so many things, correction is mandatory. The rule of thumb from this article — expected false positives ≈ (number of tests) × α — is a fast way to sanity-check whether your uncorrected hit list is plausible or almost certainly noise-dominated.