Statistical vs Biological Importance

Statistical significance is a binary verdict produced by a hypothesis test: given the data, is it implausible that the observed effect arose purely from sampling variation around a true effect of zero? The verdict depends on three things — the size of the effect, the amount of noise in the measurements, and the sample size. Critically, a test statistic like t is (roughly) proportional to:

Because t scales with √n, any nonzero effect, no matter how small, will eventually become "significant" if n is large enough. The p-value is therefore a joint function of effect size and sample size — it cannot be read as a pure measure of either one.

Effect size is a separate, complementary quantity: it measures the magnitude of the biological phenomenon in units that don't evaporate as sample size grows. Common effect-size measures in biology:

Effect sizes do not shrink or inflate with sample size (in expectation); they are estimates of a fixed population parameter, and larger samples make that estimate more precise, not larger. This is exactly why effect size must be paired with a confidence interval: the CI communicates the plausible range of the true effect, folding in both magnitude and uncertainty in one interval. A p-value collapses all of this into a single number that discards the magnitude entirely.

Biological importance (also called practical, clinical, or scientific importance) is a judgment made against an external, domain-specific yardstick — a minimum clinically important difference (MCID) in medicine, a threshold population decline for conservation status, a selection coefficient large enough to matter over ecological timescales, an effect size large enough to justify a costly agricultural intervention. This yardstick is not a statistical quantity. It must be set beforehand from clinical experience, physiological plausibility, cost-benefit analysis, or evolutionary theory — never read off from software output or improvised after seeing the p-value.

Step 1 — the test statistic is built from a ratio. For a two-sample t-test comparing treatment and control means:

The numerator is the effect size (in raw units); the denominator — the standard error — shrinks as $1/\sqrt{n}$. As n grows, SE shrinks toward zero, so t grows without bound even if the numerator (the true effect) is fixed and tiny. This is not a flaw in the test; it is exactly what a hypothesis test is designed to do: detect ever-smaller departures from the null as data accumulate. The flaw is in the reader who treats "p < 0.05" as "large and important."

Step 2 — power scales with sample size, not with importance. Statistical power is the probability of rejecting a false null hypothesis. For a fixed true effect size δ and standard deviation σ, power rises toward 1 as n increases — this is the entire logic behind power calculations. A trial with n = 50,000 has power near 1 even for a physiologically trivial effect (δ = 1.2 mg/dL LDL), because the denominator SE has been driven down to almost nothing. Conversely, a trial with n = 20 may have power of only 0.15 to detect a physiologically enormous effect (δ = 30 mg/dL), producing a "non-significant" result that is actually just an underpowered, uninformative one. Absence of significance is not evidence of absence of effect unless power was adequate.

Step 3 — the confidence interval carries the information a p-value throws away. A 95% CI for the mean difference gives a plausible range for the true effect size, in the original biological units. Two trials can produce the identical p-value (say, p = 0.03) with wildly different practical meaning:

• Trial A: mean difference = 25 mg/dL, 95% CI [3, 47] — wide, uncertain, but centered on a clinically relevant effect.
• Trial B: mean difference = 1.2 mg/dL, 95% CI [0.3, 2.1] — narrow, precise, but centered on a clinically irrelevant effect.

Both are "statistically significant." Only the CI reveals that Trial A found something that might matter, while Trial B found something real but trivial. This is the single most important habit in quantitative biology: read the CI's location and width, not just whether it excludes zero.

Step 4 — judge against a pre-specified biological threshold, not against zero. The null hypothesis of "effect = 0" is a statistical convenience, almost never a biologically expected value. Biological importance requires comparing the entire confidence interval to a threshold Δ decided in advance — e.g., "an LDL reduction must exceed 15 mg/dL to justify this drug's side-effect profile and cost." If the CI lies entirely below Δ, the effect, however "significant," is unimportant. If the CI straddles Δ, more data are needed before a decision can be made. If the CI lies entirely above Δ, the effect is both real and important.

Step 5 — three failure modes to recognize. 1. Significant but unimportant (Trial B above): huge n detects a real but trivially small effect. 2. Non-significant but potentially important (an underpowered study): wide CI straddling both zero and the threshold Δ — the honest conclusion is "inconclusive," not "no effect." 3. Significant and important (Trial A, ideally with a narrower CI from adequate power): the effect is both statistically distinguishable from zero and large enough to matter.

There is no single "Meselson–Stahl" experiment for this concept — it is a methodological lesson learned from decades of replication failures and meta-analyses, and formalized through statistical reform movements. Three concrete episodes established the case.

1. Jacob Cohen's power surveys (1962, 1988). Cohen examined published psychology and behavioral-science journals and found that the median statistical power to detect a medium effect size was around 0.48 — worse than a coin flip. This meant roughly half of true, medium-sized effects were being reported as "non-significant" and discarded, while journals selectively published the (often inflated, by chance) significant results that did occur. Cohen's response was to formalize effect-size measures (Cohen's d, f, w) as mandatory companions to p-values, explicitly to stop researchers from treating a p-value as if it measured effect magnitude. This body of work is the direct ancestor of the modern requirement to report effect sizes.

2. The vitamin/antioxidant mega-trials (1990s–2000s). Several enormous randomized trials (e.g., the Physicians' Health Study, HOPE trial extensions, and various beta-carotene / vitamin E trials with tens of thousands of participants) produced statistically significant associations between supplement use and small shifts in biomarkers or event rates. Follow-up meta-analyses and clinical guideline panels repeatedly concluded that the absolute risk reductions were too small — often under 1% absolute risk difference — to justify universal supplementation, despite p < 0.05 results, because the confidence intervals for absolute benefit sat well below the pre-agreed clinically meaningful threshold. This became a canonical teaching case in evidence-based medicine for separating "detected" from "worth treating."

3. The replication crisis and the ASA's 2016 statement on p-values. Beginning with high-profile replication failures in psychology and preclinical cancer biology (the Reproducibility Project: Cancer Biology, and the Open Science Collaboration's psychology replications, found that only about 36–39% of studies replicated the original statistically significant effect at a similar magnitude), the American Statistical Association issued a formal statement in 2016 explicitly warning: "statistical significance is not equivalent to scientific, human, or economic significance" and that "a p-value near 0.05 taken by itself offers only weak evidence". Major journals (e.g., Basic and Applied Social Psychology, and clinical journals following CONSORT) subsequently mandated effect sizes and confidence intervals in every reported result. This is the formal, citable "evidence" that the field itself recognized and corrected the conflation of significance with importance — not a single experiment, but a documented, quantifiable pattern of overstatement that forced a reporting reform. Together, these three lines of evidence show the same lesson from three angles: underpowered studies hide real effects (Cohen), overpowered studies inflate trivial effects into "significant" findings (mega-trials), and systematic surveys of the literature (replication crisis) confirmed that p-values alone were poor guides to what would hold up or matter.

Example 1 — a plant biology field experiment (effect size and CI by hand). A researcher tests whether a rhizobium inoculant increases soybean pod yield. Two groups of plots:

Mean difference: $\bar{x}_1 - \bar{x}_2 = 142.3 - 136.1 = 6.2$ g/plant. Pooled SD (approx., since n's and SDs are close):

Standard error of the difference:

Not significant at α = 0.05. But look at effect size: Cohen's d = 6.2 / 18.16 ≈ 0.34 (a small-to-medium effect by conventional benchmarks), and the 95% CI for the mean difference is roughly

The CI is wide and straddles zero — this is an underpowered, inconclusive result (bottom-right zone of the four-quadrant figure above), not evidence that the inoculant "doesn't work." If the agronomic threshold for a commercially worthwhile yield boost is, say, 5 g/plant (about a 3.5% yield gain), the CI [-0.5, 12.9] is still consistent with an economically important effect. The correct recommendation is not "abandon the inoculant" but "run a larger trial" — exactly the reasoning a power analysis should have flagged before the study began.

Example 2 — a clinical trial with a huge sample (significance without importance). A cardiovascular drug is tested in 48,000 patients for its effect on systolic blood pressure (SBP).

Mean difference: $129.3 - 128.6 = 0.7$ mmHg.

This is overwhelmingly "significant" — and correctly so, because the sample is large enough to estimate this small difference very precisely. The 95% CI:

Now bring in the domain threshold. Hypertension guidelines generally treat a systolic reduction below roughly 3–5 mmHg as clinically negligible for cardiovascular risk reduction at the population level (real trials such as the major statin and antihypertensive mega-trials use thresholds in this range for "clinically meaningful" SBP change). The entire CI — [0.43, 0.97] mmHg — sits far below this threshold. The drug's effect on blood pressure is statistically airtight and clinically negligible: exactly the top-left "significant, trivial effect" zone from the mechanism figure. A prescribing decision built on the p-value alone, ignoring the CI's location relative to the clinical threshold, would be a serious error — unless the drug has other, independently important benefits.

There is no single discoverer of "statistical vs. biological importance" — it is a methodological principle that crystallized gradually — but Jacob Cohen is the figure most responsible for forcing effect size into mainstream biological and behavioral statistics. Cohen was a clinical psychologist and statistician who spent decades documenting how researchers misused and misread significance tests. His 1962 survey of the Journal of Abnormal and Social Psychology found the average power to detect a medium effect size was under 50% — meaning most published null results were essentially uninformative coin flips, not evidence of "no effect." His 1988 book Statistical Power Analysis for the Behavioral Sciences formalized standardized effect-size measures (d, f, w, f²) precisely so that magnitude could be reported, compared across studies, and used in power calculations, independent of any particular sample size. Cohen's later essay "The Earth Is Round (p < .05)" (1994) is a direct, pointed attack on the practice of treating p-values as measures of importance, and remains one of the most cited methodological critiques in the quantitative sciences. His conceptual descendants include the CONSORT reporting guidelines for clinical trials and the American Statistical Association's 2016 statement on p-values, both of which now mandate effect sizes and confidence intervals as standard practice.

Clinical trial design and drug approval. Regulatory agencies (FDA, EMA) increasingly require pre-specified minimum clinically important differences (MCIDs) for primary endpoints — e.g., a specified minimum improvement in FEV1 for a COPD drug — so that trials cannot claim success purely from statistical significance in an oversized sample.

Genome-wide association studies (GWAS). With sample sizes in the hundreds of thousands, GWAS routinely find p-values of $10^{-30}$ for single nucleotide polymorphisms that individually explain a small fraction of 1% of trait variance. Effect size (typically reported as odds ratio or variance explained) is essential to avoid overstating any single variant's biological or clinical relevance — most GWAS hits are significant but individually trivial contributors.

Conservation biology. Detecting a "statistically significant" population decline in a large, well-monitored species is easy; the biologically relevant question is whether the rate and magnitude of decline crosses IUCN Red List thresholds (e.g., a decline of ≥30% over three generations for "Vulnerable" status) — a domain-specific threshold, not a p-value.

Agriculture and crop science. Field trials with hundreds of plots can detect yield differences of a few percent as "significant," but a farmer's decision to switch inputs depends on whether that gain exceeds the cost of the new seed or fertilizer — an economic threshold layered on top of the statistical one.

Ecotoxicology and environmental regulation. Dose-response studies for pollutants must distinguish a statistically detectable physiological change at very low doses from a change large enough to threaten organism survival or reproduction — the basis for setting regulatory exposure limits (e.g., "no observed adverse effect level," NOAEL) using effect size and biological plausibility, not raw significance.

Precision medicine and biomarker research. A biomarker that is "significantly" associated with disease risk in a huge cohort may still have poor discriminative power (small AUC, small odds ratio) and be useless for individual-patient decision-making — reinforcing why effect size, not p-value, drives clinical utility assessments.

Ask the tutor: How should I decide on a minimum clinically/biologically important difference before running my own experiment?