The p-Value & Its Interpretation

A p-value is the probability, calculated under the assumption that the null hypothesis (H₀) is true, of obtaining a test statistic as extreme as, or more extreme than, the one actually observed. Formally, if $T$ is a test statistic and $t_{obs}$ its observed value, the p-value for a two-sided test is

This definition has three parts that must all be held simultaneously: it is a probability, it concerns data as extreme or more extreme than what was observed (not the exact observed value alone), and it is computed conditional on the null hypothesis being true. Drop any one of these three clauses and you get one of the classic misinterpretations. The p-value is the output of a null distribution — the probability distribution of the test statistic that would arise if H₀ were true and the experiment were repeated infinitely many times with only sampling variation causing differences between runs. Constructing this null distribution (analytically, via a known family like the t- or F-distribution, or numerically via simulation/permutation) is the technical heart of any hypothesis test. The p-value is simply the tail area of this distribution beyond the observed statistic. Crucially, the p-value is a statement about data, not about hypotheses directly. It tells you how compatible your data are with a particular null model. A small p-value means the data are hard to reconcile with H₀ — either H₀ is false, or a rare event happened, or an assumption of the test was violated (non-random sampling, non-independence, wrong distributional form). A large p-value means the data are quite consistent with H₀, but this is weak evidence for H₀ — it may just mean the study lacked the power to detect a real effect, foreshadowing the concept of Type I & Type II Errors.

Step 1: State H₀ precisely. The null hypothesis must be a specific, quantitative statement — e.g., "the true difference in mean plant height between fertilizers A and B is zero" (μ_A − μ_B = 0) — not a vague statement like "fertilizer has no effect." Precision is what allows a probability to be computed at all.

Step 2: Choose a test statistic that summarizes the discrepancy between data and H₀. For a one-sample or two-sample mean comparison this is typically a t-statistic:

where $SE$ is the standard error of the estimate. Large $|t|$ means the sample mean sits far from the null value relative to sampling noise.

Step 3: Derive the null distribution of that statistic. If H₀ is true and the standard assumptions hold (random sampling, approximate normality or large n, independence), the t-statistic follows a known t-distribution with degrees of freedom determined by sample size. This distribution is centered at zero — because under H₀ the expected discrepancy is zero — and its spread reflects how much sampling variability is expected purely by chance.

Step 4: Locate the observed statistic in that distribution and compute the tail area. The p-value is the total probability mass in the tail(s) at least as extreme as $t_{obs}$. For a two-tailed test this means both tails are summed, because "more extreme" includes deviations in either direction from the null.

Step 5: Compare p to a pre-chosen significance threshold α (conventionally 0.05). If $p \leq \alpha$, the result is declared "statistically significant" and H₀ is rejected in favor of the alternative. This threshold must be fixed before looking at the data to preserve the guarantees of the testing framework — a point often violated in practice ("p-hacking"). It matters enormously which tail area is computed. A one-tailed test computes the probability of exceeding $t_{obs}$ in only one direction, appropriate only when the alternative hypothesis specifies a direction decided in advance; a two-tailed test (the default in most biological work) sums both tails, appropriate when deviations in either direction count as evidence against H₀.

An important, easily overlooked consequence follows directly from Step 3: the null distribution's spread is governed by the standard error, which shrinks as sample size grows. This means the same observed effect size will produce a smaller p-value in a larger sample. A p-value therefore entangles effect size, variability, and sample size into one number — which is exactly why it cannot be read as a measure of biological importance on its own.

The p-value's modern form traces to Ronald A. Fisher, who in Statistical Methods for Research Workers (1925) and The Design of Experiments (1935) formalized significance testing as a tool for agricultural and biological experimentation at Rothamsted Experimental Station. Fisher's own touchstone example — later immortalized as the "Lady Tasting Tea" experiment — is worth walking through because it shows the logic in its purest form, uncontaminated by later procedural add-ons.

The experiment. A colleague, Dr. Muriel Bristol, claimed she could tell by taste whether milk had been poured into a cup before or after the tea. Fisher designed a test: present her with eight cups, four prepared milk-first and four tea-first, in random order, and ask her to sort them into the two groups of four. The null hypothesis was that she had no discriminating ability and was merely guessing, so any correct classification arose purely by chance under random guessing.

The calculation. If she is truly guessing and simply picks which four cups she believes are "milk-first," the number of ways to choose 4 cups out of 8 is

Only one of those 70 selections is the entirely correct grouping. So under H₀ (pure guessing), the probability of getting all eight cups correct is

Fisher used this design specifically because the exact null distribution (a hypergeometric distribution over all possible groupings) could be enumerated exactly — no approximation needed, no distributional assumptions to defend. This is the cleanest possible illustration that the p-value answers: "if she has zero ability, how improbable is this particular result (or one even better)?" The historical record indicates Bristol did in fact correctly classify all eight cups in the actual test, giving p ≈ 0.0143 — small enough, by later convention, to reject the guessing hypothesis.

What this proved, and what it did not. The low p-value provided evidence against the null hypothesis of pure guessing; it did not prove she had perfect tasting ability in general, did not quantify how much better than chance she was in a generalizable sense, and did not give the probability that the "no ability" hypothesis was true. Fisher himself treated a small p-value as grounds to act as though the null were false for practical purposes, not as a computed posterior probability — a distinction that subsequent popularizations of statistics frequently blurred.

A second, larger-scale line of evidence: the replication crisis as an empirical test of p-value misuse. In 2015 the Open Science Collaboration (led by Brian Nosek) attempted to directly replicate 100 psychology studies that had originally reported statistically significant (p < 0.05) results. Only 36% of replications also reached p < 0.05, and average effect sizes shrank by roughly half compared to the original studies. This project is now a canonical empirical demonstration that a single p < 0.05 result, taken in isolation, is a much weaker guarantee of a "real, reproducible" effect than casual practice assumes — precisely because p-values are computed from one sample and are themselves random variables that vary considerably from study to study, even when a true effect exists. Similar large-scale replication projects in cancer biology (the Reproducibility Project: Cancer Biology) found comparably sobering replication rates, reinforcing that p < 0.05 in one study is evidence, not proof.

Example 1 — a two-sample comparison in ecology (asymmetric sample sizes). An ecologist compares clutch size (number of eggs) in a bird species between two habitat types: forest edge (n₁ = 8 nests) and forest interior (n₂ = 22 nests).

H₀: μ_edge − μ_interior = 0 (no true difference in mean clutch size). Pooled standard error (using a standard two-sample formula with unequal n):

With approximate degrees of freedom around 12–13 (Welch correction for unequal variances/sample sizes), a two-tailed t of 1.89 gives p ≈ 0.08. This p-value is above the conventional α = 0.05 threshold: the data do not provide strong evidence against H₀. Note what this does not mean: it does not prove clutch sizes are equal between habitats — the small edge-habitat sample (n₁ = 8) gives the test low power, so a real biological difference could easily be missed (a Type II error, foreshadowing the error-rate framework). The honest conclusion is "no significant difference detected," not "no difference exists."

Example 2 — a genetics cross with a chi-square p-value (asymmetric expected ratio). A geneticist crosses two heterozygous pea plants expected to produce a 3:1 ratio of purple to white flowers (standard Mendelian dominance) and observes:

H₀: the true ratio is 3:1 (Mendelian segregation with no distorting factor). Chi-square statistic:

With 1 degree of freedom (2 categories − 1), a χ² of 3.84 corresponds to p ≈ 0.05 (this is, not coincidentally, essentially the classic critical value for df = 1 at α = 0.05). This is a borderline case: a value this large or larger would occur only about 5% of the time if the true ratio really were 3:1. Depending on the pre-registered α, a researcher might call this "marginally significant" — and this example is worth using precisely because it shows how a result can sit right on the conventional boundary, illustrating why the p-value should be reported as a continuous number (p ≈ 0.05) rather than collapsed into a bare "significant/not significant" verdict.

Ronald Aylmer Fisher (1890–1962) was a British statistician, evolutionary biologist, and geneticist whose work at the Rothamsted Experimental Station transformed agricultural science into a rigorously quantitative discipline. Fisher was strikingly interdisciplinary: he co-founded modern population genetics (the "modern synthesis" linking Mendelian genetics to Darwinian evolution), invented the analysis of variance (ANOVA) and the logic of randomized experimental design, and popularized the p-value as a practical tool for deciding whether experimental results were worth taking seriously. He was famously combative in scientific disputes — including a decades-long feud with Karl Pearson and later sharp disagreements with Jerzy Neyman and Egon Pearson over the philosophical foundations of hypothesis testing (Fisher favored p-values as continuous measures of evidence against a single null; Neyman–Pearson favored a pre-specified decision rule balancing two named error types, foreshadowing the Type I/Type II framework). Fisher also held views on eugenics and on the tobacco-cancer link (he controversially argued the causal link was not proven) that are now recognized as seriously flawed, a reminder that towering scientific contributions do not immunize a researcher from being wrong elsewhere.

Clinical trials and drug approval. Regulatory agencies such as the FDA historically require p < 0.05 (often from two independent, adequately powered trials) before a drug's efficacy claim is accepted, though modern guidance increasingly demands effect sizes and confidence intervals alongside p-values to judge clinical relevance, not just statistical detectability.

Genome-wide association studies (GWAS). Because researchers test hundreds of thousands to millions of genetic variants simultaneously for association with disease, the conventional α = 0.05 threshold would generate enormous numbers of false positives by chance alone. GWAS studies instead use a stringent genome-wide significance threshold of roughly p < 5 × 10⁻⁸, derived from correcting for the huge number of independent tests performed — a direct, large-scale application of understanding what a p-value does and does not control for.

Ecology and conservation. Environmental impact assessments, population viability analyses, and studies of climate effects on species ranges routinely report p-values alongside effect sizes and confidence intervals; regulatory and conservation decisions increasingly require the latter because a "non-significant" p-value in a small, expensive field study (low power) must not be mistaken for evidence that a threatened population is unaffected by a stressor.

The 2016 American Statistical Association (ASA) statement on p-values. In direct response to widespread misuse across biomedical and social science literature, the ASA issued a formal statement enumerating exactly the misconceptions catalogued above, and recommending that p-values be interpreted alongside effect sizes, confidence intervals, and pre-registration of hypotheses — a policy response that traces directly back to the mechanistic and historical points made in this article.

Journal editorial policy. Some journals (notably in psychology, e.g., Basic and Applied Social Psychology in 2015) have banned p-values outright in favor of effect sizes and confidence intervals; most biological journals instead now require p-values to be reported as exact values (not just "p < 0.05") together with effect sizes, directly counteracting the misconceptions this article addresses.

Ask the tutor: How do I choose the right α threshold, and how does that choice trade off against Type I and Type II error rates in my specific experiment?