The Logic of Hypothesis Testing

At the center of hypothesis testing sits a deliberately unglamorous statement called the null hypothesis, denoted H₀. It asserts that nothing interesting is happening: no difference between groups, no association between variables, no deviation from a theoretical ratio. The null hypothesis is not necessarily what the researcher believes — often it is precisely the opposite of what she suspects — but it is chosen because it is specific enough to make a quantitative prediction. "Pollution has no effect on beetle wing length" is a precise, falsifiable claim: it predicts that the two population means are exactly equal. "Pollution affects wing length" is vague; it predicts almost nothing about how large or in which direction an effect should be. Precision is what makes a hypothesis testable, and this asymmetry — a sharp null versus a diffuse alternative — is the engine of the whole method. The logical skeleton mirrors proof by contradiction from mathematics, with one crucial modification. In pure logic, assuming a false premise eventually produces a contradiction — a flat impossibility — and the premise is rejected outright. In statistics, assuming H₀ never produces an impossibility, only a probability distribution of what the data should look like if H₀ were true. The researcher then locates her actual, observed data within that distribution. If the data sit in a plausible, central region of the distribution, H₀ survives unscathed — the data are exactly the kind of thing you'd expect by chance. If the data sit way out in the tail, in a region that would rarely occur under H₀, the researcher treats this as evidence against H₀, because either something rare just happened, or the assumption was wrong to begin with. Statistics conventionally leans toward the second explanation once "rare" crosses an agreed threshold. This is why hypothesis testing is sometimes called an argument by probabilistic contradiction rather than a strict reductio ad absurdum. Nothing is ever proven false with certainty; a small probability always remains that the data really were a rare fluke under a true H₀. This residual uncertainty is not a flaw to be patched — it is the honest price of reasoning under randomness, and it is exactly what later articles on Null & Alternative Hypotheses and The p-Value & Its Interpretation formalize into decision error rates.

The procedure unfolds as a fixed sequence of moves, and understanding why each move is necessary is more important than memorizing the sequence.

Step 1: State H₀ as an exact, falsifiable claim. The null hypothesis must pin down a specific value — a population mean equals some number, two population means are equal, a proportion equals 0.5. Vagueness is disallowed here on purpose: only an exact claim generates an exact prediction for what the data should look like.

Step 2: State the alternative, Hₐ. This captures what the researcher actually suspects — a difference exists, an effect is present, a ratio deviates. It need not specify a value, only that H₀'s value is wrong (or wrong in a particular direction, for one-sided tests).

Step 3: Choose a test statistic. This is a single number computed from the sample — a difference in means, a t-statistic, a chi-square statistic — engineered so that its behavior under H₀ is mathematically known or well-approximated. The statistic condenses an entire dataset down to one number whose "typical range under H₀" can be described.

Step 4: Derive the sampling distribution under H₀. This is the conceptual heart of the method, and it is a direct extension of the sampling-distribution reasoning from Point Estimation & Sampling Error. Imagine — not literally performing, but mathematically imagining — drawing an enormous number of samples of the same size from a world in which H₀ is exactly true. Each hypothetical sample yields its own value of the test statistic. Plotting all of these values produces a distribution: the null distribution. It shows the full range of statistic-values chance alone would produce, and importantly, how often each value or range of values would occur.

Step 5: Locate the observed statistic and judge its rarity. The researcher calculates the test statistic from her real sample and asks where it falls on the null distribution. A value deep in the dense middle is unremarkable — exactly the kind of number chance regularly produces. A value far out in a thin tail is rare under H₀. The probability of obtaining a result this extreme or more extreme, if H₀ were true, is what later articles name the p-value — introduced here only as "a formal measure of surprise," with its full mechanics deferred to The p-Value & Its Interpretation.

Step 6: Decide. If the observed statistic is rare enough under H₀ (conventionally, the kind of outcome that would happen less than 5% of the time by chance), the researcher rejects H₀ as no longer tenable. If not, she fails to reject H₀ — a deliberately weaker verb than "accept," because a boring, unremarkable dataset is compatible with H₀ being true and with H₀ being subtly false but undetected by this sample size. A visual way to hold this together: the null distribution is a bell-shaped (or otherwise known-shaped) landscape of "what chance alone produces," and the observed data is a single dot placed somewhere on that landscape.

Two features of this landscape deserve emphasis. First, the null distribution is not measured from data — it is derived mathematically or by simulation, purely from the assumption that H₀ is true and from known properties of random sampling (such as the Central Limit Theorem, previewed in Point Estimation & Sampling Error). Second, "rare" is a matter of degree, not a bright natural line; the familiar 5% convention (α = 0.05) is a human convention for where to draw the boundary between "unremarkable" and "surprising," not a law of nature. This connects tightly to confidence intervals. A confidence interval, built as in Confidence Intervals for a Mean, is a range of plausible population values consistent with the observed sample. A hypothesis test is the mirror image: it asks whether one specific hypothesized value (the one in H₀) falls inside or outside that range of plausibility. If a 95% confidence interval for a mean difference excludes zero, a two-sided test at α = 0.05 will reject "no difference" — the two frameworks are answering the same underlying question from opposite directions.

Hypothesis testing as a formal method was not derived by pure logic alone — it was forged to solve real, physically constrained experimental problems, and its earliest complete demonstration is one of the most elegant small experiments in the history of science: the "Lady Tasting Tea" experiment, devised by Ronald A. Fisher in the late 1920s and published in his 1935 book The Design of Experiments. The setup: a colleague, Muriel Bristol, claimed she could tell by taste alone whether milk had been poured into a cup before or after the tea. Fisher, skeptical, designed a test rather than dismissing her outright. He prepared eight cups of tea — four with milk added first, four with tea added first — presented in random order, and told her in advance that four of each existed. Her task was to identify all four "milk-first" cups. Fisher's reasoning is a textbook case of the exact logic developed above. His null hypothesis, H₀, was stark and specific: Bristol has no ability to discriminate; her selections are equivalent to picking four cups at random out of eight. This is a precise, falsifiable claim, because "choosing 4 of 8 at random" generates an exact, calculable probability distribution over possible numbers of correct identifications — no vague hand-waving required. Under this null hypothesis, the number of ways to choose 4 cups correctly out of 8, versus all possible ways of choosing any 4 of the 8, can be computed exactly using combinatorics: ``` Total ways to choose 4 cups from 8: C(8,4) = 70 Ways to get all 4 milk-first cups right: C(4,4)·C(4,0) = 1 ``` So under pure guessing, the probability of identifying all four cups correctly is 1/70 ≈ 0.014 — a genuinely rare event, comfortably below the 5% threshold that later became conventional. This is the null distribution in miniature: a complete, exact accounting of what chance alone should produce, computed before looking at what Bristol actually did. The historical record on the actual outcome is not perfectly settled — some accounts say Bristol correctly identified all eight cups (getting all four milk-first cups right necessarily identifies all four tea-first cups too, since there were only four of each), while other retellings vary in detail. What matters for the logic is not the folklore but the method: Fisher had, before the tea was poured, specified an exact null hypothesis, computed its exact sampling distribution by combinatorics, and pre-committed to a rule for how rare a result would need to be to cast doubt on that null. This is the complete hypothesis-testing procedure in its purest, most transparent form — no approximations, no need for a bell curve, just exact counting — which is precisely why Fisher chose it as his pedagogical anchor for a book that would define statistical practice for the rest of the century. The deeper scientific point the tea experiment established is this: randomization plus a precise null hypothesis converts an anecdote into a testable claim. Before Fisher, "she says she can tell the difference" was an unfalsifiable assertion resting on personal conviction. After Fisher's design, it became a claim with a computable, objective, falsifiable prediction — the same transformation that later underlies clinical trials, agricultural field experiments (Fisher's own original context at Rothamsted Experimental Station), and virtually every controlled experiment in modern biology.

Example 1 — a genetics cross with an asymmetric, non-Mendelian-looking outcome. A plant geneticist crosses two heterozygous pea plants expected to follow a simple dominant/recessive pattern for seed color (yellow dominant, green recessive), predicting a classic 3:1 ratio in the F₂ generation. She grows and counts 160 offspring: ``` Observed: 118 yellow, 42 green Expected under 3:1 ratio (H0): 120 yellow, 40 green ``` H₀: the true ratio in the population of offspring is 3 yellow : 1 green (ordinary Mendelian segregation, no distorting effect). Hₐ: the true ratio deviates from 3:1 (something — linked lethal allele, scoring error, segregation distortion — is disturbing it). A chi-square goodness-of-fit statistic compares observed to expected counts: ``` χ² = Σ (observed − expected)² / expected = (118−120)²/120 + (42−40)²/40 = (4/120) + (4/40) = 0.033 + 0.100 = 0.133 ``` Under H₀, this statistic follows a known chi-square distribution (1 degree of freedom for a two-category comparison), and a value this small — far below the conventional cutoff of 3.84 that marks the rare 5% tail for 1 df — sits comfortably in the dense, unremarkable middle of that distribution. The researcher fails to reject H₀: this data is entirely consistent with ordinary 3:1 segregation, and the 118/42 split is exactly the kind of wobble ordinary sampling produces around a true 3:1 ratio in a sample of 160. Note carefully what this does not say — it does not prove the ratio is exactly 3:1, only that this sample gives no reason to doubt it.

Example 2 — a physiological/clinical scenario with a directional suspicion and a small sample. A physiologist suspects that a new beta-blocker lowers resting heart rate below the population-typical value of 72 beats per minute (bpm). She measures 15 patients after treatment: ``` Sample mean: x̄ = 66.8 bpm Sample SD: s = 9.4 bpm n = 15 Hypothesized population mean under H0: μ0 = 72 bpm ``` H₀: μ = 72 bpm (the drug has no effect; patients' true mean resting heart rate remains at the untreated population value). Hₐ: μ < 72 bpm (the drug lowers heart rate — a one-sided alternative, justified because the physiological mechanism of beta-blockade is specifically rate-lowering, not rate-changing in an unknown direction). Standard error of the mean, using the logic from Point Estimation & Sampling Error: ``` SE = s / √n = 9.4 / √15 = 9.4 / 3.873 ≈ 2.427 bpm ``` Test statistic (t, since population SD is unknown and n is small): ``` t = (x̄ − μ0) / SE = (66.8 − 72) / 2.427 = −5.2 / 2.427 ≈ −2.14 ``` With 14 degrees of freedom, a one-sided t-statistic of −2.14 falls in the tail beyond the critical value of roughly −1.76 (the boundary for the rarest 5% of outcomes under H₀, on the low side). This observed result would be uncommon if the drug truly did nothing — comfortably past the conventional threshold for surprise. The physiologist rejects H₀ and concludes the data support a genuine rate-lowering effect. Note the asymmetric inputs matter here: a small n (15) inflates the standard error and demands a larger critical t-value than the familiar 1.96 used for large samples — a direct link back to how sample size governs the width of a confidence interval.

Ronald Aylmer Fisher (1890–1962) is the figure most responsible for turning "how surprising is this data" from an intuitive question into a rigorous, teachable, reproducible method. Trained in mathematics and astronomy at Cambridge, Fisher spent much of his early career at Rothamsted Experimental Station in England, an agricultural research institution overflowing with messy field data — crop yields, fertilizer trials, weather variation — that classical mathematics had no good tools to handle. Confronted with genuinely noisy biological measurements rather than idealized physics problems, Fisher built the statistical machinery almost from scratch: he formalized the null hypothesis, developed analysis of variance (ANOVA), refined the theory of experimental design (including randomization and replication as safeguards against confounding), and popularized the significance-testing framework in his enormously influential 1925 book Statistical Methods for Research Workers and his 1935 The Design of Experiments, home of the tea-tasting story. Fisher was also a leading population geneticist who helped reconcile Mendelian genetics with Darwinian natural selection — a synthesis that itself relied on exactly the sampling-variation reasoning his statistical work formalized. His views on eugenics, common among scientists of his era, are rightly criticized today and are a sober reminder that scientific method and scientific ethics are separate axes of evaluation.

Every randomized clinical trial that determines whether a new drug is approved runs on this exact logic: assume the drug has no effect on the outcome (survival, blood pressure, viral load), then ask how surprising the observed difference between treatment and placebo groups would be under that assumption. Regulatory agencies such as the FDA require pre-registered null hypotheses and pre-specified significance thresholds precisely to prevent researchers from hunting through data after the fact for a "surprising" result that might just be noise. In agricultural biotechnology, field trials of genetically modified crop varieties use this framework to test whether yield, drought tolerance, or pest resistance differs from conventional varieties, guarding against the natural field-to-field variability that would otherwise make every plot "look different" by chance alone. In conservation biology, population geneticists use hypothesis testing to detect whether allele frequencies in an endangered population deviate from Hardy-Weinberg expectations — a signal of inbreeding, genetic drift, or non-random mating — against the null hypothesis of a randomly mating, evolutionarily unremarkable population. In public health surveillance, epidemiologists testing whether a disease cluster around a factory represents a genuine environmental hazard, versus an artifact of the same random clumping that appears anywhere data are collected, rely on precisely the reasoning developed here: assume no elevated risk, then measure how improbable the observed cluster would be under ordinary background variation. In genomics, thousands of hypothesis tests run simultaneously across the genome (testing each gene for differential expression, for instance) forced statisticians to extend this logic further — multiple-testing correction — because running the same 5%-rarity threshold ten thousand times guarantees hundreds of false alarms by chance alone, a direct and important consequence of the core logic taught here.

Ask the tutor: How do we decide, before collecting data, exactly how rare is "rare enough" to reject the null hypothesis — and what happens statistically when that threshold is set too strict or too loose?