Reproducibility and P-Hacking

A p-value answers a narrow question: assuming the null hypothesis is true, how surprising is data at least this extreme? The entire logic of hypothesis testing rests on the assumption that the test was specified — the hypothesis, the sample size, the exclusion criteria, the outcome variable, the statistical model — before the data were seen. When any of these are instead chosen or adjusted after looking at the data, in whatever way makes the result look best, the nominal p-value no longer reflects the true probability of a false positive. This is p-hacking, and it does not require dishonesty: a researcher genuinely believing each individual choice was "the right way to analyze the data" can still, collectively, be running an unacknowledged multiple-testing problem against themselves. The technical term for the raw material of p-hacking is researcher degrees of freedom — the large number of seemingly minor, defensible decisions available at every stage of a study. Should an "outlier" more than 2 SD from the mean be excluded, or 3 SD, or not at all? Should sex, age, or batch be included as a covariate? Should the primary outcome be measured at day 7 or day 14, or both? Should the experiment stop at n = 20 per group, or continue until something looks significant? Any single one of these choices is often statistically legitimate in isolation. The trouble is combinatorial: if there are 20 plausible analysis pipelines and a researcher (consciously or not) gravitates toward the one that crosses the significance threshold, the effective probability of finding some "significant" result under the null hypothesis is vastly higher than 5%, even though only one pipeline — and one p-value — is ever reported to readers. It is essential to distinguish p-hacking from two things it is not. It is not the same as fraud (fabricating or falsifying data), which is rarer and easier to detect. It is also not the same as legitimate, principled exploratory data analysis — looking broadly at data to generate new hypotheses is a valuable scientific activity, but only if it is honestly labeled as exploratory or hypothesis-generating rather than presented as a single confirmatory test of a pre-specified hypothesis. The harm of p-hacking comes specifically from undisclosed flexibility masquerading as confirmatory inference — reporting a result as if it were the single outcome of a single pre-planned test, when in fact it was selected, after the fact, from many candidate analyses.

Step 1: The null hypothesis is actually true (or the true effect is null/negligible). In biology this happens constantly — most candidate genes are not really associated with a given disease, most compounds screened do not really affect the pathway of interest, most subgroup differences are noise. Under a true null, a correctly specified single test rejects the null (p < 0.05) only 5% of the time, by construction.

Step 2: The researcher has several "reasonable" ways to test the same broad question. Suppose there are two candidate outcome measures, two plausible covariate sets, and a choice of whether to exclude two ambiguous samples. That is 2 × 2 × 2 = 8 distinct, individually-defensible analysis pipelines applied to the same underlying dataset.

Step 3: The analyses are not independent, but they are not identical either. Because each pipeline uses overlapping data and similar statistical machinery, each has roughly (not exactly) a 5% chance of crossing p < 0.05 under the null, and these chances are positively correlated across pipelines. The key quantitative fact is that even modest correlation is not enough to prevent inflation: the family-wise probability that at least one of the 8 pipelines yields p < 0.05, purely by chance, climbs well above 5% — often into the 20–60% range depending on how correlated the pipelines are.

Step 4: The researcher reports only the pipeline that "worked." Crucially, no single step in this process need be dishonest. Excluding an outlier can be entirely justified on its own terms; adding a covariate can be entirely justified on its own terms. The problem is selection after the fact, combined with silence about the alternatives that were tried. The published paper shows one test, one p-value, and no trace of the seven roads not taken.

Step 5: The reported p-value is systematically too small relative to what it is claimed to represent. A reader interprets "p = 0.032" as "a 3.2% chance of a false positive under this exact pre-specified test." The true chance of some false positive somewhere in the search the researcher (often unconsciously) conducted may be many times higher. The gap between nominal α and actual false-positive rate is the entire mechanism of p-hacking. A closely related practice is optional stopping: continuing to collect data and re-running the test after each new batch, stopping as soon as p < 0.05 is reached. Because a random walk of p-values will cross any threshold eventually with high probability if you keep checking, repeated "peeking" at accumulating data, without a pre-specified stopping rule, is itself a powerful hidden multiple-comparisons problem — often the single most potent form of p-hacking because it can be repeated many times within one experiment.

The second part of the mechanism concerns what happens across many independent studies of the same true-null effect, which is really the same logic scaled up to the level of a whole field, and connects p-hacking back to the The Multiple-Testing Problem at the level of the literature rather than the single paper. Suppose 100 laboratories worldwide each test, independently, whether a candidate biomarker predicts disease — and suppose, in truth, it does not. By chance alone, about five of those 100 studies will find p < 0.05 in the "right" direction. Because of publication bias — the tendency of journals, reviewers, and researchers themselves to preferentially write up and accept "significant," novel-looking findings, and to file away "null" results in a drawer — those five false positives are disproportionately likely to become the visible published literature, while the 95 null results vanish. This is sometimes called the file-drawer problem. A field can end up with a published literature dominated by chance flukes, not because any individual researcher cheated, but because the filter connecting experiments to publications is itself biased toward "significant" results, and p-hacking supplies extra fuel by helping marginal, near-null findings cross the publication threshold that would otherwise have kept them in the file drawer.

Simmons, Nelson & Simonsohn (2011), "False-Positive Psychology." This paper is the canonical demonstration of p-hacking's power, and though published in a psychology journal, its logic applies directly to any experimental biology with flexible analysis pipelines (behavioral assays, cell-culture experiments, small-animal studies). The authors identified four common, individually-defensible researcher degrees of freedom: (1) trying two or more outcome variables and reporting only the significant one; (2) deciding whether to collect more data after an initial peek; (3) controlling for gender or other covariates only when it helps; (4) reporting only a subset of experimental conditions. Using computer simulation of data drawn from a true null (no real effect whatsoever), they showed that exploiting just these four degrees of freedom pushed the false-positive rate from the nominal 5% up past 60% — meaning a researcher using this flexibility, testing a completely fake effect, would find a "statistically significant" result more often than not. In a companion empirical demonstration, they used exactly this kind of flexibility to "prove," using real collected data, that listening to a particular song about aging changed people's actual chronological age — an outcome known to be impossible — illustrating that p < 0.05 obtained through flexible analysis carries almost no evidentiary weight.

Begley & Ellis (2012), Nature. Researchers at the biotechnology company Amgen attempted to replicate 53 landmark preclinical cancer biology studies — papers that had shaped drug-development strategy across the field. Working sometimes with the original authors and their exact reagents and protocols, Amgen's team could confirm the original results in only 6 of 53 studies (about 11%). The failures were not attributed to fraud but to a combination of small sample sizes, selective reporting of favorable replicates, inadequate blinding, and the kind of flexible, unreported analytical choices that define p-hacking. The paper's stark headline number — roughly 89% of high-profile findings failing to replicate — helped trigger the "reproducibility crisis" conversation across all of experimental biology, not only cancer research.

The Open Science Collaboration (2015), Science, "Estimating the reproducibility of psychological science." A coordinated effort of over 270 researchers repeated 100 published psychology experiments using the original materials and much larger, pre-registered replication protocols. While 97% of the original studies had reported a statistically significant result, only 36% of the direct replications did, and the average replicated effect size was roughly half the originally published size. Because the replication protocols were pre-registered — locking in hypotheses, sample sizes, and analysis plans before data collection — this project is also the clearest large-scale demonstration that preregistration itself changes outcomes: the very act of removing researcher degrees of freedom from the replication attempt is part of why so many original "discoveries" failed to reappear.

John, Loewenstein & Prelec (2012), "Measuring the prevalence of questionable research practices." Using an anonymous survey with an incentive structure designed to encourage honest self-report even of unethical behavior, the authors found that a majority of practicing research psychologists admitted to at least one questionable research practice, such as deciding whether to collect more data after checking if results were significant (reported by more than half of respondents), or selectively reporting studies that "worked." This evidence matters because it shows p-hacking is not a rare pathology of a few bad actors but a widespread, often normalized set of habits — precisely because each individual practice can feel statistically innocuous in the moment it is used. Together these three lines of evidence — controlled simulation showing the mathematics of inflation, biotechnology-industry replication attempts showing catastrophic real-world failure rates, and survey evidence showing the underlying practices are common — triangulate on the same conclusion: flexible, undisclosed analysis reliably manufactures false positives at a scale large enough to distort entire fields.

Example 1 — quantifying the inflation from three independent binary choices. Suppose a lab has three genuinely independent researcher degrees of freedom, each of which, under the true null, has an approximately 5% chance of independently producing p < 0.05 (this independence assumption is a simplification — in reality choices are correlated — but it isolates the arithmetic cleanly):

• Include or exclude 2 flagged outlier samples (2 choices)
• Use raw values or log-transform the outcome (2 choices)
• Test the treatment effect with or without a batch covariate (2 choices)

That gives 2 × 2 × 2 = 8 independent-seeming pipelines. If each has probability 0.05 of a false positive and they were fully independent, the probability that at least one pipeline shows p < 0.05 is:

So even under full independence, the chance of stumbling on "significance" somewhere among 8 plausible pipelines is about 33.7%, nearly seven times the nominal 5%. In practice, correlated pipelines (they share most of the same data) reduce this somewhat compared to fully independent tests, but Simmons et al.'s simulations, which modeled realistic correlation among just four researcher degrees of freedom, still found inflation to roughly 61% — demonstrating that correlation attenuates but does not remotely eliminate the effect. This is the same combinatorial logic as The Multiple-Testing Problem, applied not across genes or SNPs but across a single researcher's private menu of analysis choices.

Example 2 — a preregistration decision in a CRISPR knockout screen. A lab wants to test whether knocking out gene X changes cell proliferation rate in a cancer cell line, relative to a scrambled-guide control. They plan n = 6 biological replicates per group, measured by an automated cell counter at 72 hours, analyzed with a two-sample t-test at α = 0.05. Path A — preregistered, confirmatory. Before running the experiment, the lab publicly time-stamps (e.g., via the Open Science Framework) the hypothesis ("knockout reduces proliferation"), the exact outcome variable (cell count at 72 h), the sample size (n = 6/group, fixed in advance), the exclusion rule (none, unless a well shows visible contamination under a pre-specified blinded check), and the single statistical test (two-sided t-test, α = 0.05). They run the experiment once. Suppose they obtain t = 2.61, df = 10, giving p = 0.026. Because every choice was fixed in advance, this p-value can be interpreted at face value: under the null of no true effect, results this extreme would occur about 2.6% of the time. Path B — same data, p-hacked. Instead, suppose after seeing a "promising but not quite significant" result at n = 6 (p = 0.083), the lab adds four more replicates (now n = 10), checks again (p = 0.061), tries measuring at 96 hours instead of 72 (p = 0.049), and reports only this last analysis, with no mention of the earlier failed attempts. The final reported "p = 0.049 (n = 10, 96 h)" looks like ordinary science but is the product of at least three sequential hidden tests plus optional stopping. Using logic parallel to Example 1, the true probability that some combination along this search path crosses p < 0.05, even under a genuinely null effect of the knockout, is very plausibly above 20–30%, not 5%. The two paths can start from the identical raw wells of cells; only the reporting discipline differs, and only Path A licenses the textbook interpretation of the p-value.

The modern reproducibility movement in experimental biology and psychology does not trace to one solitary discoverer in the way Mendelian genetics traces to Mendel; it is better understood as a diagnosis made collectively and then formalized by a handful of methodologists. Among the most influential is John P. A. Ioannidis, a physician-scientist and meta-researcher whose 2005 paper "Why Most Published Research Findings Are False" used a mathematical model of statistical power, bias, and the number of teams testing a hypothesis to argue that, across many fields of biomedicine, the majority of published "positive" findings were likely to be false positives — a claim later borne out empirically by the Begley & Ellis and Open Science Collaboration replication efforts. Ioannidis's model formalized exactly the mechanism this article describes: as the number of hypotheses tested rises, as the flexibility of analysis rises, and as publication bias toward positive results rises, the post-study probability that a "significant" finding is true falls, sometimes drastically. His subsequent empirical audits of specific literatures — nutrition epidemiology, gene-association studies, neuroimaging — repeatedly found the same pattern: small studies, flexible analyses, and strong publication-bias filters together generate a literature far noisier than its p-values suggest. Alongside Ioannidis, the psychologists Joseph Simmons, Leif Nelson, and Uri Simonsohn supplied the quantitative demonstration described above, and Brian Nosek, co-founder of the Center for Open Science, led the large coordinated replication projects and championed preregistration and "registered reports" — a publishing format in which a journal reviews and accepts a study's hypotheses and methods before the results are known, structurally removing the incentive to p-hack because publication no longer depends on the outcome.

In preclinical drug development, pharmaceutical and biotechnology companies now routinely budget for internal replication of academic findings before committing to costly clinical programs, directly because of the Begley & Ellis-style failure rates; some funders and companies now require preregistered protocols from academic collaborators before investing. In clinical trials, preregistration is legally mandated in many jurisdictions (e.g., ClinicalTrials.gov registration is required before enrollment for trials seeking FDA approval), specifically to prevent post-hoc switching of primary outcomes — a well-documented historical form of p-hacking in medicine, in which a trial that failed on its planned primary endpoint was published emphasizing a secondary endpoint that happened to reach significance. In genomics and high-throughput biology, where thousands of genes or variants are tested simultaneously, preregistration of the analysis pipeline (which normalization method, which multiple-testing correction, which covariates) prevents the same p-hacking logic from operating at the level of pipeline choice rather than single-hypothesis choice, complementing the statistical corrections taught in The Multiple-Testing Problem. In ecology and conservation biology, where field data collection is expensive and sample sizes are often small, journals increasingly offer "registered report" tracks so that null results — a species does not respond to a management intervention, a pollutant does not affect reproduction — get published and inform policy, rather than vanishing into the file drawer. In funding and hiring, granting agencies (e.g., the U.S. National Institutes of Health) have introduced rigor-and-reproducibility criteria into review, and some university and journal policies now require data and code sharing so that others can audit the analytical path actually taken — a direct, practical countermeasure to the invisibility that makes p-hacking possible in the first place. Understanding this mechanism also motivates the shift toward Bayesian Inference in Biology: An Introduction, where prior evidence directly downweights implausible claims regardless of a single p-value, and toward Simulation and Resampling Methods, which let researchers pre-register and stress-test an entire analysis pipeline's error rate before ever touching real data.

Ask the tutor: If I already ran an exploratory analysis and found something interesting, how should I honestly report it without it counting as p-hacking?