Statistical Power & Sample Size
A conservation biologist tests a new fertilizer meant to rescue an endangered orchid. She treats 6 plants and finds no significant growth difference. She concludes the fertilizer does nothing. But the study never had a realistic chance of detecting even a large, biologically important effect. The experiment was not wrong to run. It was simply too small to succeed. Its failure was fated before a single seed was planted. This is the problem of statistical power: the probability that a study will actually detect a real effect, given that one exists. Power is not a philosophical afterthought bolted onto a finished experiment. It is a quantity biologists can calculate in advance. It is determined by exactly four ingredients. These are how big the true effect is, how noisy the biological system is, how strict a false-positive rate we tolerate, and how many observations we collect. Understanding power converts experimental design from guesswork into engineering. The stakes are high across biology. Underpowered clinical trials waste patient risk and money. They also fail to detect drugs that truly work. Underpowered field ecology studies contribute to reproducibility crises. They fill the literature with "no effect found" results that are really "no chance of finding an effect" results. Overpowered studies, conversely, can burn resources. Or worse, they can turn statistically significant but biologically trivial effects into headlines. Power analysis is the tool that keeps effect detection efficient, ethical, and honest. This article builds power from its logical foundation in Type I & Type II Errors. It shows precisely how effect size, variability, alpha, and sample size interlock to determine it. It walks through the classical statistical logic and simulation evidence that cemented power analysis into standard practice. It also gives worked calculations biologists actually perform before collecting data.
- Power = 1 − β = probability of correctly rejecting a false null hypothesis; it depends on effect size, variability (σ), α, and n.
- Larger true effects, lower variability, larger α, and larger n all increase power — but only n is usually under the researcher's control.
- Power and sample size are calculated before data collection, from a specified minimum detectable effect size, not after seeing a non-significant p-value.
- The relationship between n and power is not linear: sample size requirements scale roughly with the square of the effect size's inverse.
- Jacob Cohen formalized power analysis and conventions (small/medium/large effect sizes, the 0.80 power standard) that remain the field norm.
- "Not significant" from an underpowered study is not evidence of no effect — it is often just evidence of insufficient data.
Statistical power is the pre-planned probability of detecting a real effect of a given size, and it can — and should — be calculated before an experiment is run, not diagnosed afterward from a disappointing p-value.
The core idea
A hypothesis test can go wrong in two ways. A Type I error rejects a true null hypothesis (a false positive), controlled by the significance threshold α, conventionally 0.05. A Type II error fails to reject a false null hypothesis (a false negative), with probability denoted β. Power is defined as:
Power is the probability that a test correctly detects a real effect — that is, the probability of rejecting H₀ when H₀ is in fact false and some specific alternative effect size is true. Because β (and hence power) depends on how large the true effect actually is, power is never a single number for an experimental design; it is a curve, evaluated against a hypothesized or minimum-interesting effect size. Four quantities govern power, and they trade off against one another in predictable ways:
| Quantity | Symbol | Effect on power as it increases |
|---|---|---|
| True effect size | δ or Δ | ↑ power (bigger signal, easier to detect) |
| Population variability | σ | ↓ power (more noise obscures signal) |
| Significance threshold | α | ↑ power (looser threshold, easier to reject H₀) |
| Sample size | n | ↑ power (more data shrinks the standard error) |
The crucial insight is that the first three are usually fixed by biology and convention — the effect size is a property of nature, σ a property of the measured trait, and α is fixed at 0.05 by disciplinary norm. Sample size, n, is the only lever the experimenter fully controls. Power analysis is therefore, in practice, almost always sample-size planning: given the effect size worth detecting, the expected variability, and the desired α and power, how many replicates are needed? The reason n matters comes directly from the The Standard Error of the Mean: the standard error of a sample mean shrinks as . Quadrupling n only halves the standard error — power buys precision at a decelerating rate, which is why detecting small effects requires disproportionately large samples.
Mechanism — how power actually arises
Step 1: Two overlapping sampling distributions. Under the null hypothesis, the test statistic (say, a sample mean difference) is centered at 0, with a spread determined by the standard error, . Under the true alternative, the same statistic is centered at the true effect size δ, with the same standard error (assuming equal variances). Power is the area of the alternative distribution that falls beyond the critical value set by α under the null.
Step 2: The critical value carves out the rejection region. For a two-tailed test at α = 0.05, the null distribution's rejection region begins at roughly ±1.96 standard errors from zero (for large n, using the normal approximation; smaller n uses the t-distribution with correspondingly wider critical values). Any observed statistic beyond this cutoff triggers rejection of H₀.
Step 3: Overlap between the two distributions determines β and power. If the alternative distribution's mean (δ) sits far from zero relative to the standard error, only a small sliver of the alternative distribution falls short of the critical value (this sliver is β), and most of it clears the threshold (this majority is power). If δ is small relative to SE, the two distributions overlap heavily, β is large, and power is small.
Step 4: n rescales the whole system. Increasing n shrinks the standard error for both distributions symmetrically, pulling them tighter around their respective centers without moving the centers themselves. Tighter distributions separate more cleanly at fixed δ, shrinking the overlap and increasing power. This is the entire mechanical content of power analysis: n is a knob that controls how tightly clustered your sampling distributions are, and tighter clustering resolves smaller true differences.
Step 5: Standardizing effect size makes calculations general. Because δ and σ both matter, statisticians work with a standardized effect size, Cohen's d:
This dimensionless number expresses the true difference in units of standard deviations, letting sample-size formulas apply across traits measured in wildly different units (grams, millimeters, enzyme units). Jacob Cohen proposed rough conventions:
| Cohen's d | Interpretation |
|---|---|
| 0.2 | small effect |
| 0.5 | medium effect |
| 0.8 | large effect |
These are heuristic benchmarks for fields lacking prior data, not universal biological truths — a d of 0.2 can be enormous for a cancer survival trial and trivial for a leaf-length comparison.
The evidence — how power analysis became standard practice
Power was not always a routine calculation. Neyman and Pearson formalized the Type I/Type II error framework in 1933, defining power mathematically as part of their theory of hypothesis testing, but for decades biological and psychological research proceeded with essentially no attention to whether studies had adequate sample sizes to detect anything.
Jacob Cohen's landmark 1962 survey. The turning point was an empirical audit, not a new formula. Jacob Cohen (1962) systematically reviewed every study published in a single year (1960) of the Journal of Abnormal and Social Psychology — 70 studies. For each, he computed the statistical power the study actually had to detect small, medium, and large effects (using his own newly defined d conventions), assuming α = 0.05.
What he found: the median power to detect a medium effect size was only about 0.48 — worse than a coin flip. For small effects, median power was a dismal 0.18. Even for large effects, many studies fell well short of adequate power. This meant the majority of published psychological research was, by design (whether or not by intention), unlikely to detect real effects of realistic sizes even when those effects existed. Non-significant results filling the literature were largely uninformative — not evidence against real effects, but symptoms of studies that never had a fair chance.
What this proved. Cohen's audit demonstrated empirically that sample sizes were being chosen by convention or convenience (common lab group sizes, available grant money, historical habit) rather than by any principled calculation tied to the effect the researcher actually cared about detecting. It showed that "failure to reject H₀" was being routinely and wrongly interpreted as "no effect exists," when in fact it was frequently "the study lacked the power to see it." This single audit, repeated for other fields in later decades (medicine, ecology, neuroscience each have their own versions), catalyzed the now-standard practice of a priori power analysis: specifying the smallest effect size worth detecting before running an experiment, and calculating the n required to detect it with conventionally accepted power (usually 0.80) at a given α.
Later replication-era confirmation. The logic Cohen established was validated again during the reproducibility crisis of the 2010s. Button et al. (2013, Nature Reviews Neuroscience) surveyed neuroscience studies and found median statistical power around 0.21, meaning most published neuroscience findings, even when true, had less than a one-in-five chance of being detected in any single study — a direct descendant of Cohen's original diagnosis, replicated across a different field half a century later. This convergent evidence — from psychology in 1962 and neuroscience in 2013 — is what elevated power analysis from a statistician's abstraction to a mandatory step in modern grant applications, IRB/IACUC protocols, and pre-registration documents.
Worked examples
Example 1 — Planning a two-group physiology experiment (unequal groups, asymmetric costs). A physiologist wants to know whether a drug raises resting heart rate in zebrafish. Pilot data suggest:
- Population standard deviation of heart rate, σ ≈ 12 beats/min (assumed equal in both groups)
- The smallest biologically meaningful increase worth detecting: δ = 6 beats/min
- α = 0.05 (two-tailed), desired power = 0.80
Step 1 — standardized effect size:
Step 2 — apply the standard two-sample sample-size formula for equal-size groups:
With α/2 = 0.025 → z = 1.96, and power = 0.80 → z_β = 0.84:
So n ≈ 63 fish per group, 126 total, to have an 80% chance of detecting a true 6 bpm difference at α = 0.05. Now suppose fish are expensive and the lab can only afford 30 per group. Solving the formula backward for the detectable effect size at n = 30, power = 0.80:
With only 30 fish per group, the experiment can reliably detect only effects of about 8.7 bpm or larger — a true effect of 6 bpm would likely be missed. This is the essential trade: the researcher must either accept a higher minimum detectable effect, increase n, reduce variability (e.g., standardize water temperature to shrink σ), or accept lower power.
Example 2 — Diagnosing why a field ecology study found "no effect" (post-hoc reasoning, asymmetric groups). An ecologist compares seed dispersal distance for a shrub in grazed (n₁ = 8 plots) versus ungrazed (n₂ = 22 plots) grassland — unequal, opportunistic sample sizes reflecting land access, not design. Observed means: grazed = 4.1 m, ungrazed = 5.3 m, pooled SD ≈ 2.8 m. The two-sample t-test gives p = 0.11 — not significant at α = 0.05. The ecologist is tempted to write "grazing has no effect on dispersal distance." Check power retrospectively (as a design lesson, not as a formal "observed power," which is discouraged — instead ask: what effect size could this design have reliably detected?). Standardized effect size actually observed:
This is close to Cohen's "medium" benchmark (0.5) — plausibly a real and biologically meaningful effect, not a null one. Using the harmonic-mean-adjusted sample size for unequal groups (n₁=8, n₂=22):
Compare against the ~63-per-group benchmark style calculation for d = 0.5 target power 0.80 (roughly n ≈ 64 per group for that effect size using the same formula structure) — the study's effective sample size of under 12 is wildly inadequate. Recomputing power at d = 0.43, n₁=8, n₂=22 (using standard power tables) gives power on the order of 0.25–0.30 — meaning even if the true effect is exactly what was observed, this design had roughly a 1-in-4 chance of detecting it. The correct conclusion is not "grazing has no effect," but "this study was underpowered to detect a plausible medium-sized effect; the data are consistent with a real difference that this design could not reliably confirm." A follow-up study should target roughly 60–65 plots per treatment.
Common misconceptions & traps
| Misconception | Correction |
|---|---|
| "Non-significant p-value proves the null hypothesis is true (no effect exists)." | It only shows the data didn't clear the α threshold; an underpowered study will produce this result routinely even when a real effect exists. |
| "Power analysis should be done after the experiment, using the observed effect size, to explain a non-significant result (post-hoc/'observed' power)." | Post-hoc power computed from the observed effect is mathematically just a rescaling of the p-value and adds no new information; power should be planned a priori from a scientifically meaningful effect size. |
| "A bigger sample size always makes a study 'better'." | Excess power can make trivially small, biologically meaningless effects statistically significant — see Statistical vs Biological Importance. |
| "Power depends only on sample size." | Power depends jointly on n, effect size, variability, and α — increasing n cannot fully compensate for a true effect near zero. |
| "80% power means the effect is 80% likely to be real." | Power is a pre-experiment property of the design conditional on an assumed true effect; it says nothing about the probability the null or alternative is true after data are collected. |
| "Reducing α to be 'more rigorous' always improves a study." | Lowering α (e.g., to 0.01) reduces Type I error risk but simultaneously reduces power, increasing Type II error risk, unless n is also increased. |
The scientist behind it
Jacob Cohen (1923–1998) was an American psychologist and statistician at New York University whose career, unusually for a methodologist, began in applied clinical and personality assessment work before pivoting to become one of the most influential statistical reformers in the behavioral sciences. He did not invent the mathematics of power — that lineage runs through Jerzy Neyman and Egon Pearson's 1933 formalization of hypothesis testing — but Cohen turned power from an abstract quantity buried in statistical theory into a practical, computable, moral obligation for empirical researchers. His 1962 audit of a year's worth of published psychology papers exposed, with uncomfortable numerical precision, that the field's studies were mostly too small to find what they were looking for. He followed this with Statistical Power Analysis for the Behavioral Sciences (1969, expanded 1988), which provided tables and formulas that let any researcher, without heavy mathematics, look up the sample size needed for a target power — a book credited with changing grant-review and journal-editorial standards across psychology, medicine, ecology, and education.
Origins · 1 minJacob Cohen and how this was discovered
Where this matters today
Clinical trial design. Regulatory agencies (FDA, EMA) require pre-specified power calculations in trial protocols; a trial's target sample size is legally and ethically justified by a stated minimum clinically important difference, not chosen by convenience. Underpowered trials can expose patients to experimental risk without any realistic chance of a conclusive answer, which institutional review boards treat as an ethical violation, not just a statistical inefficiency.
Conservation and wildlife biology. Monitoring programs for endangered species (population viability analyses, mark-recapture surveys) must be powered to detect population declines of a specified magnitude within a specified time window — an underpowered monitoring program can let a real decline continue undetected for years, delaying conservation action.
Genome-wide association studies (GWAS). Because individual genetic variants typically explain tiny fractions of trait variance (small Cohen's d equivalents), GWAS require sample sizes in the tens or hundreds of thousands; power calculations directly dictate consortium sizes and multi-site collaboration structures in human genetics.
Preclinical and animal research ethics. The 3Rs framework (Replacement, Reduction, Refinement) explicitly incorporates power analysis: using fewer animals than needed to reach adequate power wastes the animals used (their suffering produces no reliable answer), while using more than needed wastes lives unnecessarily — power analysis defines the ethically and scientifically optimal n.
Agricultural field trials. Crop breeding trials testing new cultivars or fertilizer regimes against high plot-to-plot soil variability use power analysis to decide plot replication numbers, balancing land and labor costs against the ability to detect yield improvements worth commercializing.
Public health surveillance. Epidemiological studies screening for associations between environmental exposures and disease outcomes rely on power calculations to determine cohort sizes, especially when the outcome (e.g., a rare cancer) has low baseline incidence, requiring very large n to achieve any meaningful power at all.
Ask the tutor: How do I choose the right minimum effect size (δ) to plug into a power calculation when no pilot data exist yet?
What exactly is the difference between power and confidence level (1 − α)?
Confidence level, 1 − α, describes the test's behavior when the null hypothesis is true: it is the probability of correctly failing to reject H₀ (i.e., not making a Type I error) across repeated sampling. Power describes the test's behavior when the null hypothesis is false: it is the probability of correctly rejecting H₀ when a specific real effect exists. They govern opposite scenarios and opposite error types — confidence level bounds false positives, power quantifies the ability to avoid false negatives. Both are pre-specified properties of the testing procedure and sample design, not judgments made about a particular completed dataset.
Why is the conventional target power set at 0.80 rather than 0.95, to match the usual α = 0.05?
This is a convention, not a mathematical necessity, popularized by Jacob Cohen. The reasoning is a pragmatic asymmetry: Type I errors (false positives) are traditionally treated as more costly to the scientific literature (introducing false "discoveries"), so α is kept strict at 0.05, while Type II errors (false negatives, missed real effects) are treated as costly but somewhat more tolerable, especially since failed studies can be repeated. Setting β at 0.20 (power = 0.80) roughly reflects accepting Type II error at four times the rate of Type I error. Some fields — clinical trials for serious diseases, for instance — routinely demand higher power (0.90 or 0.95) given the cost of missing a real treatment effect.
Can a study ever have "too much" power? Is that a problem?
Yes, in a specific sense. Extremely large samples give a study power to detect almost any nonzero effect, no matter how tiny — including differences too small to matter biologically, clinically, or ecologically. A drug that lowers blood pressure by 0.3 mmHg can become "highly significant" (p < 0.0001) with a large enough trial, even though no physician would prescribe it for that effect alone. This is why modern practice pairs significance testing with reporting of effect sizes and confidence intervals, distinguishing statistical detectability from practical importance — the subject of Statistical vs Biological Importance.
How do I estimate σ (variability) before running the study, since power analysis needs it as an input?
Three standard sources are used, in order of preference: (1) pilot data collected on a small trial run of the actual system; (2) published values of the same or a closely related measurement from prior studies in the literature; (3) a conservative estimate from range or expert judgment when no data exist (e.g., using range/4 as a rough SD approximation for roughly normal traits). Because power calculations are sensitive to σ, researchers often perform sensitivity analyses across a plausible range of σ values, or run an interim analysis partway through data collection to re-estimate variability and adjust the target n.
Does power analysis apply only to two-sample t-tests, or does it generalize to other statistical tests?
Power analysis generalizes to essentially every inferential test — ANOVA, regression, chi-square tests of association, correlation tests, survival analysis, and mixed models — though the specific formula for computing n changes with the test's structure. The unifying logic is always the same: specify a minimum meaningful effect size (a mean difference, a correlation coefficient, an odds ratio, a slope), an estimate of variability, α, and target power, then solve for n. Modern practice typically uses dedicated statistical software (e.g., G*Power, R's `pwr` package) rather than manual formulas, especially for complex designs like factorial ANOVA or multilevel models.
If my study is already complete and underpowered, is there anything statistically valid I can do?
Yes. Report the effect size with its confidence interval rather than relying on the p-value alone — the confidence interval directly shows the range of effect sizes still consistent with the data, making clear that "no significant difference" may simply reflect a wide, uninformative interval rather than a precise null result. Meta-analysis is the other principled remedy: combining the underpowered study with others measuring the same effect increases the effective sample size and can resolve what any single small study could not. What is not valid is recomputing "observed power" from the study's own effect size and treating it as new evidence — this simply restates the p-value in different units.
How does power analysis relate to Type I and Type II errors covered earlier?
Power is defined directly in terms of the Type II error rate: Power = 1 − β. The two concepts are two views of the same 2×2 table of test outcomes (reject/fail to reject H₀, crossed with H₀ true/false). Understanding Type I & Type II Errors establishes what can go wrong in a single test; power analysis asks the forward-looking design question of how likely the Type II error is to occur given a specific real effect, and uses that probability to choose a sample size that keeps the Type II error rate acceptably low before any data exist.