Hypothesis Testing: Power and Multiple Testing

Power is the probability that a hypothesis test correctly rejects a false null hypothesis:

Power = 1 - β = P (reject H_{0} ∣ H_{1} true)

. Multiple testing addresses the inflation of error rates that occurs when many hypotheses are tested simultaneously — testing 20 true nulls at

α = 0.05

yields a 64% chance of at least one false positive. Used everywhere from clinical trials to genomics; formalized chiefly by Neyman, Pearson, Bonferroni, and later Benjamini–Hochberg (1995).

The power of a statistical test is its sensitivity: the probability of detecting a genuine effect when one exists. It complements the significance level $α$ (the false-positive rate) and equals $1 - β$ , where $β$ is the Type II error probability. Power matters because an underpowered study wastes resources and yields ambiguous null results, while overpowered studies detect trivial effects. Multiple testing arises when several hypotheses are examined together: each test carries its own error risk, and these accumulate, so the probability of some false discovery balloons. A surprising fact: even with all null hypotheses true, performing enough independent tests makes a "significant" result almost certain — which is why correction procedures (Bonferroni, Holm, Benjamini–Hochberg) are essential for honest inference.

The Core Insight

A hypothesis test trades off two errors. Lowering $α$ (false positives) raises $β$ (false negatives) unless you compensate with sample size or effect size. Power quantifies the false-negative side. When many tests are run, the per-comparison error $α$ is no longer the right thing to control; you must control a family-wide quantity — either the chance of any error (FWER) or the expected proportion of errors among rejections (FDR). The deep tension: aggressive multiplicity correction protects against false discoveries but destroys power. The entire field is a balancing act between these two failure modes.

Formal Statement

The Two Errors and Power

For a test of $H_{0}$ against $H_{1}$ with rejection region $R$ :

Type I error (false positive): rejecting $H_{0}$ when it is true. Probability $α = P (reject H_{0} ∣ H_{0})$ .
Type II error (false negative): failing to reject $H_{0}$ when it is false. Probability $β = P (fail to reject H_{0} ∣ H_{1})$ .
Power: the probability of correctly rejecting a false $H_{0}$ .

Power (θ) = 1 - β (θ) = P_{θ} (X \in R), θ \in Θ_{1}

⟳Power is just the probability of landing in the rejection region, computed under the true (alternative) parameter

θ

, not under

H_{0}

Here $X$ is the data, $R$ the rejection region, $Θ_{1}$ the alternative parameter set, and $θ$ the true parameter value.

Power of the One-Sample z-Test

For testing $H_{0} : μ = μ_{0}$ vs $H_{1} : μ = μ_{1} > μ_{0}$ with known variance $σ^{2}$ and sample size $n$ , at level $α$ , the power is:

Power = Φ (\frac{μ _{1} - μ _{0}}{σ / n} - z_{1 - α})

⟳Reject when

\overset{ˉ}{X} > μ_{0} + z_{1 - α} σ / n

; standardize

\overset{ˉ}{X}

under

μ = μ_{1}

(\overset{ˉ}{X} - μ_{1}) / (σ / n) \sim N (0, 1)

, then read off the tail probability.

Φ

is the standard normal CDF,

z_{1 - α}

the

(1 - α)

quantile, and the term

δ = (μ_{1} - μ_{0}) n / σ

is the noncentrality parameter — the standardized effect size scaled by

n

Multiple Testing Error Rates

Suppose $m$ hypotheses are tested, of which $m_{0}$ nulls are true. Let $V$ = number of true nulls rejected (false discoveries), $S$ = true alternatives rejected, $R = V + S$ = total rejections.

Family-Wise Error Rate: $FWER = P (V \geq 1)$ — probability of any false positive.
False Discovery Rate: $FDR = E [\frac{V}{R}]$ (with $V / R \equiv 0$ when $R = 0$ ) — expected proportion of false positives among rejections.

FWER_{indep} = 1 - (1 - α)^{m} \approx m α (α small)

⟳Under

m

independent true nulls each tested at

α

, the probability that none is rejected is

(1 - α)^{m}

; complement gives the chance of at least one false rejection. First-order Taylor expansion gives

m α

Neyman–Pearson Lemma

Among all tests of a simple null $H_{0} : θ = θ_{0}$ versus a simple alternative $H_{1} : θ = θ_{1}$ at level $α$ , the likelihood ratio test that rejects when

Λ (x) = \frac{L ( θ _{1} ∣ x )}{L ( θ _{0} ∣ x )} > k

⟳Choose threshold

k

so the size equals

α

; any other level-

α

test can be shown to have rejection region overlapping less with the high-likelihood-ratio region, hence less power.

is most powerful — no other level- $α$ test has greater power. Here $L (θ ∣ x)$ is the likelihood and $k$ is set so $P_{θ_{0}} (Λ > k) = α$ .

Bonferroni Inequality

If each of $m$ tests is conducted at level $α / m$ , then the family-wise error rate is controlled:

FWER = P (i = 1 ⋃ m A_{i}) \leq i = 1 \sum m P (A_{i}) = m \cdot \frac{α}{m} = α

⟳

A_{i}

is the event "test

i

falsely rejects." Boole's inequality (union bound) bounds the probability of a union by the sum of probabilities — no independence required.

Why It Holds / Derivation

Power of the z-Test (Equation 2)

Step 1. The level-

α

rejection rule for

H_{1} : μ > μ_{0}

rejects when

\overset{ˉ}{X} > μ_{0} + z_{1 - α} \frac{σ}{n} .

Step 2. Power is the probability of this event computed under the true mean

μ_{1}

Power = P_{μ_{1}} (\overset{ˉ}{X} > μ_{0} + z_{1 - α} \frac{σ}{n}) .

Step 3. Under

μ = μ_{1}

, standardize:

Z = (\overset{ˉ}{X} - μ_{1}) / (σ / n) \sim N (0, 1)

. Subtract

μ_{1}

from both sides inside the probability:

Power = P (Z > \frac{μ _{0} - μ _{1}}{σ / n} + z_{1 - α}) .

Step 4. Flip the inequality using symmetry

P (Z > c) = Φ (- c)

Power = Φ (\frac{μ _{1} - μ _{0}}{σ / n} - z_{1 - α}) . ■

This shows power increases with effect size $∣ μ_{1} - μ_{0} ∣$ , sample size $n$ , and decreasing $σ$ .

Why FWER Inflates (Equation 3)

Step 1. Let

m

true nulls be tested independently, each rejected with probability

α

under

H_{0}

. Step 2. The event "no false rejection" requires every test to fail to reject:

P (V = 0) = i = 1 \prod m (1 - α) = (1 - α)^{m} .

Step 3. FWER is the complement:

FWER = 1 - (1 - α)^{m} .

Step 4. For

m = 20, α = 0.05

FWER = 1 - 0.9 5^{20} = 1 - 0.358 = 0.642

. A 64% chance of at least one false alarm despite each test being "valid."

■

Jerzy Neyman & Egon Pearson | 1894–1981 / 1895–1980

Neyman (Polish-American) and Pearson (British, son of Karl Pearson) collaborated from 1928 to develop the decision-theoretic foundation of hypothesis testing. Their 1933 paper introduced the explicit distinction between Type I and Type II errors, the concept of power, and the Neyman–Pearson Lemma identifying the most powerful test. This reframed Fisher's significance testing into a formal accept/reject framework with controllable error rates — the dominant paradigm of 20th-century applied statistics. Neyman also founded confidence-interval theory; Pearson edited Biometrika for decades.

Building Intuition

Picture two overlapping bell curves: one centered at $μ_{0}$ (the null) and one at $μ_{1}$ (the truth). You draw a vertical line — the critical value — and reject whenever data falls right of it.

The area of the null curve to the right of the line is $α$ (false positives).
The area of the alternative curve to the right of the line is power.
The area of the alternative curve to the left is $β$ (misses).

Four levers move power: (1) slide the curves apart (bigger effect), (2) shrink them (more data / less noise), (3) move the line left (raise $α$ — but you trade off more false positives), (4) use a one-sided test if direction is known.

For multiple testing, imagine playing the lottery once vs. buying 1000 tickets. Each ticket (test) rarely wins (false positive), but buy enough and a "win" becomes near-certain. Correction shrinks each ticket's win-probability so the whole batch stays honest.

Power, Type I, and Type II errors as overlapping distributions

<svg viewBox="0 0 640 360" xmlns="http://www.w3.org/2000/svg">
  <rect width="640" height="360" fill="#0a0e17"/>
  <!-- axes -->
  <line x1="50" y1="300" x2="600" y2="300" stroke="#94a3b8" stroke-width="1.5"/>
  <text x="320" y="335" fill="#e2e8f0" font-size="13" text-anchor="middle">test statistic</text>
  <!-- null curve centered at x=230 -->
  <path d="M 70,300 Q 150,300 230,90 Q 310,300 390,300" fill="none" stroke="#94a3b8" stroke-width="2"/>
  <text x="180" y="120" fill="#94a3b8" font-size="12">H₀ (null)</text>
  <!-- alternative curve centered at x=400 -->
  <path d="M 240,300 Q 320,300 400,110 Q 480,300 560,300" fill="none" stroke="#d4a853" stroke-width="2.5"/>
  <text x="450" y="135" fill="#d4a853" font-size="12">H₁ (true)</text>
  <!-- critical line at x=320 -->
  <line x1="320" y1="60" x2="320" y2="300" stroke="#e2e8f0" stroke-width="1.5" stroke-dasharray="5,4"/>
  <text x="320" y="52" fill="#e2e8f0" font-size="12" text-anchor="middle">critical value</text>
  <!-- alpha region (null tail right of line) -->
  <path d="M 320,300 Q 355,300 390,300 Q 355,280 320,240 Z" fill="#94a3b8" opacity="0.4"/>
  <text x="358" y="290" fill="#e2e8f0" font-size="12">α</text>
  <!-- power region (alt right of line) -->
  <path d="M 320,300 L 560,300 Q 480,300 400,110 Q 360,200 320,240 Z" fill="#d4a853" opacity="0.35"/>
  <text x="455" y="250" fill="#e2e8f0" font-size="12">Power = 1−β</text>
  <!-- beta region (alt left of line) -->
  <text x="265" y="285" fill="#e2e8f0" font-size="12">β</text>
  <path d="M 240,300 Q 280,300 320,240 Q 280,290 240,300 Z" fill="#475569" opacity="0.5"/>
</svg>

Worked Examples

Sample Size for Target Power

Given: A drug is expected to lower blood pressure by

μ_{1} - μ_{0} = 5

mmHg. Population SD

σ = 12

mmHg. We want

α = 0.05

(one-sided) and power

= 0.80

. Find required

n

. Step 1. Required quantiles:

z_{1 - α} = z_{0.95} = 1.645

; for power

0.80

z_{power} = z_{0.80} = 0.842

. Step 2. The sample-size formula (rearranging Equation 2 so power

= Φ (z_{power})

) is:

n = \frac{( z _{1 - α} + z _{power} ) ^{2} σ ^{2}}{( μ _{1} - μ _{0} ) ^{2}} .

Step 3. Plug in:

n = \frac{( 1.645 + 0.842 ) ^{2} \times 1 2 ^{2}}{5 ^{2}} = \frac{( 2.487 ) ^{2} \times 144}{25} = \frac{6.185 \times 144}{25} = \frac{890.6}{25} = 35.6.

Step 4. Round up.

Answer: $n = 36 patients$ to achieve 80% power.

Bonferroni vs. Benjamini–Hochberg

Given:

m = 5

genes tested for differential expression with p-values

0.001, 0.008, 0.02, 0.04, 0.20

. Desired error control

α = 0.05

. Step 1 — Bonferroni (FWER). Threshold each p against

α / m = 0.05/5 = 0.01

$0.001 < 0.01$ ✓ reject
$0.008 < 0.01$ ✓ reject
$0.02, 0.04, 0.20 >$ 0.01 ✗

Bonferroni rejects 2 hypotheses.

Step 2 — Benjamini–Hochberg (FDR). Sort p-values (already sorted), rank

i = 1 \dots 5

. Compare

p_{(i)}

\frac{i}{m} α

$i = 1 : 0.001 \leq \frac{1}{5} (0.05) = 0.010$ ✓
$i = 2 : 0.008 \leq \frac{2}{5} (0.05) = 0.020$ ✓
$i = 3 : 0.02 \leq \frac{3}{5} (0.05) = 0.030$ ✓
$i = 4 : 0.04 \leq \frac{4}{5} (0.05) = 0.040$ ✓
$i = 5 : 0.20 \leq \frac{5}{5} (0.05) = 0.050$ ✗

Step 3. BH rejects all up to the largest

i

satisfying the condition, which is

i = 4

Answer: Bonferroni rejects $2$ genes; BH rejects $4$ genes. BH is more powerful but controls only the expected proportion of false discoveries, not the chance of any.

Common Misconceptions

"A non-significant result means the null is true." No — failure to reject can simply mean low power. With

n

too small, even large true effects produce

p > 0.05

. Always report power or confidence intervals: a wide CI around zero signals an inconclusive (underpowered) study, not evidence of "no effect." " $α$ is the probability the null is true given a significant result." False.

α = P (reject ∣ H_{0})

, not

P (H_{0} ∣ reject)

. Confusing these is the prosecutor's fallacy. "Power is a single number for a test." Power is a function of the true parameter

θ

. A test has high power for large effects and low power for tiny ones. Always specify the effect size at which power is computed.

The Look-Elsewhere / Garden of Forking Paths

A researcher tests 20 candidate dietary supplements for an effect on weight loss, each at $α = 0.05$ , and finds that "green tea extract" is significant ( $p = 0.03$ ). They publish.

But under the global null (no supplement works), the probability of at least one false positive is $1 - 0.9 5^{20} = 0.64$ . Finding a significant result was the expected outcome of pure chance. Without correction, this single "discovery" is essentially noise — a textbook false positive. Reporting only the one winning test (and hiding the 19 failures) makes the result irreproducible, the central engine of the replication crisis.

Connections

Power and multiplicity sit at the heart of inferential statistics. The Neyman–Pearson framework that defines power extends Hypothesis Testing: Fundamentals by adding the alternative-hypothesis perspective. Power calculations rely on the Central Limit Theorem to justify normal approximations for test statistics, and on Effect Size to quantify the signal being detected. The noncentrality parameter links directly to the Noncentral Distributions (noncentral $t$ , $χ^{2}$ , $F$ ). Multiple-testing corrections connect to Confidence Intervals via simultaneous coverage (e.g., Bonferroni-adjusted intervals) and to Bayesian Inference, which sidesteps multiplicity by modeling all effects jointly with shrinkage priors. The FDR framework is closely tied to Empirical Bayes Methods.

Real-World Applications

Clinical trials (FDA). Phase III trials must be powered (typically 80–90%) to detect a clinically meaningful effect; regulators require pre-registered power calculations and multiplicity adjustment for multiple endpoints (e.g., Hochberg or gatekeeping procedures).
Genome-Wide Association Studies (GWAS). Testing millions of SNPs uses a genome-wide significance threshold of $p < 5 \times 1 0^{- 8}$ (Bonferroni-style for ~10⁶ independent loci).
Particle physics (CERN). The $5 σ$ standard for the Higgs boson discovery ( $p \approx 3 \times 1 0^{- 7}$ ) accounts for the "look-elsewhere effect" — multiplicity across the mass spectrum.
A/B testing (tech). Companies running thousands of simultaneous experiments apply FDR control (Benjamini–Hochberg) to avoid shipping spurious "improvements."
Functional MRI. Tens of thousands of brain voxels are tested; uncorrected analysis famously "detected" activity in a dead salmon — driving adoption of cluster-based FWER and FDR methods.
Drug screening (pharma). High-throughput screens of thousands of compounds use FDR to prioritize hits while controlling wasted follow-up.
Quality control / agriculture. ANOVA post-hoc comparisons (Tukey's HSD, Scheffé) control FWER across all pairwise treatment contrasts.

Historical Development

1900 — Karl Pearson introduces the $χ^{2}$ goodness-of-fit test, an early formal significance procedure.
1925 — R.A. Fisher's Statistical Methods for Research Workers popularizes the p-value and the $α = 0.05$ convention, but treats significance as evidence, not decision.
1928–1933 — Neyman & Pearson formalize Type I/II errors, power, and the most-powerful-test lemma, founding the accept/reject framework.
1935–1936 — Carlo Emilio Bonferroni publishes the inequalities (Boole's union bound applied to simultaneous events) underlying the eponymous correction.
1953 — John Tukey develops the Honestly Significant Difference test for multiple pairwise comparisons in ANOVA.
1979 — Sture Holm introduces the step-down Holm–Bonferroni method, uniformly more powerful than plain Bonferroni while still controlling FWER.
1988 — Yosef Hochberg publishes a step-up FWER procedure exploiting Simes' inequality.
1995 — Yoav Benjamini & Yosef Hochberg introduce the False Discovery Rate, revolutionizing high-dimensional testing in genomics and beyond.
2001 — Benjamini & Yekutieli extend FDR control to arbitrary dependence; Storey introduces the q-value and positive-FDR, refining the framework for the genomics era.

Subtleties & Limits

Dependence structure. Most clean results (independent FWER, basic BH) assume independence or specific positive-dependence (PRDS) conditions. Real data — correlated genes, spatially smooth neuroimaging voxels — violate these. The Benjamini–Yekutieli correction guarantees FDR control under arbitrary dependence but at a cost of a factor

\sum_{i = 1}^{m} 1/ i \approx ln m

, sharply reducing power. Finding tight, dependence-aware procedures remains active research. Choosing the right error rate. FWER vs. FDR is not a settled question — it depends on the cost of a false discovery relative to a missed one. Hybrid notions (per-family error rate, k-FWER, FDP confidence bounds) proliferate without consensus. Post-selection inference. When the same data is used to select hypotheses and then test them (data dredging, adaptive analysis), classical guarantees collapse. Selective inference and conditional inference (Lee, Taylor, et al.) are an emerging but technically demanding remedy. Optimal power under multiplicity. There is no fully general theory of "most powerful" multiple-testing procedure; optimality depends on the unknown configuration of true effects, leaving practical procedure choice partly heuristic.