Hypothesis Testing Framework

A pharmaceutical company has a new drug. Does it actually lower blood pressure more than a placebo? Or could the observed improvement be explained by chance? This is the central question of hypothesis testing. It is one of the two great pillars of statistical inference, alongside estimation. Confidence Intervals: Concept and Construction asks "what is the value of this parameter." Hypothesis testing instead asks a sharper, binary question. Is this specific claim about the parameter consistent with the data? Or should we reject it? The framework matters because science, medicine, engineering, and law all run on decisions made under uncertainty. Hypothesis testing is the formal machine that turns "the data look suggestive" into a decision procedure. That procedure has quantified error rates. It forces the analyst to commit, in advance, to what would count as evidence against a claim. It also requires stating honestly how often such a procedure would be wrong, even when used correctly. This discipline — deciding the rule before seeing the outcome — is what separates statistical inference from post-hoc storytelling. The big picture is a courtroom analogy formalized with probability. There is a default position, the null hypothesis. It is presumed true until the data provide sufficiently strong evidence against it. This is exactly as a defendant is presumed innocent. There is a competing claim, the alternative hypothesis, which the data must convince us to prefer. A test statistic condenses the sample into one number. A rejection region specifies which values of that number count as "sufficiently strong evidence." The two ways the procedure can err are quantified and traded off against each other. These are convicting the innocent (Type I error) and acquitting the guilty (Type II error). This article builds the full architecture. It covers the formal definitions of hypotheses, test statistics, and rejection regions. It explains the precise meanings of significance level and power. It presents the fundamental theorem connecting rejection regions to error probabilities. It also includes two complete worked examples with numerically distinct, asymmetric inputs. It closes by pointing toward the machinery that makes this framework operational in practice. That machinery includes The Neyman-Pearson Lemma, p-values, and the standard named tests.

Key takeaways

A hypothesis test decides between a null hypothesis $H_0$ and an alternative $H_1$ using sample data and a fixed decision rule.
A test statistic $T$ is a function of the data; a rejection region $C$ is the set of $T$ -values that lead to rejecting $H_0$ .
Type I error (rejecting a true $H_0$ ) has probability $\alpha$ , the significance level; Type II error (failing to reject a false $H_0$ ) has probability $\beta$ .
Power is $1-\beta$ , the probability of correctly rejecting a false $H_0$ ; it depends on the true parameter value, sample size, and $\alpha$ .
Tests are built by fixing $\alpha$ first, then choosing the rejection region to maximize power — this ordering is not optional.
Every conclusion is conditional on the model assumptions (distribution, independence, sample size) used to derive the sampling distribution of $T$ .
Key idea: a hypothesis test is a pre-committed decision rule whose two error rates are explicitly quantified and traded off, never a post-hoc judgment about "significance."

Where this lives today

Hypothesis testing is the backbone of experimental science: clinical trials use it to decide whether a treatment effect is real, with regulatory agencies mandating pre-registered $\alpha$ levels (commonly 0.05 or 0.025 one-sided) precisely to prevent the data-dependent rejection-region abuse illustrated in the counterexample above. In physics, discovery claims (such as the Higgs boson) are framed as hypothesis tests against a "no new physics" null, but with an extremely stringent significance threshold (the "5-sigma" convention) reflecting the cost of false discovery in a field with many simultaneous searches. In engineering and manufacturing, statistical process control charts are built from repeated hypothesis tests that flag when a process mean or variance has shifted out of specification, directly extending the mechanics of Example 1. In machine learning and A/B testing, the same architecture (null: no difference between algorithm variants; test statistic: a standardized difference in conversion rates; rejection region: extreme values) governs decisions about which website design or model to deploy, though at internet scale the framework's dependence on pre-registered rejection regions is often violated by "peeking" at results early, an error structurally identical to the counterexample. In cryptography and statistical hypothesis-testing-based intrusion detection, the same null/alternative structure with controlled false-alarm ( $\alpha$ ) and missed-detection ( $\beta$ ) rates underlies anomaly detection systems. The theoretical core developed here — size, power, and the tradeoff between them — sets up directly for The Neyman-Pearson Lemma (optimal tests), the theory of p-values and significance (a continuous-valued summary of the evidence against $H_0$ ), and the standard named procedures gathered under Standard Tests: t, Chi-Squared, and F.

Ask the tutor: How does the choice between a one-sided and two-sided alternative change the critical value and power of a test for the same data?

X_1,\dots,X_n

Intuition

Think of the test statistic as a dial with a needle, and the rejection region as a red zone marked on the dial before any data are collected. Once the data arrive, the needle moves to a single position; if it lands in the red zone, we reject $H_0$ . The two hypotheses are not symmetric: $H_0$ is the default, and the burden of proof is on the data to push the needle into the red zone. This asymmetry is deliberate — it mirrors the presumption of innocence, and it means the test is designed to control the more serious error (wrongly abandoning the default) at a fixed, small rate $\alpha$ , even before considering how well it detects real effects. Power is the answer to "if $H_1$ is actually true, and specifically true at some value $\theta$ , how likely is the needle to land in the red zone?" This depends on how far $\theta$ is from $\Theta_0$ : values of $\theta$ deep inside $H_1$ 's territory are easy to detect (high power), while values just barely inside $H_1$ are nearly indistinguishable from $H_0$ (power near $\alpha$ ). Enlarging the rejection region increases power but also increases $\alpha$ — the two error types are locked in a seesaw, and sample size is the lever that lets you push both down simultaneously.

Figure

Rejection region for a one-sided z-test at level α = 0.05, testing H₀: μ = 0 vs H₁: μ > 0

T

P_{\theta_0}(T \ge c_\alpha) = \alpha

Worked examples

Example 1: One-sample z-test for a mean, one-sided alternative, asymmetric numbers. A quality-control engineer believes a new machine setting increases the mean fill weight of cereal boxes above the nominal target. Historical process variability is well characterized: fill weights are normally distributed with known standard deviation $\sigma = 4.2$ grams (this is realistic in mature manufacturing lines with long historical records). The target mean is $\mu_0 = 500$ grams. She wants to test

H_0: \mu = 500 \quad \text{vs} \quad H_1: \mu > 500,

at significance level $\alpha = 0.02$ . She draws $n = 36$ boxes and observes a sample mean $\bar x = 501.6$ grams. Step 1 — test statistic. Under $H_0$ , by the sampling distribution of the mean, $\bar X \sim N(\mu_0, \sigma^2/n)$ exactly (normal population, known $\sigma$ ). Standardize:

Z = \frac{\bar X - \mu_0}{\sigma/\sqrt n} \sim N(0,1) \text{ under } H_0.

Step 2 — rejection region. This is a one-sided upper test; reject for large $Z$ . We need $c$ with $P(Z \ge c) = 0.02$ . From the standard normal table, $z_{0.02} = 2.054$ . So $C = \{Z \ge 2.054\}$ . Step 3 — compute the statistic.

Z = \frac{501.6 - 500}{4.2/\sqrt{36}} = \frac{1.6}{4.2/6} = \frac{1.6}{0.7} = 2.2857\ldots

Step 4 — decision. Since $2.2857 > 2.054$ , $Z$ falls in the rejection region: reject $H_0$ at level $\alpha = 0.02$ . The data provide statistically significant evidence, at this level, that the true mean fill weight exceeds 500 grams. Step 5 — power at a specific alternative. Suppose the engineer wants to know the power of this test if the machine setting truly shifted the mean to $\mu_1 = 502$ grams. Under $\mu_1$ ,

\bar X \sim N(502, \, 4.2^2/36) = N(502, 0.49),

so $\sigma_{\bar X} = 0.7$ . The rejection region in terms of $\bar X$ is $\bar X \ge \mu_0 + 2.054(0.7) = 500 + 1.4378 = 501.4378$ . Power is

\pi(502) = P_{\mu_1=502}(\bar X \ge 501.4378) = P\!\left(Z \ge \frac{501.4378 - 502}{0.7}\right) = P(Z \ge -0.8031).

From the normal table, $\Phi(0.8031) \approx 0.7890$ , so $\pi(502) = P(Z \ge -0.8031) = \Phi(0.8031) \approx 0.789$ . Thus if the true mean is 502 grams, this test correctly detects the shift about 78.9% of the time; the Type II error rate there is $\beta \approx 0.211$ .

Example 2: Two-sided test with a small sample and unknown variance (previewing the t-test), composite null. An agronomist tests whether a new fertilizer changes (in either direction) the mean yield per plot, historically $\mu_0 = 48.0$ bushels/acre. Population variance is unknown. She samples $n = 9$ plots and observes sample mean $\bar x = 50.4$ and sample standard deviation $s = 3.6$ (using the unbiased estimator with $n-1$ in the denominator). She sets $\alpha = 0.05$ and tests

H_0: \mu = 48.0 \quad \text{vs} \quad H_1: \mu \ne 48.0.

Step 1 — test statistic. Because $\sigma$ is unknown and $n$ is small, the exact normal-based $Z$ statistic is unavailable; the standardized statistic using $s$ in place of $\sigma$ follows a Student $t$ distribution with $n-1 = 8$ degrees of freedom under $H_0$ (this fact — the exact distribution of $(\bar X-\mu_0)/(S/\sqrt n)$ under normality of the population — is the content of the t-test, used here only as the known sampling distribution required by the general framework):

T = \frac{\bar X - \mu_0}{S/\sqrt n} \sim t_{8} \text{ under } H_0.

Step 2 — rejection region. Two-sided test at $\alpha = 0.05$ splits the error equally into both tails: reject if $|T| \ge t_{8,\,0.025}$ . From the $t$ -table, $t_{8,0.025} = 2.306$ . So $C = \{|T| \ge 2.306\}$ . Step 3 — compute the statistic.

T = \frac{50.4 - 48.0}{3.6/\sqrt 9} = \frac{2.4}{3.6/3} = \frac{2.4}{1.2} = 2.0.

Step 4 — decision. Since $|T| = 2.0 < 2.306$ , $T$ does not fall in the rejection region: we do not reject $H_0$ at level $\alpha = 0.05$ . Despite a seemingly large observed difference (2.4 bushels/acre), the small sample size and moderate variability leave insufficient evidence to conclude the fertilizer changes mean yield. Step 5 — a lesson on error types. This is exactly the situation in which a Type II error is a live risk: if the fertilizer truly raises mean yield by, say, 3 bushels/acre, a sample of only $n=9$ plots may lack the power to detect it reliably. This is precisely why, in practice, one performs power analysis before collecting data — to choose $n$ large enough that a scientifically important effect size is detected with high probability, rather than discovering after the fact that the test was underpowered.

\alpha

Scope, edge cases & common traps

Situation	What changes	Common pitfall
Simple $H_0$ , simple $H_1$	Power function has one value under $H_1$ ; size and power fully determine the test	Confusing "not rejecting $H_0$ " with "proving $H_0$ true"
Composite $H_0$ (e.g. $\mu \le \mu_0$ )	Size is a supremum over $\Theta_0$ , not a single probability	Reporting only $\pi(\theta_0)$ as "the" Type I error rate
One-sided vs two-sided alternative	Rejection region is one tail vs split into two tails	Choosing the side of a one-sided test after seeing which direction the data lean
Unknown variance, small $n$	Test statistic follows $t_{n-1}$ , not exactly $N(0,1)$	Using the normal table's critical values with small $n$ and unknown $\sigma$
Large $n$	By Theorem Part 3, power $\to 1$ for any fixed true $\theta \ne \theta_0$	Treating statistical significance at huge $n$ as automatically practically important
Multiple tests on the same data	Overall Type I error rate compounds across tests	Running many tests and reporting only the "significant" ones (see the counterexample)

The mathematician behind it

Origins · 1 minJerzy Neyman and how this was discovered

Jerzy Neyman

1894–1981 · Statistics, Poland/UK/USA

Neyman, working with Egon Pearson at University College London in the late 1920s and 1930s, transformed hypothesis testing from an intuitive practice of "significance" (as used by Ronald Fisher) into a rigorous decision-theoretic framework with two hypotheses, two types of error, and an optimality theory for choosing rejection regions. Their joint 1933 paper introduced the size–power tradeoff and the lemma that bears their names, which identifies the most powerful test for a fixed size in the simple-versus-simple case. Neyman later emigrated to Berkeley, where he built one of the world's leading statistics departments and contributed foundational work on confidence intervals, survey sampling, and the philosophy of frequentist inference.

History

- 1900: Karl Pearson introduces the chi-squared goodness-of-fit test, an early ancestor of formal hypothesis testing.

- 1925: Ronald Fisher's Statistical Methods for Research Workers popularizes significance testing and the 5% convention.

- 1928: Neyman and Egon Pearson begin their collaboration on the theory of hypothesis testing.

- 1933: Neyman and Pearson publish "On the Problem of the Most Efficient Tests of Statistical Hypotheses," establishing Type I/II errors, power, and the fundamental lemma.

- 1938 onward: Neyman develops the theory further at Berkeley, cementing the null/alternative/rejection-region framework as the standard structure of statistical testing.

What exactly is the difference between the null and alternative hypotheses?

The null hypothesis $H_0$ is the default or status-quo claim, typically a statement of "no effect" or "no difference," and it is what the test is designed to potentially reject. The alternative $H_1$ is the claim the researcher often suspects or hopes to support, and it is only adopted if the data provide sufficiently strong evidence against $H_0$ . The asymmetry is structural, not cosmetic: the test's error-rate guarantee ( $\alpha$ ) is computed under $H_0$ , while power is computed under $H_1$ ; swapping which hypothesis plays which role changes the entire error-control interpretation of the procedure, even if the mathematical sets $\Theta_0$ and $\Theta_1$ stay the same.

Why must the significance level α be chosen before looking at the data?

Choosing $\alpha$ (and hence the rejection region) in advance is what makes the probability statement $P_{\theta_0}(T \in C) = \alpha$ a valid, pre-experimental guarantee about the long-run behavior of the procedure. If the region or the level is adjusted after seeing the test statistic's value, the actual probability of a false rejection under repeated use of that adjusted procedure is no longer $\alpha$ , typically larger, as shown explicitly in the counterexample where a data-dependent choice doubled the true Type I error rate. This principle is why pre-registration of statistical analysis plans is required in clinical trials and increasingly encouraged across experimental science.

Is a small p-value or a rejected null proof that the alternative hypothesis is true?

No. A hypothesis test controls the probability of specific decision errors under repeated sampling; it does not compute the probability that any particular hypothesis is true, which would require a prior distribution over hypotheses (a Bayesian framework, not this one). Rejecting $H_0$ at level $\alpha$ means: under a decision rule that would falsely reject a true $H_0$ at most $100\alpha\%$ of the time in repeated use, this particular dataset triggered rejection. It is evidence against $H_0$ calibrated by a known false-alarm rate, not a probabilistic certificate on $H_1$ .

How is power related to sample size, and why does this matter for study design?

Under the conditions of the theorem's Part 3, for any fixed true alternative $\theta_1 \ne \theta_0$ , the power $\pi(\theta_1)$ increases toward 1 as the sample size $n \to \infty$ , because the sampling distribution of the test statistic under $\theta_1$ separates further from its distribution under $\theta_0$ . This is why researchers perform a power analysis before data collection: given a minimally important effect size, a desired power (commonly 0.80 or 0.90), and a chosen $\alpha$ , one can solve for the sample size $n$ needed so the study is not doomed to a Type II error from being too small, as nearly happened in Example 2.

What is the relationship between hypothesis testing and confidence intervals?

A two-sided level- $\alpha$ test of $H_0: \theta = \theta_0$ rejects if and only if $\theta_0$ falls outside the corresponding $(1-\alpha)$ -confidence interval for $\theta$ , built from the same test statistic and its sampling distribution as covered in Confidence Intervals: Concept and Construction. This duality means every confidence interval implicitly defines a whole family of hypothesis tests (one for each candidate $\theta_0$ ), and every family of size- $\alpha$ tests, indexed by $\theta_0$ , can be inverted to produce a $(1-\alpha)$ -confidence set. The two frameworks answer the same underlying question — how compatible is a parameter value with the data — from complementary angles: estimation (a set of plausible values) versus decision (accept or reject one specific value).

Why can't a rejection region be based on a statistic whose null distribution is unknown?

The entire quantitative content of a hypothesis test — the value of $\alpha$ , the location of the critical value, the computation of power — rests on knowing (exactly or via a reliable approximation, such as a large-sample normal or $t$ approximation) the distribution of $T$ when $H_0$ is true. Without this, the claim " $P_{\theta_0}(T \in C) = \alpha$ " cannot be verified or even stated, and the procedure reduces to an arbitrary rule with unknown error properties. This is exactly why the derivation of sampling distributions, covered under Sampling Distributions, is a prerequisite: it supplies the null distributions (normal, $t$ , chi-squared, $F$ ) on which every classical test in Standard Tests: t, Chi-Squared, and F is built.

What is the difference between Type I error, Type II error, and power in plain terms?

Type I error is rejecting a true $H_0$ — a false alarm — and its probability is bounded by $\alpha$ , fixed by the analyst before data collection. Type II error is failing to reject a false $H_0$ — a missed detection — and its probability $\beta$ depends on which specific alternative $\theta_1 \in \Theta_1$ is true, on $n$ , and on $\alpha$ . Power is $1-\beta$ , the probability of correctly detecting that alternative; because $\alpha$ and $\beta$ trade off against each other for fixed $n$ , and both can be driven down simultaneously only by increasing $n$ , no single number can describe a test's overall error behavior without specifying which alternative and which sample size are in view.