Hypothesis Testing and Significance

Every claim in experimental physics eventually reduces to one uncomfortable question: could the data have looked like this by chance alone? Hypothesis testing turns that question into a number. You write down a null hypothesis $H_0$ — the boring default, "nothing new here, only known background" — and an alternative $H_1$ , "there is a signal". You choose a test statistic $T(\mathbf{X})$ , a single number squeezed from the data whose behaviour under $H_0$ you can compute exactly. If the observed $T$ falls deep in the tail of what $H_0$ predicts, the data are surprising and you reject $H_0$ . The whole edifice controls one thing you never see directly: the rate at which you fool yourself. The significance level $\alpha$ is the probability, fixed in advance, of rejecting a true $H_0$ — a Type I error, a false alarm. Its partner $\beta$ is the probability of missing a real effect (Type II), and $1-\beta$ , the power, is the number an experimenter loses sleep over. There is no free lunch: tighten the cut to suppress false alarms and you miss more real signals. The art — the theorem at the centre of this article — is to squeeze maximum power out of a fixed false-alarm budget. That theorem is the Neyman–Pearson lemma, and its answer is startlingly clean: rank every outcome by its likelihood ratio $\Lambda=\mathcal{L}(\mathbf{x}\mid H_0)/\mathcal{L}(\mathbf{x}\mid H_1)$ and reject those $H_1$ favours most. From that one idea flow the $z$ -test, the $t$ -test, Pearson's $\chi^2$ , Wilks' theorem, and the $5\sigma$ convention for when a particle counts as discovered. Threaded through all of it is the misconception that has done more damage than any other in quantitative science: the $p$ -value is not the probability that $H_0$ is true.

Key takeaways

Hypothesis testing is a decision rule: posit a null $H_0$ and alternative $H_1$ , compute a test statistic, and reject $H_0$ when it lands in a pre-chosen critical region.
The significance level $\alpha$ is the fixed probability of a Type I error (rejecting a true $H_0$ ); the power $1-\beta$ correctly rejects a false $H_0$ . Shrinking one tail trades against the other.
The $p$ -value is the tail probability under $H_0$ of a statistic at least as extreme as observed; reject when $p<\alpha$ . It is uniform on the unit interval under $H_0$ — and emphatically not $P(H_0\mid\text{data})$ .
The Neyman–Pearson lemma proves the likelihood ratio gives the most powerful test at fixed $\alpha$ for a simple null against a simple alternative.
For Gaussian data this collapses to the $z$ -, $t$ - and $\chi^2$ -tests; Wilks' theorem makes $-2\ln\Lambda$ asymptotically $\chi^2$ , the workhorse of profile-likelihood testing.
Key idea: fix the false-positive rate α in advance, then among all tests with that rate choose the one with greatest power; the Neyman–Pearson lemma shows the optimal decision boundary is a contour of constant likelihood ratio Λ(x) = ℒ(x|H₀)/ℒ(x|H₁).

The core insight

Fix the false-positive rate $\alpha$ in advance, then among all tests with that rate choose the one with greatest power; the Neyman–Pearson lemma shows the optimal decision boundary is a contour of constant likelihood ratio $\Lambda(\mathbf{x})=\mathcal{L}(\mathbf{x}\mid H_0)/\mathcal{L}(\mathbf{x}\mid H_1)$ .

Where this lives today

The $5\sigma$ convention is where this machinery becomes headline physics. When ATLAS and CMS announced the Higgs boson in July 2012, the number that mattered was a local $p$ -value at $5.0\sigma$ ( $p\approx2.87\times10^{-7}$ ): each tested $H_0$ = "background only" against $H_1$ = "background plus a resonance" using $-2\ln\Lambda$ whose distribution Wilks' theorem supplies. The stringency defends against the look-elsewhere effect — scanning many mass bins inflates the chance that some bin fluctuates high, so the local $p$ must be corrected to a global one first. The same framework governs the LIGO–Virgo gravitational-wave detections, neutrino oscillations, and the muon $g-2$ anomaly at Fermilab, whose $\sim4$ – $5\sigma$ tension with theory is exactly a $z$ -test of a measured against a predicted number. Meanwhile the replication crisis in biomedical and social science — driven by $p$ -hacking, underpowered studies, and precisely the $p$ -value misreading this article warns against — has pushed proposals to lower defaults to $p<0.005$ . The century-old Fisher–Neyman argument is, remarkably, still live.

The physical picture

Picture a coin you suspect is loaded. Under $H_0$ it is fair, $p=\tfrac12$ ; flip it $n$ times and the head-count clusters near $n/2$ with a known binomial spread. A count far out in the tail would be a freak event for a fair coin, so it is evidence against fairness. The significance level is the line you draw before flipping: "declare the coin loaded only if the result is so extreme it would happen by chance less than $\alpha$ of the time." Every physics test is this coin dressed up: a known null distribution, a cut in its tail, a decision. The sharper picture is two overlapping bell curves — the statistic under $H_0$ and under $H_1$ — with the critical region a single cut on the axis. The $H_0$ area past the cut is $\alpha$ ; the $H_1$ area before the cut is $\beta$ , the signal you miss. Slide the cut right and $\alpha$ shrinks while $\beta$ grows — both are small only when the curves are far apart, which is what more data buys.

Figure

Two sampling distributions of the test statistic — H₀ (blue) and a shifted H₁ (green). The vertical cut at the critical value z = 1.645 puts α = 0.05 of the H₀ area in the red rejection tail, while the green area left of the cut is β, the missed-signal probability. Sliding the cut trades α against β.

A $p$ -value is a ruler for surprise, uniform on the unit interval under $H_0$ : $p=0.01$ means a fair world would produce data this extreme only one time in a hundred. Small $p$ says the data sit in the tail of the null's prediction — but it speaks only about compatibility with $H_0$ , never its truth, which needs a prior. Particle physics pushes the ruler to an extreme: a discovery needs the statistic five standard deviations out, a one-sided tail of $2.87\times10^{-7}$ , about one in 3.5 million.

Figure

The significance ladder: the standard normal N(0,1) under H₀, with the one-sided rejection tail beyond z = 1.645 (α = 0.05) shaded red. The p-value is the tail area past the observed z. Ladder rungs: 1σ → p = 0.159, 2σ → 0.0228, 3σ → 1.35×10⁻³ (evidence), 5σ → 2.87×10⁻⁷ (discovery).

Power is what experimenters actually care about: if the effect is real, how likely am I to catch it? A test with airtight $\alpha$ but tiny power wastes data — a null result then tells you almost nothing, because you never had the sensitivity to see the signal. Designing an experiment is choosing the sample size and statistic so the power at the smallest interesting signal is respectable, conventionally $1-\beta\geq0.8$ .

The core equations

The starting point is the pair of error rates. A test partitions the sample space into acceptance and rejection (critical) regions; the two ways to be wrong live on opposite hypotheses and are computed under different distributions — this asymmetry is the whole game.

\alpha = P(T\in W \mid H_0),\qquad \beta = P(T\notin W \mid H_1),\qquad \text{power}=1-\beta \qquad (1)

where: $T$ — test statistic, a function of the data $\mathbf{X}$ ; $W$ — critical (rejection) region in the sample space of $T$ ; $\alpha$ — significance level, the Type I (false-positive) error probability; $\beta$ — Type II (false-negative) error probability; $1-\beta$ — power. Reading: $\alpha$ is chosen by fiat before data; $\beta$ then follows from the alternative and the sample size. The $p$ -value repackages the observed statistic as a probability, comparable across tests: it is the smallest $\alpha$ at which the data would already be rejected.

p = P\big(T \ge t_{\text{obs}} \,\big|\, H_0\big)\quad(\text{one-sided}),\qquad \text{reject } H_0 \text{ if } p<\alpha \qquad (2)

where: $t_{\text{obs}}$ — observed value of the statistic; $p$ — tail probability under $H_0$ of statistics at least as extreme as observed. Reading: a symmetric two-sided test doubles the tail, $p=2\,P(T\ge|t_{\text{obs}}|\mid H_0)$ ; crucially $p$ is conditioned on $H_0$ , not a probability of $H_0$ . The likelihood ratio is the quantity the Neyman–Pearson lemma singles out as optimal — it compares how well the two hypotheses explain the same data, the decision boundary being a level set of the ratio.

\Lambda(\mathbf{x}) = \frac{\mathcal{L}(\mathbf{x}\mid H_0)}{\mathcal{L}(\mathbf{x}\mid H_1)},\qquad \text{reject } H_0 \text{ if } \Lambda(\mathbf{x}) \le k_\alpha \qquad (3)

where: $\mathcal{L}(\mathbf{x}\mid H)$ — likelihood of the data under $H$ ; $\Lambda$ — likelihood-ratio statistic; $k_\alpha$ — threshold fixed so $P(\Lambda\le k_\alpha\mid H_0)=\alpha$ . Reading: small $\Lambda$ means $H_1$ explains the data far better, so small $\Lambda$ is the natural rejection region. For Gaussian data with known variance, the likelihood ratio for a mean collapses to a linear cut on the sample mean — the $z$ -statistic, whose null distribution is standard normal.

Z = \frac{\bar{X}-\mu_0}{\sigma/\sqrt{n}},\qquad \bar{X}=\frac1n\sum_{i=1}^n X_i \qquad (4)

where: $\bar{X}$ — sample mean of $n$ observations; $\mu_0$ — mean hypothesised by $H_0$ ; $\sigma$ — known population standard deviation; $n$ — sample size. Reading: the denominator $\sigma/\sqrt{n}$ is the standard error, so $Z$ measures the discrepancy in units of how well you know the mean. When $\sigma$ is unknown and estimated from the same small sample, the extra uncertainty fattens the tails: the statistic follows Student's $t$ , not the normal.

T = \frac{\bar{X}-\mu_0}{s/\sqrt{n}},\qquad s^2=\frac{1}{n-1}\sum_{i=1}^n (X_i-\bar{X})^2 \qquad (5)

where: $s$ — sample standard deviation with the unbiased $n-1$ (Bessel) denominator; $T$ — follows $t_{n-1}$ under $H_0$ . Reading: for $n\gtrsim30$ the $t$ is nearly normal; for $n=5$ its critical value swells noticeably above $z$ . For binned or categorical data, Pearson's statistic sums squared standardised residuals over bins and is asymptotically $\chi^2$ -distributed.

\chi^2 = \sum_{i=1}^{k} \frac{(O_i-E_i)^2}{E_i} \qquad (6)

where: $O_i$ — observed count in bin $i$ ; $E_i$ — count expected under $H_0$ ; $k$ — number of bins. Reading: each term is $(\text{residual})^2/(\text{Poisson variance})$ ; degrees of freedom $\nu=k-1-m$ after the normalisation constraint and $m$ fitted parameters. The likelihood ratio generalises to composite hypotheses through the profile likelihood ratio, whose asymptotic distribution Wilks' theorem supplies — letting physicists read a significance straight off a fit.

-2\ln\Lambda \;=\; -2\ln\frac{\sup_{\theta\in\Theta_0}\mathcal{L}(\theta)}{\sup_{\theta\in\Theta}\mathcal{L}(\theta)} \;\xrightarrow{\;n\to\infty\;}\; \chi^2_r \qquad (7)

where: $\Theta_0$ — parameter subspace fixed by $H_0$ ; $\Theta$ — the full parameter space; $r$ — number of constraints (codimension of $\Theta_0$ ). Reading: the suprema profile out nuisance parameters, and the ratio never exceeds 1, so $-2\ln\Lambda\ge0$ . For $r=1$ the significance in $\sigma$ is $\sqrt{-2\ln\Lambda}$ , and the discovery conventions read straight off the normal tail: $3\sigma\Rightarrow p=1.35\times10^{-3}$ ("evidence"), $5\sigma\Rightarrow p=1-\Phi(5)=2.87\times10^{-7}$ (discovery, one-sided).

Derivation AThe Neyman-Pearson lemma

The goal is to prove that, for a simple null $H_0:\theta=\theta_0$ against a simple alternative $H_1:\theta=\theta_1$ , no test of size $\alpha$ is more powerful than the one whose rejection region is a level set of the likelihood ratio. We propose the likelihood-ratio region $W^*$ , take any competitor $W$ of size at most $\alpha$ , and show the difference in their powers can never be negative. Write $f_0=\mathcal{L}(\cdot\mid\theta_0)$ and $f_1=\mathcal{L}(\cdot\mid\theta_1)$ .

Step 1. Define the candidate region as everything the alternative favours by at least a factor $c$ : $W^*=\{\mathbf{x}:f_1(\mathbf{x})\ge c\,f_0(\mathbf{x})\}$ , with $c$ tuned so $\int_{W^*}f_0=\alpha$ . Its power is the alternative mass captured, $\text{Power}(W^*)=\int_{W^*}f_1$ . Why: this is exactly the "reject where $\Lambda=f_0/f_1\le 1/c$ " rule of equation (3), with $c$ carrying the size constraint.

Step 2. Let $W$ be any region with $\int_W f_0\le\alpha$ . The power difference $\text{Power}(W^*)-\text{Power}(W)=\int_{W^*}f_1-\int_{W}f_1$ ; the shared part $W^*\cap W$ cancels, leaving the symmetric difference $\int_{W^*\setminus W}f_1-\int_{W\setminus W^*}f_1$ . Why: only the points where the two tests disagree can change the power.

Step 3. On $W^*\setminus W$ every point has $f_1\ge c f_0$ by construction; on $W\setminus W^*$ every point lies outside $W^*$ , so $f_1< c f_0$ . Substituting, $\int_{W^*\setminus W}f_1-\int_{W\setminus W^*}f_1 \ge c\big(\int_{W^*\setminus W}f_0-\int_{W\setminus W^*}f_0\big)$ . Why: lower-bounding $f_1$ on the region we add and upper-bounding it on the region we remove can only decrease the expression, so the inequality points the right way.

Step 4. Re-adding the cancelled overlap turns the bracket back into full sizes: $c\big(P(W^*\mid\theta_0)-P(W\mid\theta_0)\big)=c(\alpha-P(W\mid\theta_0))$ . Since $c>0$ and $P(W\mid\theta_0)\le\alpha$ , this is $\ge0$ , so $\text{Power}(W^*)\ge\text{Power}(W)$ . Why: the competitor's size constraint is exactly what makes the bracket non-negative — spending the full $\alpha$ budget optimally beats every alternative allocation. *Why this matters concretely: for two Gaussians with equal variance and means $\mu_0<\mu_1$ , the criterion $f_1\ge cf_0$ reduces algebraically to $\bar{X}\ge c'$ , recovering the one-sided z-test of the next derivation.*

Go deeper · Derivation B · 1 minThe z-test as the most powerful test of a Gaussian mean

Derivation BThe z-test as the most powerful test of a Gaussian mean

The goal is to run the Neyman–Pearson recipe on $n$ independent normal observations with known variance $\sigma^2$ , testing $H_0:\mu=\mu_0$ against $H_1:\mu=\mu_1>\mu_0$ , and show the optimal region is a simple cut on the sample mean whose calibration gives the familiar $z_{1-\alpha}$ critical values. Because that region does not depend on the specific $\mu_1$ , the same test is optimal against every $\mu_1>\mu_0$ at once — uniformly most powerful. Start from $\mathcal{L}(\mu)=(2\pi\sigma^2)^{-n/2}\exp[-\tfrac{1}{2\sigma^2}\sum_i(x_i-\mu)^2]$ .

Step 1. Form the ratio $\mathcal{L}(\mu_1)/\mathcal{L}(\mu_0)=\exp[-\tfrac{1}{2\sigma^2}\sum_i((x_i-\mu_1)^2-(x_i-\mu_0)^2)]$ ; the normalisation prefactors cancel. Why: only the exponent carries the data, so the whole test lives in one quadratic-difference term.

Step 2. Expand: $(x_i-\mu_1)^2-(x_i-\mu_0)^2=-2x_i(\mu_1-\mu_0)+(\mu_1^2-\mu_0^2)$ , in which the $x_i^2$ pieces cancel and only a term linear in $x_i$ survives. Why: linearity in the data is what makes the boundary a straight cut rather than a curved region.

Step 3. Collect the sum: $\mathcal{L}(\mu_1)/\mathcal{L}(\mu_0)=\exp[\tfrac{\mu_1-\mu_0}{\sigma^2}\sum_i x_i-\tfrac{n(\mu_1^2-\mu_0^2)}{2\sigma^2}]$ , which since $\mu_1>\mu_0$ is strictly increasing in $\sum_i x_i$ , hence in $\bar{X}$ . So " $\Lambda$ small" is identical to " $\bar{X}$ large", and the rejection region is $\{\bar{X}\ge c'\}$ . Why: this monotone-likelihood-ratio property makes a single cut optimal, and since $\mu_1$ dropped out of the inequality's direction the same cut is optimal for all $\mu_1>\mu_0$ — a uniformly most powerful test.

Step 4. Calibrate to size $\alpha$ . Under $H_0$ , $\bar{X}\sim\mathcal{N}(\mu_0,\sigma^2/n)$ , so $P(\bar{X}\ge c'\mid\mu_0)=\alpha$ standardises to $(c'-\mu_0)/(\sigma/\sqrt{n})=z_{1-\alpha}$ , i.e. reject when $Z=(\bar{X}-\mu_0)/(\sigma/\sqrt{n})>z_{1-\alpha}$ . For $\alpha=0.05$ one-sided, $z_{0.95}=1.645$ ; two-sided uses $z_{0.975}=1.960$ . Why: this is where the abstract threshold $c$ becomes a tabulated number — the standard-normal quantile every $z$ -test quotes. *As a consistency check: the critical values $z_{0.95}=1.645$ and $z_{0.975}=1.960$ correspond exactly to $\Phi(1.645)=0.950$ and $\Phi(1.960)=0.975$ from the standard normal CDF.*

Go deeper · Derivation C · 1 minWilks' theorem and the minus-two-log-lambda statistic

Derivation CWilks' theorem and the minus-two-log-lambda statistic

The goal is to see why the profile likelihood ratio $-2\ln\Lambda$ becomes $\chi^2_r$ as the sample grows — the result that converts a fit into a significance without simulating the null. The mechanism is a quadratic (Laplace) expansion of the log-likelihood about its maximum plus the asymptotic normality of the MLE, which together turn a difference of log-likelihoods into a sum of $r$ squared standard normals. Let $\hat\theta$ be the unconstrained MLE, $\hat\theta_0$ the MLE constrained to the $r$ -codimensional null surface $\Theta_0$ , and $\ell=\ln\mathcal{L}$ .

Step 1. Taylor-expand about $\hat\theta$ : $\ell(\theta)\approx\ell(\hat\theta)-\tfrac12(\theta-\hat\theta)^{\mathsf T}I(\hat\theta)(\theta-\hat\theta)$ , the linear term vanishing at the maximum, with $I$ the observed Fisher information. Why: near a smooth maximum the log-likelihood is a paraboloid, and the curvature $I$ is exactly the metric that standardises coordinate differences.

Step 2. Evaluate at the constrained maximum: $-2\ln\Lambda=2[\ell(\hat\theta)-\ell(\hat\theta_0)]\approx(\hat\theta-\hat\theta_0)^{\mathsf T}I(\hat\theta)(\hat\theta-\hat\theta_0)$ . Why: the statistic is just the "distance" from the unconstrained to the constrained estimate in the Fisher metric.

Step 3. Invoke the MLE asymptotics $\sqrt{n}(\hat\theta-\theta_{\text{true}})\xrightarrow{d}\mathcal{N}(0,I_1^{-1})$ ( $I_1$ the per-observation information). Under $H_0$ the displacement $\hat\theta-\hat\theta_0$ projects onto the $r$ constrained directions as a centred Gaussian whose Fisher-metric covariance is the identity — $r$ independent standard normals. Why: the metric $I$ whitens the Gaussian, so the quadratic form becomes an unweighted sum of squares.

Step 4. Therefore $(\hat\theta-\hat\theta_0)^{\mathsf T}I(\hat\theta-\hat\theta_0)\xrightarrow{d}\sum_{j=1}^r Z_j^2=\chi^2_r$ . One compares the observed $-2\ln\Lambda$ to $\chi^2_r$ quantiles, and for $r=1$ the significance in $\sigma$ is $\sqrt{-2\ln\Lambda}$ . Why: a whitened quadratic form in $r$ normals is a $\chi^2_r$ — the bridge from a likelihood fit to a tabulated tail probability. *As a consistency check: for $r=1$ , the values $-2\ln\Lambda=1,4,9,25$ give significances $1,2,3,5\sigma$ respectively, since $\sqrt{-2\ln\Lambda}=1,2,3,5$ , in agreement with the $\chi^2_1$ quantiles.*

Limiting cases

The standard tests are not separate inventions but special cases of the likelihood-ratio idea, selected by what is known and how the statistic is distributed:

Case	Result	Meaning
Known variance, Gaussian mean	$Z=\dfrac{\bar{X}-\mu_0}{\sigma/\sqrt{n}}\sim\mathcal{N}(0,1)$	The $z$ -test; two-sided rejection when $\lvert Z\rvert>z_{1-\alpha/2}$ ( $z_{0.975}=1.96$ ).
Unknown variance, small $n$	$T=\dfrac{\bar{X}-\mu_0}{s/\sqrt{n}}\sim t_{n-1}$	Student's $t$ -test; estimating $s$ fattens the tails. Recovers $z$ as $n\to\infty$ .
Binned counts / goodness of fit	$\chi^2=\sum_i\dfrac{(O_i-E_i)^2}{E_i}\sim\chi^2_{k-1-m}$	Pearson's test; $m$ fitted parameters cut the d.o.f. Large $\chi^2$ rejects the model.
Composite null, large $n$	$-2\ln\Lambda\;\xrightarrow{n\to\infty}\;\chi^2_{r}$	Wilks' theorem; $r$ constrained parameters — the basis of profile-likelihood significance.

Common trapThe p-value is the probability that H₀ is true

It is not. The $p$ -value is $P(\text{data at least this extreme}\mid H_0)$ — computed assuming $H_0$ — whereas $P(H_0\mid\text{data})$ is a posterior requiring a prior and Bayes' theorem to even define. A $p=0.05$ result does not mean a 5% chance $H_0$ is true; under realistic priors it can be far weaker evidence, sometimes barely 2:1 against the null. The trap runs the other way too: failing to reject $H_0$ is never proof it is true — a null result from an underpowered test is simply uninformative, the statistical equivalent of not having looked. Only when power against the relevant alternative is high does "fail to reject" carry weight.

Worked examples

1 · A measurement versus theory (confidence builder)

A precision experiment measures a quantity over $n=25$ runs, reporting a mean $\bar{X}=100.8$ in instrument units. Theory predicts $\mu_0=100.0$ , and the apparatus has a known per-run scatter $\sigma=2.0$ . Is the measurement in tension with theory at the 5% level?

Step 1. Form the standard error of the mean: $\sigma/\sqrt{n}=2.0/\sqrt{25}=0.4$ — how well 25 runs pin down the mean.

Step 2. Compute the $z$ -statistic: $Z=(100.8-100.0)/0.4=\mathbf{2.0}$ — the measurement sits two standard errors above the prediction.

Step 3. Convert to a two-sided $p$ -value (theory could err either way, so both tails count): $p=2[1-\Phi(2.0)]=2(0.02275)=\mathbf{0.0455}$ . Since $0.0455<0.05$ , you reject $H_0$ at $\alpha=0.05$ — but only barely; $2\sigma$ is "interesting, keep watching", not "discovery". Z = 2.0, two-sided p = 0.046, reject at 5% but far below the 5σ discovery bar.

2 · Goodness of fit of a die

A die is rolled $n=60$ times to test fairness. The six faces come up $8,12,9,11,14,6$ . Under $H_0$ each face is expected $E=60/6=10$ times. Is the die consistent with fairness?

Step 1. Compute Pearson's statistic with $E=10$ : the squared residuals are $4,4,1,1,16,16$ (summing to 42), so $\chi^2=42/10=\mathbf{4.2}$ .

Step 2. Degrees of freedom: $k=6$ bins, one constraint (counts sum to 60), no fitted parameters, so $\nu=k-1-m=6-1-0=\mathbf{5}$ .

Step 3. Compare to $\chi^2_5$ : the 5% critical value is $11.07$ , and $4.2\ll11.07$ , so you fail to reject ( $p\approx0.52$ ). The reduced $\chi^2/\nu=4.2/5=0.84\approx1$ confirms an unremarkable fit. χ² = 4.2 on 5 d.o.f., p ≈ 0.52: no evidence the die is loaded.

3 · Sizing an experiment for a target power

You plan a one-sided test of $H_0:\mu=\mu_0$ against $H_1:\mu>\mu_0$ at $\alpha=0.05$ . The smallest interesting shift is $\delta=\mu_1-\mu_0=0.5\sigma$ . How many samples $n$ are needed for power $1-\beta=0.90$ ?

Step 1. The design equation sets the cut $z_{1-\alpha}$ standard errors above $\mu_0$ and simultaneously $z_{1-\beta}$ below $\mu_1$ : $z_{1-\alpha}+z_{1-\beta}=\delta\sqrt{n}/\sigma$ , so $n=\sigma^2(z_{1-\alpha}+z_{1-\beta})^2/\delta^2$ .

Step 2. Insert the quantiles $z_{0.95}=1.645$ and $z_{0.90}=1.282$ , and the standardised effect $\delta/\sigma=0.5$ : $n=(1.645+1.282)^2/0.5^2=(2.927)^2/0.25=8.57/0.25=34.3$ . Round up to be safe: $\mathbf{n=35}$ .

Step 3. Check the delivered power at $n=35$ : the signal sits $\sqrt{35}\cdot0.5=2.96$ standard errors above $\mu_0$ ; subtracting the cut $z_{0.95}=1.645$ leaves $1.31$ of margin, and $\Phi(1.31)=0.905\ge0.90$ as required. n = 35 samples deliver 90% power to detect a 0.5σ shift at α = 0.05.

Physics of the portrait

Hypothesis testing has no single inventor; it is a two-decade argument among a handful of statisticians that hardened into the frequentist framework physics still runs on.

Origins · 1 minJerzy Neyman and how this was discovered

Jerzy Neyman

1894–1981 · frequentist hypothesis testing and the Neyman–Pearson lemma

Jerzy Neyman, born in the Russian Empire and trained in Kharkiv and Warsaw, carried the rigour of measure-theoretic probability into a field that in the 1920s was still a bag of recipes. The decisive collaboration began in 1926 with Egon Pearson — son of Karl Pearson, who had invented the $\chi^2$ goodness-of-fit test in 1900. Between 1928 and 1933 the two built testing as a decision problem: they named the two error types, defined power, and proved the lemma that bears their names — the likelihood ratio gives the most powerful test at fixed size. This broke sharply with Ronald Fisher, whose significance tests and $p$ -values (systematised in his 1925 Statistical Methods for Research Workers) framed testing as evidence against a null, with no explicit alternative and no power. The Fisher–Neyman feud was bitter and lifelong, yet the working physicist uses an unspoken hybrid: Fisher's $p$ -values and Neyman–Pearson's error rates in the same breath. Neyman also built confidence intervals (1937) as the dual of testing and, after emigrating, founded the Berkeley statistics department. The remaining threads came from William Gosset — "Student", the Guinness brewer who derived the $t$ -distribution in 1908 — and Samuel Wilks, whose 1938 theorem made $-2\ln\Lambda$ asymptotically $\chi^2$ and handed particle physics its significance calculator. A quarrel over the meaning of evidence became the quiet arithmetic behind every discovery claim.

History

1900

— Karl Pearson introduces the χ² goodness-of-fit statistic

1908

— William Gosset, as Student, publishes the t-distribution for small samples

1925

— Fisher systematises significance tests and p-values in Statistical Methods for Research Workers

1933

— Neyman and Egon Pearson prove the most-powerful-test lemma and formalise Type I and Type II errors

1938

— Samuel Wilks derives the asymptotic χ² distribution of the likelihood-ratio statistic

1937

— Neyman constructs confidence intervals as the dual of hypothesis tests

Going further

Texts. Cowan, Statistical Data Analysis (Oxford, 1998), the physicist's standard · Stone & Goldbart, Mathematics for Physics (Cambridge, 2009), statistics chapter · Arfken, Weber & Harris, Mathematical Methods for Physicists, 7th ed. · Riley, Hobson & Bence, Mathematical Methods for Physics and Engineering, 3rd ed., Ch. 31 · Kendall & Stuart, The Advanced Theory of Statistics, Vol. 2, for the full proofs · James, Statistical Methods in Experimental Physics, 2nd ed. Key numerical anchors: the standard-normal quantiles $z_{0.95}=1.645$ , $z_{0.975}=1.960$ , $z_{0.90}=1.282$ ; the discovery-ladder tails $1-\Phi(5)=2.87\times10^{-7}$ and $1-\Phi(3)=1.35\times10^{-3}$ . Worked-example results: $Z=2.0\Rightarrow p_{\text{2-sided}}=0.0455$ ; die $\chi^2=4.2$ on $\nu=5$ ( $p\approx0.52$ ); sample-size design $n=(1.645+1.282)^2/0.5^2=34.3\to35$ , delivering power $\Phi(1.31)=0.905$ .

Builds on: Random Variables and Probability Distributions · Cumulative Distribution Functions · Gaussian (Normal) Distribution · Central Limit Theorem · Conditional Probability and Bayes' Theorem. Leads to: Maximum Likelihood Estimation · Confidence Intervals · Bayesian Inference.

Ask the tutor: Ask me why the same $\bar{X}$ can reject a theory in one experiment yet leave it standing in another — the hidden lever is power, and I will show you how to compute it.

Is a smaller p-value stronger evidence against H₀?

Broadly yes, but with caveats. Under $H_0$ the $p$ -value is uniform on the unit interval, so a $p$ of $10^{-7}$ is genuinely harder to produce by chance than a $p$ of $0.05$ . But $p$ measures only incompatibility with $H_0$ , not the probability of $H_1$ : a tiny $p$ can come from a huge sample detecting a physically trivial effect, or from an unmodelled systematic masquerading as signal. And $p$ -values do not combine by averaging or multiplying — pooling results properly means combining likelihoods. Read $p$ as a calibrated surprise index, always paired with an effect size and the test's power.

When do I use a t-test instead of a z-test?

Use the $z$ -test only when the population standard deviation $\sigma$ is genuinely known — from a prior calibration, say — and the sample mean is Gaussian. Use the $t$ -test when you must estimate the scatter from the same data via $s$ . Replacing the fixed $\sigma$ by the random $s$ injects extra uncertainty, which Student's $t_{n-1}$ absorbs with heavier tails, so its critical values are larger for small $n$ : at $n=5$ the two-sided 5% value is $2.78$ versus the normal's $1.96$ . By $n\approx30$ they agree to a couple of percent, and for large samples the distinction is cosmetic.

What is the difference between a one-sided and a two-sided test?

It is a statement about the alternative. A one-sided test asks "is the mean greater than $\mu_0$ ?" and puts the whole budget $\alpha$ in one tail; a two-sided test asks "is the mean different, either way?" and splits $\alpha$ into two tails of $\alpha/2$ . So the two-sided critical value is larger ( $z_{0.975}=1.96$ versus $z_{0.95}=1.645$ at $\alpha=0.05$ ) and the two-sided $p$ -value is double the one-sided tail. Choose one-sided only when a deviation the other way would be meaningless — and choose it before seeing data, since switching post hoc silently doubles your true false-alarm rate.

Why exactly 5σ for a discovery — isn't that overkill?

It is deliberately conservative. First, the look-elsewhere effect: scanning hundreds of mass bins gives many chances to fluctuate, so a local $3\sigma$ bump somewhere is almost expected and the global significance is far weaker. Second, systematics are hard to model in the deep tails, where the Gaussian $-2\ln\Lambda\to\sigma$ conversion can itself drift, so a wide margin buys robustness. Third, the field's history is littered with $3$ – $4\sigma$ "signals" that evaporated. The $p=2.87\times10^{-7}$ bar is a sociological choice about acceptable false-discovery rates, not a law of nature; lower-stakes fields legitimately use $2\sigma$ or $3\sigma$ .

Can I prove H₀ true by failing to reject it?

No — the single most common logical slip in applied testing. Failing to reject means only that the data are not surprising under $H_0$ ; it cannot distinguish " $H_0$ is true" from "my test had no power". A study with a few noisy samples fails to reject almost any null, not because the null is right but because it was blind. The honest reading depends on power: 95% power to detect the smallest interesting effect and still nothing is real support for $H_0$ ; 20% power is simply uninformative. This is why serious null claims are reported as upper limits or confidence intervals — they quantify what effect sizes have been excluded.

How are hypothesis tests and confidence intervals related?

They are two views of one computation — a duality Neyman built deliberately. A two-sided level- $\alpha$ test of $H_0:\mu=\mu_0$ rejects exactly those $\mu_0$ lying outside the $100(1-\alpha)\%$ confidence interval for $\mu$ . So the interval is the set of null values the data would not reject: "reject every $\mu_0$ more than $1.96$ standard errors from $\bar{X}$ " is identical to "the 95% interval is $\bar{X}\pm1.96\,\sigma/\sqrt{n}$ ". This is why physicists increasingly quote an interval or upper limit rather than a bare $p$ -value: it carries the same decision plus the effect size and its uncertainty.