p-values and Significance

P-values remain the standard evidentiary currency of clinical trials, where regulatory agencies require pre-registered significance thresholds (often $\alpha=0.05$, sometimes stricter for multiple endpoints) before a drug can claim efficacy. In particle physics, the discovery of the Higgs boson in 2012 was reported using a p-value corresponding to a "5-sigma" threshold ($\alpha \approx 3\times 10^{-7}$), reflecting the much stricter conventions physicists use partly because of the enormous number of implicit comparisons in a collider search. In genomics and neuroimaging, the multiple-testing problem — computing p-values for thousands or millions of loci or voxels simultaneously — has driven the development of false-discovery-rate methods (Benjamini–Hochberg) as a companion to, and partial replacement for, per-test significance. In machine learning and A/B testing at technology companies, p-value-based stopping rules for online experiments must be corrected for sequential peeking, connecting directly to the martingale theory behind sequential hypothesis testing. Cryptographic randomness testing (e.g. the NIST statistical test suite) applies p-value machinery to test whether a bit generator's output is distinguishable from a true uniform random source, directly using the Uniform(0,1) theorem proved above. All of these applications build on exactly the machinery formalized here, and are made computationally concrete via Standard Tests: t, Chi-Squared, and F and generalized via Likelihood Ratio Tests.

Ask the tutor: How does the p-value framework generalize into the likelihood-ratio test statistic used for composite hypotheses with nuisance parameters?

Picture the null hypothesis as generating a known distribution of possible test statistics — a bell curve, say, or some other shape entirely determined by $H_0$. The observed data produce one point on this distribution. The p-value is the tail area beyond that point: the total probability mass, under the null's distribution, of getting a result as extreme or more extreme than what was actually seen. A p-value of $0.03$ says: "if $H_0$ were exactly true and I repeated this experiment many times, only about $3\%$ of the time would I see a statistic this extreme or more so." The key mental trap is direction of conditioning. The p-value conditions on the null being true and asks about the data; it never conditions on the data and asks about the null. Bayes' theorem shows these two conditional probabilities can differ enormously depending on the prior plausibility of $H_0$ — a fact demonstrated numerically below. A second useful picture: significance level $\alpha$ is a line drawn in the sand before the experiment, fixing how much Type I error risk is tolerable. The p-value is where the data actually landed relative to that line. Confusing the two — treating the p-value itself as a post-hoc adjustable threshold, or treating $\alpha$ as a statement about this one data set — is the second classic trap.

Example 1 — one-sided z-test with asymmetric numbers. A factory claims its light bulbs have mean lifetime $\mu_0 = 1000$ hours. An engineer suspects a new manufacturing defect has reduced mean lifetime, so she tests

Lifetimes are modeled as $X_i \sim N(\mu, \sigma^2)$ with known $\sigma = 84$ hours (from long production history). She samples $n = 49$ bulbs and observes sample mean $\bar{x} = 976.4$ hours. The natural test statistic is

Under $H_0$, $Z \sim N(0,1)$ exactly (since $X_i$ are normal with known variance). Small (very negative) $Z$ is evidence against $H_0$ in favor of $\mu < 1000$, so this is a left-tailed test; the p-value is $P(Z \le z_{\mathrm{obs}})$. Compute:

So $z_{\mathrm{obs}} \approx -1.9667$. The p-value is

where $\Phi$ is the standard normal CDF. From standard normal tables, $\Phi(-1.96) = 0.0250$ and $\Phi(-1.97) \approx 0.0244$; interpolating at $-1.9667$ gives

So $p \approx 0.0246$.

Interpretation. If the true mean lifetime were exactly $1000$ hours, a sample mean at least as far below $1000$ as $976.4$ (using this exact $n$ and $\sigma$) would occur with probability about $2.46\%$. At a pre-declared significance level $\alpha = 0.05$, since $0.0246 \le 0.05$, the engineer rejects $H_0$ and concludes the data are statistically significant evidence of reduced lifetime. Note what this does not say: it does not say there is a $2.46\%$ chance the true mean is $1000$, and it does not by itself say the reduction (23.6 hours, about $2.4\%$ of nominal life) is of practical importance — that depends on engineering tolerances, a separate judgment from the p-value.

Example 2 — two-sided test with a discrete statistic, illustrating conservativeness. A quality-control officer wants to test whether a packing machine mislabels boxes at a rate different from the historical baseline of $\pi_0 = 0.10$. She inspects $n = 20$ boxes and finds $k = 1$ mislabeled box. Model $K \sim \mathrm{Binomial}(20, \pi)$, testing

This is a two-sided test with a discrete statistic, so a common convention (not the only one) defines the p-value via the exact test: sum the probabilities of all outcomes $k'$ that are at least as extreme as the observed $k=1$, where "extreme" is measured by having binomial probability under $H_0$ no larger than $P(K=1)$. First compute the null pmf at relevant points, $\pi_0 = 0.10$, $n=20$:

• $P(K=0) = (0.9)^{20}$. Compute $(0.9)^{20}$: $(0.9)^{10} \approx 0.348678$, so $(0.9)^{20} \approx 0.348678^2 \approx 0.121577$.
• $P(K=1) = 20 (0.1)(0.9)^{19} = 2 (0.9)^{19}$. Since $(0.9)^{19} = (0.9)^{20}/0.9 \approx 0.121577/0.9 \approx 0.135086$, we get $P(K=1) \approx 2 \times 0.135086 = 0.270171$...

Wait — recompute carefully: $20 \times 0.1 = 2$, and this multiplies $(0.9)^{19}$, so $P(K=1) = 2 \times 0.135086 = 0.270171$.

• $P(K=2) = \binom{20}{2}(0.1)^2(0.9)^{18} = 190 \times 0.01 \times (0.9)^{18}$. $(0.9)^{18} = (0.9)^{19}/0.9 \approx 0.135086/0.9 \approx 0.150096$. So $P(K=2) \approx 190 \times 0.01 \times 0.150096 = 1.9 \times 0.150096 \approx 0.285182$.

We need all $k'$ with $P(K=k') \le P(K=1) \approx 0.270171$. Check both tails: Lower tail: $P(K=0) \approx 0.121577 \le 0.270171$. Yes, include $k'=0$. $P(K=1) = 0.270171$ itself, included (equal). There is no $k' < 0$. Upper tail: we need $k' \ge 2$ with $P(K=k') \le 0.270171$. We found $P(K=2) \approx 0.285182 > 0.270171$, so $k'=2$ is excluded (it is actually more probable than the observed value, hence not "as extreme"). Check $P(K=3)$:

$(0.9)^{17} = (0.9)^{18}/0.9 \approx 0.150096/0.9 \approx 0.166773$. So $P(K=3) \approx 1.14 \times 0.166773 \approx 0.190121 \le 0.270171$. Include $k'=3$. Since $P(K)$ is unimodal (increasing then decreasing near the mode $n\pi_0 = 2$), and $P(K=2) > P(K=1)$ while $P(K=3) < P(K=1)$, all $k' \ge 3$ automatically have $P(K=k') < P(K=3) < P(K=1)$ in the decreasing tail (binomial tail probabilities decrease monotonically past the mode for $\pi_0$ fixed and $k'$ large — this can be checked via the ratio $P(K=k'+1)/P(K=k') = \frac{n-k'}{k'+1}\cdot\frac{\pi_0}{1-\pi_0}$, which is less than $1$ once $k'$ exceeds the mode, confirming monotone decrease), so every $k' \ge 3$ qualifies as "at least as extreme." The p-value is

We have $\sum_{k'=0}^2 P(K=k') \approx 0.121577+0.270171+0.285182 = 0.676930$, so $P(K\ge 3) \approx 1-0.676930 = 0.323070$. Therefore

Interpretation. With $p \approx 0.715$, the data (1 mislabeled box out of 20) are entirely consistent with the baseline rate $\pi_0=0.10$; there is no evidence to reject $H_0$ at any conventional level. This example also illustrates the conservativeness built into exact discrete tests: because the pmf is discrete, the achievable p-values form a finite set, and the actual Type I error rate of the "reject if $p \le \alpha$" rule can be strictly less than $\alpha$ (the CDF has jumps, so the continuity hypothesis of the Uniform theorem fails, and only the inequality $P_{\theta_0}(\text{reject}) \le \alpha$ from discreteness is guaranteed, not equality).

Numerical illustration of the inversion fallacy. Suppose a screening test for a rare biomarker anomaly is run on many independent gene locations, each with prior probability $\pi = P(H_0 \text{ false, i.e. real effect}) = 0.01$ (only $1\%$ of tested loci harbor a true effect). Suppose tests with a real effect yield $p\le 0.05$ with probability (power) $0.6$, while tests with no effect yield $p \le 0.05$ with probability exactly $0.05$ (by the Uniform theorem, since $\alpha=0.05$). Among $10{,}000$ loci tested, expected counts:

• True effects: $100$ loci; of these, $60$ show $p\le 0.05$.
• No effect: $9{,}900$ loci; of these, $9900\times 0.05 = 495$ show $p\le 0.05$ purely by chance.

Among all $60+495 = 555$ loci with $p\le 0.05$, only $60$ actually have a real effect:

So even though each individual test controls Type I error at $5\%$, roughly $89\%$ of the "significant" findings in this rare-effect regime are false positives — the p-value threshold alone says nothing about this ratio without the prior $\pi$ and the power.