Errors in Inference

When we conduct a hypothesis test about a population proportion, we are making a decision under uncertainty. We either reject the null hypothesis H₀ or fail to reject it, based on sample data. Because our sample is never a perfect mirror of the entire population, there is always a chance we will reach the wrong conclusion — even when we follow every step of the procedure correctly. There are exactly two ways a hypothesis test can go wrong. A Type I error occurs when we reject a null hypothesis that is actually true — a 'false positive.' A Type II error occurs when we fail to reject a null hypothesis that is actually false — a 'false negative.' Understanding these errors is essential because every statistical decision in science, medicine, law, and business carries real consequences depending on which type of error is made. The probability of committing a Type I error is called the significance level, denoted α (alpha), and it is chosen by the researcher before the test is run. The probability of committing a Type II error is denoted β (beta). The complement of β, written 1 − β, is called the power of the test — the probability of correctly detecting a real effect. Balancing these errors is one of the central challenges of statistical inference.

Key takeaways
  • The Decision Table
  • Type I Error: False Positive
  • Type II Error: False Negative
  • The Trade-off Between α and β
  • Power of a Test

The Decision Table

Every hypothesis test produces one of four outcomes. The null hypothesis H₀ is either true or false in reality, and the test either rejects H₀ or fails to reject H₀. When H₀ is true and we fail to reject it, we make a correct decision. When H₀ is false and we reject it, we also make a correct decision (this is the power of the test). The two error cells are: (1) H₀ is true but we reject it — Type I error; (2) H₀ is false but we fail to reject it — Type II error.

Type I Error: False Positive

A Type I error is like a smoke alarm going off when there is no fire. In the context of proportions, suppose H₀: p = 0.50 and H_a: p > 0.50. If the true proportion really is 0.50 but our sample happened to give us an unusually high p̂ by chance, our test statistic might fall in the rejection region, causing us to incorrectly conclude that p > 0.50. The probability of this happening equals α. If we set α = 0.05, there is a 5% chance of a Type I error whenever H₀ is actually true.

Type II Error: False Negative

A Type II error is like a smoke alarm that fails to ring during a real fire. Using the same example, if the true proportion is p = 0.60 (H₀ is false), but our sample gives a test statistic that does not cross the critical value, we fail to reject H₀ and miss the real effect. The probability β depends on (1) how far the true proportion is from p₀, (2) the sample size n, and (3) the chosen α. A larger sample size reduces β because it gives the test more ability to detect a real difference.

The Trade-off Between α and β

Reducing α makes it harder to reject H₀, which lowers the Type I error rate but increases β (more Type II errors). Raising α has the opposite effect. The only way to reduce both simultaneously is to increase the sample size n. For a fixed n, choosing α = 0.01 instead of α = 0.05 means the test is more conservative: fewer false positives, but more false negatives. The right balance depends on the context — in medical testing, a false negative (missing a disease) can be far more costly than a false positive.

Power of a Test

The power of a hypothesis test is 1 − β, the probability of correctly rejecting a false null hypothesis. Higher power means a test is better at detecting real effects. Power increases when: (a) the sample size n is larger, (b) the significance level α is larger, or (c) the true proportion is further from p₀. When designing a study, statisticians often choose n large enough to achieve power of at least 0.80, meaning there is at most a 20% chance of a Type II error if the effect is real.

The Key Equation

P(Type I Error)=α,P(Type II Error)=β,Power=1−βP(\text{Type I Error}) = \alpha, \quad P(\text{Type II Error}) = \beta, \quad \text{Power} = 1 - \beta
  • \alpha — Significance level (probability (0 to 1)): The probability of rejecting H₀ when it is actually true. Chosen by the researcher before data collection — common values are 0.05 and 0.01.
  • \beta — Type II error probability (probability (0 to 1)): The probability of failing to reject H₀ when it is actually false. Depends on sample size, effect size, and alpha.
  • 1 - \beta — Power (probability (0 to 1)): The probability of correctly detecting a real effect (rejecting a false H₀). A well-designed study aims for power ≥ 0.80.

Worked Example

Problem. A school district claims that 70% of its students pass the state math exam (H₀: p = 0.70). A researcher suspects the true pass rate is lower and tests H_a: p < 0.70 at α = 0.05. She samples n = 100 students and finds p̂ = 0.63. The test statistic is z = (0.63 − 0.70) / √(0.70 × 0.30 / 100) ≈ −1.53. The critical value for a left-tailed test at α = 0.05 is z* = −1.645. She fails to reject H₀. Suppose the true proportion is actually p = 0.63. Identify whether a Type I or Type II error might have occurred and explain.

z=p^−p0p0(1−p0)n=0.63−0.700.70×0.30100≈−1.53>−1.645⇒Fail to reject H0z = \frac{\hat{p} - p_0}{\sqrt{\frac{p_0(1-p_0)}{n}}} = \frac{0.63 - 0.70}{\sqrt{\frac{0.70 \times 0.30}{100}}} \approx -1.53 > -1.645 \Rightarrow \text{Fail to reject } H_0

Answer. A Type II error occurred: the researcher failed to reject a false null hypothesis.

Figure
Diagram
Hypothesis Test Decision TableH₀ is TRUE(Reality)H₀ is FALSE(Reality)Fail toReject H₀Reject H₀(Decision)✓ CorrectP = 1 − αType II ErrorFalse NegativeP = βType I ErrorFalse PositiveP = α✓ CorrectPowerP = 1 − βDecision →← Reality →
Origins · 1 minJerzy Neyman and how this was discovered
JN
Jerzy Neyman
1894–1981 · Polish-American

Neyman, together with Egon Pearson, developed the formal framework of hypothesis testing that introduced the concepts of Type I and Type II errors, the significance level α, and the power of a test. Their 1933 paper established the Neyman-Pearson lemma, which provides the mathematical foundation for choosing the most powerful test for a given α. Before their work, hypothesis testing lacked a rigorous framework for comparing errors. Neyman originally trained as a mathematician in Poland and later moved to the United States after World War II, where he founded the Statistics Department at UC Berkeley and built it into one of the most prestigious in the world.

  • Hypothesis Testing for Proportions — Type I and Type II errors arise specifically within hypothesis tests. Understanding how to set up and run a one-proportion z-test is the prerequisite for understanding where these errors occur.
  • Significance Level and P-Values — The significance level α directly defines the probability of a Type I error. Knowing how to compute and interpret a p-value is essential for understanding when and how Type I errors occur.
  • Confidence Intervals for Proportions — Confidence intervals and hypothesis tests are closely related: the confidence level (1 − α) connects directly to the Type I error rate, reinforcing the meaning of α across both approaches.
  • Sample Size and Power — Power (1 − β) is the complement of the Type II error probability. Choosing an appropriate sample size to achieve sufficient power is the practical application of understanding β.