Choosing an Experimental Design
When scientists and statisticians want to investigate cause-and-effect relationships, they must plan their experiments carefully before collecting a single data point. Choosing the right experimental design is one of the most important decisions in statistics, because a poorly designed study can produce misleading results no matter how carefully the data are later analyzed. The three core principles underlying any good experiment are randomization (randomly assigning subjects to treatments), replication (using enough subjects so results are reliable), and control (holding extraneous variables constant or accounting for them). The simplest experiment assigns subjects completely at random to treatment groups — this is called a completely randomized design. However, when subjects naturally fall into groups that are similar to each other (called blocks), or when you can pair each subject with a very similar partner, more refined designs become available. A randomized block design groups similar experimental units together before randomly assigning treatments within each group, while a matched pairs design pairs each subject with either itself (repeated measures) or a closely matched partner. Choosing between these designs depends on the sources of variability in your data. Understanding which design to select requires analyzing the structure of your experiment: How many explanatory variables are there? Are there lurking variables tied to identifiable groups? Can subjects serve as their own controls? Answering these questions guides statisticians toward designs that maximize statistical power — the ability to detect a real effect — while keeping the experiment practical. This article walks through the key design choices, their mathematics, and when to apply each.
- Completely Randomized Design (CRD)
- Randomized Block Design (RBD)
- Matched Pairs Design
- Decision Framework: Which Design to Choose?
Completely Randomized Design (CRD)
In a CRD, all experimental units are randomly assigned to one of the treatment groups with no prior grouping. If there are k treatments and n total units, each unit is independently assigned so that each treatment receives n/k units (assuming equal group sizes). This design is appropriate when subjects are relatively homogeneous — meaning there are no major lurking variables tied to identifiable subgroups. Randomization itself is expected to balance out any chance differences between groups. The key advantage is simplicity; the key disadvantage is that if subjects vary greatly (e.g., by age, gender, or initial health), that variability increases experimental error and makes it harder to detect treatment effects.
Randomized Block Design (RBD)
When subjects can be grouped into blocks — subsets that are similar with respect to a variable known to affect the response — a randomized block design is preferred. Within each block, subjects are randomly assigned to treatments. For example, if you are testing three fertilizers on crops in fields with varying soil quality, group the fields by soil type (the blocking variable), then randomly assign the three fertilizers within each soil-type group. Blocking removes the variation due to soil type from the error term, making treatment comparisons sharper. Formally, if there are b blocks and k treatments, you need b × k experimental units (one per treatment per block). The variation among blocks is accounted for separately, reducing the residual variance and increasing the power of the F-test used to compare treatments.
Matched Pairs Design
A matched pairs design is a special case of a randomized block design where each block contains exactly two experimental units that are paired based on similar characteristics (age, pre-test score, body weight, etc.). One unit in each pair is randomly assigned to Treatment A and the other to Treatment B. Alternatively, the same subject receives both treatments in random order (a crossover design), acting as their own control. The analysis focuses on the differences within each pair: d_i = x_{A,i} − x_{B,i} for pair i. Because between-subject variability is eliminated, matched pairs designs are very powerful for detecting small treatment effects. They are ideal when subjects are scarce and pairing is practical, but the matching variable must genuinely be correlated with the response variable, or the pairing wastes degrees of freedom.
Decision Framework: Which Design to Choose?
Use a CRD when: (1) subjects are homogeneous, or (2) no clear blocking variable exists. Use an RBD when: (1) you can identify a categorical variable (e.g., sex, school, lab batch) strongly related to the response, and (2) you have enough subjects to fill at least two complete blocks. Use matched pairs when: (1) subjects can be meaningfully paired one-to-one (e.g., twins, left eye vs. right eye, before-and-after), and (2) you have only two treatment levels. A practical rule: if blocking reduces variability among the units within each block substantially (i.e., if the blocking variable accounts for a large portion of total variability), then blocking improves precision enough to be worthwhile. If blocks are nearly as variable as the whole sample, blocking wastes degrees of freedom and hurts the analysis.
The Key Equation
- \bar{d} — Mean difference (same as response variable): The average of the differences d_i = x_{A,i} − x_{B,i} computed for each pair i.
- s_d — Sample standard deviation of differences (same as response variable): Measures how much the within-pair differences vary from pair to pair.
- n — Number of pairs (dimensionless): The total number of matched pairs in the study; degrees of freedom = n − 1.
- t — t-statistic (dimensionless): Compared to a t-distribution with n − 1 degrees of freedom to obtain a p-value.
Worked Example
Problem. A researcher wants to test whether a new reading program improves students' reading scores. She has 24 students divided into 3 schools (8 students each). She believes school environment strongly affects reading performance. Should she use a CRD or RBD? If she uses an RBD, describe the assignment process and calculate how many students receive each of the two treatments (program vs. control).
Answer. Use a Randomized Block Design with schools as blocks. Randomly assign 4 students per treatment within each school, giving 12 students per treatment overall.
Origins · 1 minRonald A. Fisher and how this was discovered
Fisher invented the randomized block design and analysis of variance (ANOVA) while working at Rothamsted Experimental Station in the 1920s. His 1935 book 'The Design of Experiments' introduced the principles of randomization, replication, and blocking that form the foundation of all modern experimental design. He formally defined the concept of the null hypothesis and the F-distribution used to test treatment effects. Fisher developed many of his foundational ideas by analyzing decades of agricultural data stored at Rothamsted — some records dated back to the 1840s. He was also an avid pipe smoker who famously dismissed early evidence linking smoking to lung cancer, a controversy that followed him for the rest of his career.
Related Concepts
- Randomization and Random Assignment — Randomization is the mechanism within every experimental design that prevents bias; understanding how to carry it out is essential before choosing a design.
- Control Groups and the Placebo Effect — Every experimental design must include a control group; the placebo effect explains why blinding subjects to their treatment assignment is critical.
- Sampling Methods — Before designing an experiment, researchers must understand how their sample was selected, since sampling method affects whether conclusions generalize to a population.
- Matched Pairs t-Test — The statistical inference procedure for matched pairs designs; students learn how to test whether the mean within-pair difference is significantly different from zero.
- Confounding Variables and Lurking Variables — Blocking and pairing are strategies specifically designed to neutralize confounding variables; recognizing confounders motivates the choice of RBD or matched pairs over a CRD.