Planning a Statistical Study

Planning a statistical study is the foundation of all data-driven inquiry. Before a single measurement is taken, a researcher must articulate a clear research question, identify the population of interest, and choose a method of data collection that will produce reliable, unbiased results. Poor planning at this stage cannot be fixed later — no amount of sophisticated analysis can rescue a study that was poorly designed from the start. Statistical studies fall into two broad categories: observational studies, in which researchers record data without interfering with the subjects, and experiments, in which researchers deliberately impose treatments to measure their effect. The choice between these approaches depends on the research question, ethical constraints, and practical resources. An experiment can establish cause-and-effect relationships; an observational study generally can only identify associations. High-quality study design requires thinking carefully about variables (explanatory vs. response), potential sources of bias, how to select a representative sample, and how to control for confounding factors. Understanding these principles allows you to critically evaluate studies reported in the media, identify flaws in reasoning, and design your own investigations with integrity and statistical power.

Key takeaways
  • Formulate a Clear Research Question
  • Identify the Population and Choose a Sampling Method
  • Choose Observational Study or Experiment
  • Control Confounding and Reduce Bias
  • Define Variables and Plan Data Collection Instruments

Formulate a Clear Research Question

Every study begins with a specific, answerable question. A good research question identifies the population (who or what is being studied), the variables of interest, and what relationship or characteristic is under investigation. Example: 'Does daily exercise of at least 30 minutes reduce resting heart rate in adults aged 18–40?' This is better than 'Is exercise good for you?' because it specifies the population, the treatment (30 min/day exercise), and the response variable (resting heart rate).

Identify the Population and Choose a Sampling Method

The population is the entire group about which you want to draw conclusions; the sample is the subset you actually measure. To avoid bias, use probability sampling. In a Simple Random Sample (SRS), every individual has an equal chance of selection. In stratified sampling, the population is divided into homogeneous subgroups (strata) and an SRS is taken from each. In cluster sampling, naturally occurring groups are randomly selected and all members are studied. Convenience samples (grabbing whoever is nearby) introduce selection bias and should be avoided in rigorous work.

Choose Observational Study or Experiment

In an observational study, you measure variables as they naturally occur — no treatment is assigned. Observational studies include surveys, cohort studies (following a group forward in time), case-control studies (comparing people with/without an outcome), and cross-sectional studies (snapshot at one moment). In a controlled experiment, you randomly assign subjects to treatment groups, which allows you to control confounding variables and establish causation. The key principle: random assignment ≠ random sampling. Random assignment controls for confounding; random sampling ensures generalizability.

Control Confounding and Reduce Bias

A confounding variable is associated with both the explanatory variable and the response variable, potentially distorting results. For example, in a study linking ice cream sales to drowning rates, temperature is a confounder — it drives both. In experiments, randomization distributes confounders evenly across groups. Blinding (single-blind: subjects don't know their group; double-blind: neither subjects nor evaluators know) reduces bias from expectations. Replication — repeating the experiment on many subjects or in multiple trials — improves reliability.

Define Variables and Plan Data Collection Instruments

Identify the explanatory variable (also called the independent or predictor variable) and the response variable (dependent variable). Determine how each will be measured (operational definition). For example, 'resting heart rate' must be defined precisely: beats per minute measured after sitting quietly for 5 minutes. Decide on the sample size n needed for sufficient statistical power (see the equation block). Plan the data recording protocol to ensure consistency and minimize measurement error.

The Key Equation

n=z∗2⋅p^(1−p^)ME2n = \frac{z^{*2} \cdot \hat{p}(1-\hat{p})}{ME^2}
  • n — Sample size (individuals): The number of subjects needed in the sample to achieve the desired precision.
  • z^ — Critical value (dimensionless): The z-score corresponding to the desired confidence level (e.g., z = 1.96 for 95% confidence).
  • \hat{p} — Estimated proportion (dimensionless (0 to 1)): Prior estimate of the population proportion; use 0.5 when unknown to maximize n and be conservative.
  • ME — Margin of error (dimensionless (proportion scale)): The maximum acceptable difference between the sample estimate and the true population proportion.

Worked Example

Problem. A student wants to estimate the proportion of students at her school of 1,200 who use social media for more than 3 hours daily. She wants a 95% confidence level and a margin of error no greater than 4 percentage points. She has no prior estimate of the proportion. (a) What sample size n does she need? (b) Should she use an observational study or an experiment for this question? Justify.

nadjusted=n01+n0/N=6011+601/1200≈401n_{\text{adjusted}} = \frac{n_0}{1 + n_0/N} = \frac{601}{1 + 601/1200} \approx 401

Answer. She needs approximately 401 students (with finite population correction) using a survey-based observational study.

Figure
Diagram
Study Design Decision FrameworkResearch QuestionDefine population & variablesCan you assign treatments?Ethical? Practical? Feasible?NOYESObservational Study• Survey / Census• Cohort study• Case-controlShows association onlyExperiment• Random assignment• Control group• BlindingCan show causationBoth require: sampling plan, defined variables, replication
Origins · 1 minRonald A. Fisher and how this was discovered
RA
Ronald A. Fisher
1890–1962 · British

Fisher essentially invented the modern framework of experimental design. While working at Rothamsted Agricultural Research Station in the 1920s, he developed the principles of randomization, replication, and blocking to control confounding in agricultural experiments. His 1935 book 'The Design of Experiments' formalized these ideas, including the concept of the null hypothesis and the analysis of variance (ANOVA). His framework is the basis of virtually every controlled experiment conducted today. Fisher designed the famous 'Lady Tasting Tea' experiment to test whether a colleague (Muriel Bristol) could truly tell if milk was poured into tea before or after — an elegant example of hypothesis testing using a perfectly planned experiment with a known distribution of outcomes.

  • Sampling Methods — Study design requires choosing a sampling strategy (SRS, stratified, cluster) to ensure representativeness and minimize selection bias.
  • Bias and Variability — Understanding bias (systematic error) and variability (random error) is essential for evaluating whether a study design will produce accurate and precise estimates.
  • Principles of Experimental Design — Randomization, control, and replication are the core principles that distinguish experiments from observational studies and enable causal inference.
  • Confidence Intervals for Proportions — The sample size formula used in study planning directly connects to constructing confidence intervals with a specified margin of error.