Data Collection

Data collection is the first and most consequential stage of any statistical investigation: the deliberate gathering of observations or measurements that will later be summarized, displayed, and analyzed. It matters because every downstream conclusion inherits the quality of the original data — the principle of "garbage in, garbage out." A representative sample of a few thousand people can predict a national election within a few percentage points, while a poorly designed survey of millions can be wildly wrong. The surprising fact: in 1936 a magazine poll of 2.4 million respondents predicted the wrong U.S. presidential winner, while George Gallup's scientifically sampled poll of just 50,000 got it right — proving that how you collect data matters far more than how much.

The Core Insight

The goal of data collection is to obtain a sample that faithfully mirrors the population you care about. The single most important tool for achieving this is randomization — using chance to select who or what gets measured. Random selection is the only method that systematically eliminates the silent, invisible biases that creep in whenever humans choose for themselves what to measure.

Formal Statement

Population, Sample, and Census

• Population ($N$): the entire group of individuals or items we want to learn about.
• Sample ($n$): the subset of the population we actually collect data from.
• Census: data collection from every member of the population ($n=N$).
• Parameter: a numerical fact about the population (usually unknown), e.g. the true mean $\mu$.
• Statistic: a numerical fact computed from the sample, e.g. the sample mean $\bar{x}$, used to estimate the parameter.

The sampling fraction tells us what proportion of the population we have measured:

For a simple random sample of size $n$ drawn without replacement from $N$ individuals, the number of distinct possible samples is given by the binomial coefficient:

In a simple random sample, every individual has the same selection probability:

Types of Sampling Methods

• Simple Random Sample (SRS): every possible sample of size $n$ is equally likely.
• Stratified Sample: divide the population into homogeneous groups (strata), then take an SRS within each.
• Cluster Sample: divide into groups (clusters), randomly select whole clusters, sample everyone within them.
• Systematic Sample: choose a random starting point, then select every $k$-th individual.
• Convenience Sample: select whoever is easiest to reach (non-random, biased).
• Voluntary Response Sample: subjects choose to participate themselves (non-random, biased).

For systematic sampling, the interval $k$ is computed from the desired sample size:

The Representativeness Principle

If a sample is selected by a probability method (every individual has a known, nonzero chance of selection), then as the sample size $n$ increases, sample statistics converge toward the true population parameters. Crucially, no convergence is guaranteed for non-probability samples, regardless of size.

Why It Holds / Derivation

Why Randomization Defeats Bias

1. Define bias. A method is biased if it systematically over- or under-represents some part of the population, so that the expected value of the statistic differs from the parameter: $E[\bar{x}]\ne \mu$.

1. Identify the source. Bias enters when selection probability correlates with the variable being measured. Example: phoning landlines under-samples young people, who differ in their answers.

1. Apply randomization. Under an SRS, selection is determined purely by chance — a coin flip or random number — which is independent of any characteristic of the individual.

1. Conclude unbiasedness. Because selection is independent of the measured variable, every subgroup is represented in proportion to its true size on average: $E[\bar{x}]=\mu$.

1. Note remaining variability. Randomization does not eliminate random sampling error (the natural variation from sample to sample), but this error shrinks predictably as $n$ grows and is captured by a margin of error — unlike bias, which never shrinks. ∎

The margin of error for a proportion at 95% confidence illustrates how random error (not bias) decreases with sample size:

Ronald A. Fisher | 1890–1962

Though a statistician rather than a physicist, Fisher revolutionized data collection through his work at the Rothamsted agricultural research station in England (from 1919). Frustrated by decades of messy farm data, he developed the principles of randomization, replication, and blocking in his 1925 book Statistical Methods for Research Workers and 1935's The Design of Experiments. His insight — that you must design how data is collected before you collect it — transformed experimentation from passive observation into rigorous science. The famous "Lady Tasting Tea" thought experiment, in which a woman claimed to detect whether milk or tea was poured first, became his archetypal lesson in randomizing the order of trials.

Building Intuition

Imagine a giant pot of soup. To judge its saltiness, you don't drink the whole pot (a census — wasteful) — you take a spoonful (a sample). But the spoonful only tells you about the whole pot if you stir first. Stirring is randomization: it ensures your spoonful isn't drawn from an unusually salty corner. A bigger spoon (larger $n$) gives a slightly more reliable taste, but an unstirred pot will fool you no matter how big the spoon. This is why a tiny, well-stirred sample beats a huge, unstirred one — exactly the lesson of the 1936 election disaster.

Population, sampling methods, and the bias-vs-error distinction

Worked Examples

Systematic Sampling of a School

1. Compute the sampling interval using equation [4]:

1. Number all students $1$ to $1200$ (e.g., by alphabetical list).
2. Randomly choose a starting point between $1$ and $15$ — say a random draw gives $7$.
3. Select students numbered $7,22,37,52,\dots$ (add 15 each time).
4. The last selected: $7+15(79)=7+1185=1192$ ✓ (within range).

Margin of Error for a Poll

1. Apply equation [5]:

1. Compute the variance term: $(0.52)(0.48)=0.2496$.
2. Divide by $n$: $0.2496/1000=0.0002496$.
3. Take the square root: $\sqrt{0.0002496}\approx 0.0158$.
4. Multiply by 1.96: $1.96\times 0.0158\approx 0.0310$.

So the true support is plausibly $52\%\pm 3.1\%$, i.e., between about $48.9\%$ and $55.1\%$ — note this crosses 50%, so the lead is not statistically decisive.

Common Misconceptions

Bigger Is Not Always Better

A common error is believing that a larger sample is automatically more trustworthy. Sample size only controls random error, never bias. A biased collection method (e.g., an online poll where only motivated people respond) produces a systematically wrong answer that gets no closer to the truth as $n$ grows — it just becomes confidently wrong. The 1936 Literary Digest poll's 2.4 million responses lost to Gallup's 50,000 precisely because the Digest's list (car and phone owners) was biased toward wealthier, Republican-leaning voters during the Great Depression.

The Voluntary Response Trap

A TV news show asks viewers to call a number to vote "yes" or "no" on a controversial law. 100,000 people respond, with 78% voting "no." Is 78% a good estimate of public opinion? No. Only people with strong feelings — especially anger — bother to call (and they can call multiple times). This voluntary response sample systematically over-represents the most motivated, intense opinions. The huge $n$ is worthless; a true SRS of just 1,000 people would give a far more accurate picture. Large size cannot rescue a non-random method.

Confusing Observational Studies with Experiments

Collecting data by observing a group (an observational study) can reveal associations but cannot establish causation, because of lurking variables. Only a randomized experiment — where the researcher actively assigns treatments by chance — can establish cause and effect. "Coffee drinkers live longer" (observed) does not mean coffee causes longevity; perhaps coffee drinkers are also wealthier or exercise more.

Connections

Data collection is the upstream stage that feeds every other part of statistics. The numbers it produces are summarized by Measures of Central Tendency and Measures of Spread, displayed using Histograms and Frequency Distributions, and interpreted through Probability Distributions. The reliability of any Confidence Interval or Hypothesis Testing result depends entirely on the data having been collected by a valid probability method — these inference tools assume random sampling. The deliberate planning of how to collect data is the subject of Experimental Design, while the chance mechanisms underpinning randomization come from Probability Theory.

Real-World Applications

1. Political polling — Gallup, Pew Research, and YouGov use stratified random sampling to forecast elections from ~1,000–2,000 respondents.
2. Census-taking — The U.S. Census Bureau attempts a full census every 10 years, then uses sample-based surveys (the American Community Survey) for yearly detail.
3. Clinical drug trials — Pharmaceutical companies use randomized controlled experiments (FDA-mandated) where patients are randomly assigned drug or placebo.
4. Quality control in manufacturing — Factories use systematic sampling (e.g., test every 50th item) to monitor defect rates without inspecting everything.
5. Television ratings — Nielsen places meters in a randomly selected sample of households to estimate national viewership.
6. Ecology / wildlife counts — Biologists use the capture-recapture method to estimate animal populations they cannot count directly.
7. A/B testing in tech — Companies like Netflix and Amazon randomly assign users to different website versions to measure which design performs better.

Historical Development

• 1086 CE — The Domesday Book, an early attempted census of England commissioned by William the Conqueror.
• 1790 — The first U.S. Census is conducted, mandated by the Constitution for apportioning representatives.
• 1895 — Norwegian statistician Anders Kiaer proposes the "representative method" of sampling, sparking debate about whether sampling could replace a full census.
• 1906 — W. S. Gosset ("Student"), working at Guinness, develops small-sample methods, recognizing that data collection often involves limited samples.
• 1925–1935 — Ronald Fisher publishes the foundational principles of experimental design: randomization, replication, and blocking.
• 1934 — Jerzy Neyman presents the rigorous theory of stratified random sampling and confidence intervals to the Royal Statistical Society.
• 1936 — George Gallup correctly predicts Roosevelt's victory with a 50,000-person scientific sample, while the Literary Digest's 2.4-million biased poll fails — a landmark demonstration of method over size.
• 1948 — Polls (including Gallup's) wrongly predict Dewey over Truman, exposing flaws in quota sampling and pushing the field toward strict probability sampling.
• 1990s–present — The rise of telephone, then internet and "big data" collection raises new challenges of coverage bias and self-selection in the digital age.

Subtleties & Limits

Modern data collection faces deep, unresolved challenges. Nonresponse bias is growing: as people screen calls and ignore surveys, response rates have fallen below 10% for many telephone polls, and those who do respond may differ systematically from those who don't — reintroducing bias that randomization was meant to prevent. Big data appears to solve sample-size problems, but massive datasets (social media, web logs) are almost never random samples of any meaningful population, making them prone to systematic error that their size only disguises. The frame problem — obtaining an accurate, complete list of the population to sample from — remains unsolved for hard-to-reach groups (homeless populations, undocumented migrants). And in an era of privacy concerns, techniques like differential privacy deliberately add noise to collected data, forcing statisticians to weigh accuracy against individual confidentiality — an active research frontier.

See Also