Sampling Bias

Sampling bias occurs when the way a sample is selected from a population systematically over- or under-represents certain groups, so the sample is not representative of the population. Key result: a biased sample produces estimates that are systematically wrong — increasing sample size does NOT fix it. Applies wherever data is collected to make inferences: polls, surveys, medicine, science, machine learning. First rigorously diagnosed in the 1936 Literary Digest poll and 1948 Dewey/Truman failures.

Sampling bias is a flaw in the data-collection process in which some members of the intended population are more or less likely to be included in the sample than others, in a way correlated with the quantity being measured. Because the distortion is built into how data is gathered, every sample drawn the same way leans the same wrong direction — the error is systematic, not random. This matters enormously: nearly all of statistics assumes samples represent populations, and sampling bias quietly breaks that assumption, producing confident-looking but wrong conclusions. The surprising and crucial fact is that a huge biased sample is worse than a small random one, because the large sample gives false confidence (tight confidence intervals) around a wrong answer.

The Core Insight

Statistical inference rests on one bargain: if the sample is representative, the sample statistic estimates the population parameter. Sampling bias voids the bargain. The error introduced is a bias — a fixed offset between expected estimate and true value — which is fundamentally different from random sampling error, which shrinks as $n$ grows. No amount of data cures a biased process; only a better process does.

Formal Statement

Population, Sample, Parameter, Statistic

Population: the complete set of $N$ units we want to know about.
Sample: the subset of $n$ units we actually observe.
Parameter ( $θ$ ): a fixed numerical fact about the population (e.g. true mean $μ$ or proportion $p$ ).
Statistic ( $\hat{θ}$ ): a number computed from the sample to estimate $θ$ (e.g. sample mean $\overset{x}{ˉ}$ or proportion $\overset{p}{^}$ ).
Sampling bias: a feature of the selection mechanism causing $E [\hat{θ}] \neq = θ$ for reasons of who got selected.

The bias of an estimator is its expected deviation from truth:

Bias (\hat{θ}) = E [\hat{θ}] - θ

⟳take the expectation of the estimator over all possible samples the mechanism can produce, then subtract the true parameter.

The total error of an estimator decomposes into bias and variance via the mean squared error:

MSE (\hat{θ}) = systematic (Bias (\hat{θ}))^{2} + random Var (\hat{θ})

⟳expand

E [(\hat{θ} - θ)^{2}]

, add and subtract

E [\hat{θ}]

inside the square, and use that the cross term vanishes because

E [\hat{θ} - E [\hat{θ}]] = 0

For a simple random sample of proportion, the random part shrinks with $n$ :

Var (\overset{p}{^}) = \frac{p ( 1 - p )}{n}

⟳

\overset{p}{^}

is the mean of

n

independent Bernoulli(

p

) indicators; divide the single-trial variance

p (1 - p)

n

But sampling bias gives a constant offset that does not depend on $n$ . If a fraction of the population is selectively included, the limit of the estimate is the parameter of the reachable sub-population, not the whole:

n \to \infty lim \overset{p}{^} = p_{reachable} \neq = p_{population}

⟳under biased selection the law of large numbers still applies — but it converges to the mean of the distribution you are actually sampling from, which is the biased sub-population, not the target.

Bias Dominates at Large n

If a sampling mechanism has a fixed bias $b = E [\hat{θ}] - θ \neq = 0$ , then as $n \to \infty$ the MSE approaches $b^{2} > 0$ . Therefore the estimator is inconsistent: it does not converge to $θ$ no matter how much data is collected. Random error can be beaten by size; bias cannot.

Step 1. From [2],

MSE (\hat{θ}) = b^{2} + Var (\hat{θ})

. Step 2. For most sensible estimators,

Var (\hat{θ}) \to 0

n \to \infty

(e.g. [3] shows variance

\propto 1/ n

). Step 3. Taking the limit:

n \to \infty lim MSE (\hat{θ}) = b^{2} + 0 = b^{2}

. Step 4. Since

b \neq = 0

, we have

b^{2} > 0

, so the MSE never reaches zero. The estimator converges to

θ + b \neq = θ

■

George Gallup | 1901–1984

Though a market researcher rather than a physicist, Gallup is the pivotal figure. In 1936 he publicly predicted that the Literary Digest's 2.4-million-response poll would wrongly call the U.S. election — and that he could beat it with a far smaller, deliberately representative sample. He was right twice over: he forecast both the actual result AND the Digest's error. Gallup demonstrated empirically what equation [4] states formally: representativeness, not size, determines accuracy.

Building Intuition

Imagine you want the average height of everyone in a city, but you only measure people exiting a basketball arena. Measure 50 people or 50,000 — your estimate stays tall, because the door you chose filters for tall people. The filter is the bias. Random sampling error is like the slight wobble in your tape measure: it averages out with more measurements. Sampling bias is like measuring at the wrong door: more measurements just make you more confident in the wrong number. The cure is never "measure more people at the arena" — it is "choose a door that lets everyone through equally."

Random error shrinks with n; bias does not — the dartboard view

Worked Examples

The Basketball Door

Given: A city's true mean adult height is

μ = 170

cm. You sample people leaving a basketball arena, whose mean height is

185

cm. You measure

n = 10, 000

people; the sampling SD of

\overset{x}{ˉ}

7/ 10000 = 0.07

cm. Find: The bias and whether more data helps. Steps:

Expected estimate $E [\overset{x}{ˉ}] = 185$ cm (the reachable sub-population mean).
Bias $= E [\overset{x}{ˉ}] - μ = 185 - 170 = 15$ cm (from [1]).
Random error $\approx 0.07$ cm — negligible.
From [2]: $MSE = (15)^{2} + (0.07)^{2} = 225 + 0.0049 \approx 225$ cm².
Doubling $n$ changes only the $0.0049$ , never the $225$ .

Answer:

Bias = + 15 cm; MSE \approx 225 cm^{2}, dominated entirely by bias.

More data does not help.

Nonresponse in a Survey

Given: A town has true support for a policy

p = 0.50

. Supporters answer phone surveys with probability

0.8

; opponents with probability

0.4

. The pollster reaches

n = 4000

people. Find: The expected sample proportion and the bias. Steps:

Of supporters (50% of town), fraction responding $\propto 0.5 \times 0.8 = 0.40$ .
Of opponents (50%), fraction responding $\propto 0.5 \times 0.4 = 0.20$ .
Among responders, supporter share $= \frac{0.40}{0.40 + 0.20} = \frac{0.40}{0.60} = 0.667$ .
So $E [\overset{p}{^}] = 0.667$ , while $p = 0.50$ .
Bias $= 0.667 - 0.500 = + 0.167$ (from [1]). The 95% interval is $\overset{p}{^} \pm 1.96 0.667 (0.333) /4000 \approx \overset{p}{^} \pm 0.0146$ — tight, and centered on the wrong value.

Answer:

E [\overset{p}{^}] \approx 0.667, bias \approx + 16.7 percentage points.

The narrow margin of error gives false confidence.

Common Misconceptions

"A bigger sample is always better." False when the process is biased. A larger biased sample has the same bias but smaller variance, which produces a tighter confidence interval around the wrong answer — making the result more misleading, not less. The reported margin of error only accounts for random sampling error (equation [3]); it is silent about bias entirely. "Random sampling error and sampling bias are the same thing." No. Random error is the unavoidable luck-of-the-draw scatter that averages to zero over many samples and shrinks with

n

. Sampling bias is a directional error baked into the selection method; its average is nonzero and constant in

n

The 1936 Literary Digest Poll

The Literary Digest mailed surveys to 10 million people and received 2.4 million replies — an enormous sample — predicting Alf Landon would beat Franklin Roosevelt 57% to 43%. Roosevelt won 61% to 37%, one of the largest landslides in U.S. history. The cause was sampling bias: the mailing list came from telephone directories, automobile registrations, and magazine subscribers — disproportionately wealthy during the Great Depression, and disproportionately Republican. Non-response bias compounded it (Landon supporters were more motivated to reply). Two and a half million responses could not save a biased frame. This single case is the textbook proof that size ≠ accuracy.

Connections

Sampling bias is one face of a broader family. It contrasts with Random Sampling Error, which is the unbiased scatter that statistical theory handles. It is a precondition violation for the Central Limit Theorem and Confidence Interval machinery, both of which assume representative draws. It overlaps with Selection Bias (a superset including how subjects enter studies) and is distinct from Measurement Bias (errors in recording values rather than selecting units). In machine learning the same phenomenon appears as Dataset Shift and training/deployment mismatch. The corrective methods connect to Stratified Sampling and Weighting and Post-Stratification.

Real-World Applications

Election polling — pollsters use Stratified Sampling and demographic weighting to counter coverage and nonresponse bias; the 2016 U.S. polls under-weighted non-college voters, a modern recurrence.
Clinical trials — volunteer bias (healthier, more motivated patients enroll) and attrition bias (sicker patients drop out) distort treatment estimates; randomization and intention-to-treat analysis combat this.
Epidemiology — Berkson's bias in hospital-based studies links diseases spuriously because hospitalization itself is the selection filter.
Online reviews and ratings — only strongly satisfied or strongly dissatisfied customers tend to post, producing the bimodal "J-shaped" rating distribution.
Machine learning / facial recognition — training datasets over-representing certain demographics yield systems that perform worse on under-sampled groups.
Web and app analytics — survivorship bias: analyzing only active users ignores those who already left, overstating satisfaction.
Wildlife population estimates — capture–recapture methods explicitly model unequal catchability to correct for animals harder to trap.

Historical Development

1662 — John Graunt analyzes London's Bills of Mortality; early awareness that recorded data may not reflect the true population.
1895 — Anders Kiaer (Norway) proposes the "representative method," arguing samples should mirror the population — an early articulation against biased selection.
1934 — Jerzy Neyman formalizes the theory of representative sampling and stratification, giving probability sampling a rigorous foundation.
1936 — The Literary Digest poll fails spectacularly; George Gallup's smaller quota sample succeeds, dramatizing bias-over-size.
1946 — Joseph Berkson identifies the hospital-admission selection bias now bearing his name.
1948 — Gallup, Roper, and Crossley all wrongly predict Dewey over Truman; quota sampling's hidden biases force the field toward probability sampling.
1949 — Hansen and Hurwitz develop survey-sampling theory with explicit nonresponse handling at the U.S. Census Bureau.
1976 — Abraham Wald's WWII work on aircraft armor (survivorship bias) is popularized; planes returning with bullet holes mis-told where to add armor.
2016–2020 — Polling misses in U.S./UK elections renew focus on nonresponse and coverage bias in the era of cellphones and declining response rates.

Subtleties & Limits

The deepest practical difficulty is that bias is invisible from inside the data. Variance can be estimated from the sample itself; bias generally cannot, because the units you never observed leave no trace. Diagnosing sampling bias requires external information — a known census total, a benchmark, or a model of the selection mechanism — and such information is often unavailable or itself biased. Modern survey research wrestles with falling response rates (sometimes under 5%), where the assumption that nonresponse is "ignorable" becomes increasingly untenable. Methods like multilevel regression and post-stratification (MRP) and inverse-probability weighting can correct known biases, but no method corrects an unknown selection mechanism. The open frontier — sharpened by big "found" data and web scraping — is whether massive non-probability samples can ever be reliably de-biased, or whether, as the Meng (2018) "data quality" identity suggests, even tiny correlations between selection and outcome can swamp the benefit of enormous $n$ .