Populations, Samples, and Statistics

Suppose a factory produces millions of light bulbs. You want to know their average lifetime. You cannot test every bulb, since testing destroys it. So you test a handful and hope it tells you something true about the millions. This is the founding problem of statistical inference: reasoning from a sample back to a population. You may use only what randomness and mathematics allow you to justify. Nearly every method downstream — confidence intervals, hypothesis tests, regression, machine learning — is built on the precise machinery introduced here. The key intellectual move is to treat the data itself as random. Before you draw the sample, the observations you will get are unknown. So we model them as random variables. Once you commit to a probabilistic model for how the sample relates to the population, things become computable. You can compute exactly how a numerical summary of the sample — a statistic — behaves. You can also compute how close it is likely to be to the population quantity of interest. This turns "does the sample tell us the truth?" into a question with a computable, quantitative answer. This article makes rigorous three intertwined ideas. First, what a statistical population is. Second, what it means for a sample to be a random sample (in particular, an iid sample). Third, what a statistic is as a function of the data. We prove the two basic facts that make sample averages trustworthy tools. The sample mean is an unbiased estimator of the population mean. Its variance shrinks like 1/n1/n as the sample size grows. These two facts are the seed from which the entire theory of sampling distributions and estimation grows. By the end, you will have a precise vocabulary — population, parameter, random sample, statistic. You will also have one clean, fully proved theorem. It already explains why "more data is better" in a quantitative sense. Everything later in statistical inference (unbiasedness, consistency, the method of moments, confidence intervals) is a refinement of the ideas fixed here.

Key takeaways
  • A population is described by a probability distribution with unknown features called parameters.
  • A random sample X1,…,XnX_1,\dots,X_n is a collection of iid random variables drawn from that distribution.
  • A statistic is any function T(X1,…,Xn)T(X_1,\dots,X_n) of the sample that does not depend on unknown parameters.
  • The sample mean Xˉ\bar X and sample variance S2S^2 are the two most important statistics.
  • E[Xˉ]=μE[\bar X] = \mu always (unbiasedness), and Var⁡(Xˉ)=σ2/n\operatorname{Var}(\bar X) = \sigma^2/n shrinks as nn grows.
  • Independence and identical distribution are both essential — dropping either breaks the variance formula.
  • Key idea: treating unobserved data as random variables lets you compute, not just guess, how trustworthy a sample summary is.

Where this lives today

The population/sample/statistic vocabulary is the operating system beneath essentially all data-based science. In clinical trials, patients enrolled are a sample from the (hypothetical) population of all patients with a condition, and treatment-effect estimates are statistics computed from that sample. In survey sampling and polling, pollsters explicitly model respondents as a random sample from a voter population, using finite-population corrections exactly like the one in the counterexample above when the sampling fraction is non-negligible. In machine learning, a training set is treated as an iid sample from an unknown data-generating distribution, and quantities like the average training loss are statistics whose expectation and variance (over resampling of the training set) govern generalization theory. In quality control and manufacturing, control charts plot sample means Xˉ\bar X over time precisely because their known mean and shrinking variance make deviations from μ\mu statistically meaningful. In physics and engineering measurement, repeated instrument readings are modeled as iid samples around a true value, with Xˉ\bar X used to reduce measurement noise by a factor of n\sqrt{n} in standard deviation — a direct consequence of Var⁡(Xˉ)=σ2/n\operatorname{Var}(\bar X) = \sigma^2/n. In cryptography and randomized algorithms, statistical tests for randomness draw on exactly this apparatus to check whether observed bit-sequences behave like iid samples from a uniform population.

Theoremrestated informally for the quiz below

For an iid sample, E[Xˉ]=μE[\bar X]=\mu regardless of nn, while Var⁡(Xˉ)=σ2/n\operatorname{Var}(\bar X)=\sigma^2/n decreases as nn increases.

:::quiz[Why does the proof of Var⁡(Xˉ)=σ2/n\operatorname{Var}(\bar X)=\sigma^2/n require independence but the proof of E[Xˉ]=μE[\bar X]=\mu does not?]

  • Expectation is only linear for independent variables
  • Variance of a sum needs covariance terms to vanish, which independence guarantees; expectation of a sum is always additive regardless of dependence ✓
  • Independence is never actually used in either proof
  • Identical distribution alone implies independence

Ask the tutor: How does the sampling distribution of Xˉ\bar X change if the population itself is Normal versus non-Normal?

DefinitionPopulation, sample, statistic

A statistical population is a probability distribution FF (equivalently, a random variable XX with that distribution) representing the entire collection of values under study. Numerical features of FF — such as its mean μ=E[X]\mu = E[X] or variance σ2=Var⁡(X)\sigma^2 = \operatorname{Var}(X) — that are fixed but typically unknown are called parameters. A random sample of size nn from FF is a finite sequence of random variables

X1,X2,…,XnX_1, X_2, \dots, X_n

defined on a common probability space. The sample is called independent and identically distributed (iid) if: 1. (Independence) X1,…,XnX_1,\dots,X_n are mutually independent random variables. 2. (Identical distribution) Each XiX_i has the same distribution FF as the population. Unless stated otherwise, "random sample" means an iid random sample. A statistic is any Borel-measurable function

T=g(X1,…,Xn)T = g(X_1,\dots,X_n)

of the sample that contains no unknown parameters — that is, TT is itself an observable random variable, computable once the data x1,…,xnx_1,\dots,x_n are observed. Its realized numerical value t=g(x1,…,xn)t = g(x_1,\dots,x_n), computed from the actual observed data, is called the observed value of the statistic. The two central statistics are the sample mean

Xˉ=1n∑i=1nXi,\bar X = \frac{1}{n}\sum_{i=1}^n X_i,

and the sample variance

S2=1n−1∑i=1n(Xi−Xˉ)2.S^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i - \bar X)^2 .

Intuition

Think of the population as an enormous urn of numbered balls whose overall composition — the average value written on the balls, how spread out those values are — is unknown to you. Sampling means reaching in, drawing nn balls without seeing the rest, and using only what is on those nn balls to guess at the urn's composition. The word "random" is doing two jobs at once, and both matter. First, before you draw, you do not know which values X1,…,XnX_1,\dots,X_n you will get, so each XiX_i is modeled as a random variable, not a fixed number. Second, the draws should not influence each other and should each reflect the same underlying urn — this is exactly the independence and identical-distribution conditions. If the balls are drawn with replacement (or the urn is infinite, or you sample a tiny fraction of a huge finite population), the iid model is essentially exact; if you sample without replacement from a small finite population, the draws are dependent, though still identically distributed marginally — a subtlety addressed in the counterexample below. A statistic is then a rule for turning the nn random numbers you will get into a single random number — a data-driven guess. Because Xˉ\bar X is built from random variables, Xˉ\bar X is itself random: draw the sample again and you likely get a different Xˉ\bar X. The genius of the framework is that even though we cannot know the realized value of Xˉ\bar X in advance, we can compute its entire probability distribution's key features — mean and variance — from the population's parameters, before ever collecting data.

Figure
Population distribution vs. one random sample of size n = 5
Population distribution F (unknown μ, σ²) μ One random sample: X₁,…,X₅ (iid draws) X₁ X₂ X₃ X₄ X₅ X̄ (statistic)
TheoremMean and variance of the sample mean

Let X1,…,XnX_1,\dots,X_n be a random sample (iid) from a population with finite mean E[Xi]=μE[X_i] = \mu and finite variance Var⁡(Xi)=σ2<∞\operatorname{Var}(X_i) = \sigma^2 < \infty. Let

Xˉ=1n∑i=1nXi.\bar X = \frac{1}{n}\sum_{i=1}^n X_i .

Then: 1. E[Xˉ]=μE[\bar X] = \mu. 2. Var⁡(Xˉ)=σ2n\operatorname{Var}(\bar X) = \dfrac{\sigma^2}{n}. Consequently Xˉ→μ\bar X \to \mu in the sense that Var⁡(Xˉ)→0\operatorname{Var}(\bar X) \to 0 as n→∞n \to \infty; larger samples produce sample means with less variability around the true population mean.

Proof

We use only linearity of expectation and the definition of variance, together with the independence and identical-distribution hypotheses. This is a direct computational proof (no induction beyond the finite additivity already built into linearity of expectation, and no appeal to any limit theorem).

Step 1 (Linearity of expectation applies unconditionally). Expectation is linear for any finite collection of random variables with finite means, regardless of dependence structure:

E[∑i=1nXi]=∑i=1nE[Xi].E\left[\sum_{i=1}^n X_i\right] = \sum_{i=1}^n E[X_i].

This is a basic property of expectation (additivity), valid because each XiX_i has finite mean by hypothesis, so the sum ∑Xi\sum X_i has finite mean and the interchange of EE and finite sum is justified.

Step 2 (Compute E[Xˉ]E[\bar X]). Since each XiX_i has the same distribution as the population (identical distribution hypothesis), E[Xi]=μE[X_i] = \mu for every i=1,…,ni = 1,\dots,n. Hence

E[Xˉ]=E[1n∑i=1nXi]=1n∑i=1nE[Xi]=1n∑i=1nμ=1n⋅nμ=μ.E[\bar X] = E\left[\frac{1}{n}\sum_{i=1}^n X_i\right] = \frac{1}{n}\sum_{i=1}^n E[X_i] = \frac{1}{n}\sum_{i=1}^n \mu = \frac{1}{n}\cdot n\mu = \mu.

This proves part 1. Note that identical distribution alone (no independence needed) suffices for this step.

Step 3 (Variance of a sum requires independence). For any random variables Y1,…,YnY_1,\dots,Y_n with finite variances, the general bilinearity of covariance gives

Var⁡(∑i=1nYi)=∑i=1nVar⁡(Yi)+∑i≠jCov⁡(Yi,Yj).\operatorname{Var}\left(\sum_{i=1}^n Y_i\right) = \sum_{i=1}^n \operatorname{Var}(Y_i) + \sum_{i \neq j} \operatorname{Cov}(Y_i, Y_j).

This is an algebraic identity: expand Var⁡(∑Yi)=E[(∑i(Yi−E[Yi]))2]\operatorname{Var}(\sum Y_i) = E\left[\left(\sum_i (Y_i - E[Y_i])\right)^2\right] and use the bilinearity of the product of sums, which yields exactly the diagonal terms Var⁡(Yi)\operatorname{Var}(Y_i) and off-diagonal terms Cov⁡(Yi,Yj)\operatorname{Cov}(Y_i,Y_j) for i≠ji\neq j.

Step 4 (Independence kills the cross terms). By hypothesis X1,…,XnX_1,\dots,X_n are mutually independent. Independent random variables with finite variance are uncorrelated: if Xi⊥XjX_i \perp X_j (i≠ji \ne j), then E[XiXj]=E[Xi]E[Xj]E[X_iX_j] = E[X_i]E[X_j] (a standard consequence of independence for random variables with finite means, since independence implies E[XiXj]=E[Xi]E[Xj]E[X_iX_j] = E[X_i]E[X_j] whenever both expectations exist), and therefore

Cov⁡(Xi,Xj)=E[XiXj]−E[Xi]E[Xj]=0(i≠j).\operatorname{Cov}(X_i,X_j) = E[X_iX_j] - E[X_i]E[X_j] = 0 \quad (i \ne j).

Substituting into Step 3 with Yi=XiY_i = X_i, all cross terms vanish:

Var⁡(∑i=1nXi)=∑i=1nVar⁡(Xi).\operatorname{Var}\left(\sum_{i=1}^n X_i\right) = \sum_{i=1}^n \operatorname{Var}(X_i).

Step 5 (Use identical distribution). Since each XiX_i has the same distribution as the population, Var⁡(Xi)=σ2\operatorname{Var}(X_i) = \sigma^2 for every ii. Hence

Var⁡(∑i=1nXi)=∑i=1nσ2=nσ2.\operatorname{Var}\left(\sum_{i=1}^n X_i\right) = \sum_{i=1}^n \sigma^2 = n\sigma^2.

Step 6 (Scale by 1/n1/n). For any random variable YY and constant cc, Var⁡(cY)=c2Var⁡(Y)\operatorname{Var}(cY) = c^2 \operatorname{Var}(Y) — this follows directly from the definition Var⁡(cY)=E[(cY−E[cY])2]=c2E[(Y−E[Y])2]\operatorname{Var}(cY) = E[(cY - E[cY])^2] = c^2 E[(Y-E[Y])^2]. Applying this with c=1/nc = 1/n and Y=∑i=1nXiY = \sum_{i=1}^n X_i:

Var⁡(Xˉ)=Var⁡(1n∑i=1nXi)=1n2Var⁡(∑i=1nXi)=1n2⋅nσ2=σ2n.\operatorname{Var}(\bar X) = \operatorname{Var}\left(\frac{1}{n}\sum_{i=1}^n X_i\right) = \frac{1}{n^2}\operatorname{Var}\left(\sum_{i=1}^n X_i\right) = \frac{1}{n^2}\cdot n\sigma^2 = \frac{\sigma^2}{n}.

This proves part 2.

Step 7 (Consequence). Since σ2\sigma^2 is a fixed finite constant (independent of nn) and Var⁡(Xˉ)=σ2/n→0\operatorname{Var}(\bar X) = \sigma^2/n \to 0 as n→∞n \to \infty, the sample mean's variability around μ\mu shrinks to zero, while its expectation remains exactly μ\mu for every nn. ■\blacksquare

∎

Worked examples

Example 1 — Asymmetric discrete population, small sample. A quality-control population of manufactured joints has defect counts per joint following a distribution on {0,1,2,3}\{0,1,2,3\}:

P(X=0)=0.5,P(X=1)=0.3,P(X=2)=0.15,P(X=3)=0.05.P(X=0) = 0.5,\quad P(X=1)=0.3,\quad P(X=2) = 0.15,\quad P(X=3) = 0.05.

This is a genuinely asymmetric distribution. First compute the population mean and variance.

μ=E[X]=0(0.5)+1(0.3)+2(0.15)+3(0.05)=0+0.3+0.3+0.15=0.75.\mu = E[X] = 0(0.5)+1(0.3)+2(0.15)+3(0.05) = 0 + 0.3 + 0.3 + 0.15 = 0.75.
E[X2]=02(0.5)+12(0.3)+22(0.15)+32(0.05)=0+0.3+0.6+0.45=1.35.E[X^2] = 0^2(0.5)+1^2(0.3)+2^2(0.15)+3^2(0.05) = 0+0.3+0.6+0.45 = 1.35.
σ2=E[X2]−μ2=1.35−0.5625=0.7875.\sigma^2 = E[X^2] - \mu^2 = 1.35 - 0.5625 = 0.7875.

Now take a random sample of size n=4n=4, X1,X2,X3,X4X_1,X_2,X_3,X_4 iid from this distribution — think of inspecting 4 joints from a very large production run, so independence and identical distribution are realistic. By the theorem,

E[Xˉ]=μ=0.75,Var⁡(Xˉ)=σ2n=0.78754=0.196875.E[\bar X] = \mu = 0.75, \qquad \operatorname{Var}(\bar X) = \frac{\sigma^2}{n} = \frac{0.7875}{4} = 0.196875.

Suppose the actual observed defect counts were x1=1,x2=0,x3=2,x4=0x_1=1, x_2=0, x_3=2, x_4=0. The observed value of the statistic Xˉ\bar X is

xˉ=1+0+2+04=34=0.75.\bar x = \frac{1+0+2+0}{4} = \frac{3}{4} = 0.75.

Notice this single observed value happens to coincide numerically with μ\mu; that is a coincidence of this particular draw, not a guarantee — what the theorem guarantees is that averaged over all possible samples of size 4, the mean of Xˉ\bar X equals 0.750.75 exactly, and its spread (variance 0.1968750.196875, standard deviation ≈0.4437\approx 0.4437) is a known, computable quantity, not a guess. To see the shrinking-variance effect quantitatively, compare n=4n=4 to n=100n=100:

Var⁡(Xˉ)∣n=100=0.7875100=0.007875,\operatorname{Var}(\bar X)\Big|_{n=100} = \frac{0.7875}{100} = 0.007875,

a 25-fold reduction in variance (matching the 25-fold increase in nn), so the standard deviation of Xˉ\bar X shrinks by a factor of 25=5\sqrt{25}=5, from about 0.44370.4437 to about 0.08870.0887.

Example 2 — Continuous population, computing a statistic and checking unbiasedness numerically. Let the population be Exponential(λ)\mathrm{Exponential}(\lambda) with rate λ=0.4\lambda = 0.4 (mean lifetime 1/λ=2.51/\lambda = 2.5 years for a component), so μ=2.5\mu = 2.5 and σ2=1/λ2=6.25\sigma^2 = 1/\lambda^2 = 6.25. Take a random sample of size n=5n=5 representing 5 independently tested components from a very large batch (so iid is justified). Suppose the observed lifetimes (in years) are:

x1=1.8, x2=4.1, x3=0.6, x4=3.2, x5=2.4.x_1 = 1.8,\ x_2 = 4.1,\ x_3 = 0.6,\ x_4 = 3.2,\ x_5 = 2.4.

The observed sample mean is

xˉ=1.8+4.1+0.6+3.2+2.45=12.15=2.42.\bar x = \frac{1.8+4.1+0.6+3.2+2.4}{5} = \frac{12.1}{5} = 2.42.

The observed sample variance uses S2=1n−1∑(xi−xˉ)2S^2 = \frac{1}{n-1}\sum (x_i - \bar x)^2:

(1.8−2.42)2=0.3844,(4.1−2.42)2=2.8224,(0.6−2.42)2=3.3124,(1.8-2.42)^2 = 0.3844,\quad (4.1-2.42)^2 = 2.8224,\quad (0.6-2.42)^2 = 3.3124,
(3.2−2.42)2=0.6084,(2.4−2.42)2=0.0004.(3.2-2.42)^2 = 0.6084,\quad (2.4-2.42)^2 = 0.0004.

Sum =0.3844+2.8224+3.3124+0.6084+0.0004=7.128= 0.3844+2.8224+3.3124+0.6084+0.0004 = 7.128. Hence

s2=7.1284=1.782.s^2 = \frac{7.128}{4} = 1.782.

Compare with the theorem's predictions before any data were drawn: E[Xˉ]=μ=2.5E[\bar X] = \mu = 2.5 and Var⁡(Xˉ)=σ2/n=6.25/5=1.25\operatorname{Var}(\bar X) = \sigma^2/n = 6.25/5 = 1.25, so the standard deviation of Xˉ\bar X is 1.25≈1.118\sqrt{1.25} \approx 1.118. Our single observed value xˉ=2.42\bar x = 2.42 is well within one standard deviation of μ=2.5\mu = 2.5 (difference of only 0.080.08), consistent with — though of course not a proof of — the predicted behavior. This illustrates the general logic of inference: the theorem tells us in advance the distributional behavior of Xˉ\bar X (centered at μ\mu, with a computable spread), and any one observed sample is a single random draw from that distribution.

CounterexampleWhy identical distribution is needed: sampling without replacement from a finite population

Drop the identical-distribution/independence structure by sampling without replacement from a small finite population, and see the variance formula fail. Consider a tiny population of N=4N = 4 items with values {2,4,6,8}\{2, 4, 6, 8\}. Population mean:

μ=2+4+6+84=5,σ2=(2−5)2+(4−5)2+(6−5)2+(8−5)24=9+1+1+94=5.\mu = \frac{2+4+6+8}{4} = 5, \qquad \sigma^2 = \frac{(2-5)^2+(4-5)^2+(6-5)^2+(8-5)^2}{4} = \frac{9+1+1+9}{4} = 5.

(Here σ2\sigma^2 is the population variance, dividing by NN, as is standard for a finite population.) Now draw n=2n = 2 items without replacement, uniformly at random, and let X1,X2X_1, X_2 denote the first and second draws. Each XiX_i individually is uniform over {2,4,6,8}\{2,4,6,8\} (identical distribution does still hold marginally: P(X1=v)=P(X2=v)=1/4P(X_1 = v) = P(X_2=v) = 1/4 for each vv), but X1X_1 and X2X_2 are not independent, since after seeing X1X_1, only 3 values remain equally likely for X2X_2. Compute Var⁡(Xˉ)\operatorname{Var}(\bar X) directly by enumerating all 4×3=124 \times 3 = 12 equally likely ordered pairs (X1,X2)(X_1,X_2) with X1≠X2X_1 \ne X_2:

(2,4),(2,6),(2,8),(4,2),(4,6),(4,8),(6,2),(6,4),(6,8),(8,2),(8,4),(8,6).(2,4),(2,6),(2,8),(4,2),(4,6),(4,8),(6,2),(6,4),(6,8),(8,2),(8,4),(8,6).

The corresponding sample means xˉ=(x1+x2)/2\bar x = (x_1+x_2)/2 are: 3,4,5,3,5,6,4,5,7,5,6,73,4,5,3,5,6,4,5,7,5,6,7 (each with probability 1/121/12). Direct computation gives E[Xˉ]=5=μE[\bar X] = 5 = \mu (unbiasedness of Xˉ\bar X survives dependence — it only used linearity of expectation and identical marginal distributions, not independence). But now compute the variance:

E[Xˉ2]=32+42+52+32+52+62+42+52+72+52+62+7212=9+16+25+9+25+36+16+25+49+25+36+4912=32012=803.E[\bar X^2] = \frac{3^2+4^2+5^2+3^2+5^2+6^2+4^2+5^2+7^2+5^2+6^2+7^2}{12} = \frac{9+16+25+9+25+36+16+25+49+25+36+49}{12} = \frac{320}{12} = \frac{80}{3}.
Var⁡(Xˉ)=803−52=803−25=80−753=53≈1.667.\operatorname{Var}(\bar X) = \frac{80}{3} - 5^2 = \frac{80}{3} - 25 = \frac{80-75}{3} = \frac{5}{3} \approx 1.667.

If the iid formula Var⁡(Xˉ)=σ2/n\operatorname{Var}(\bar X) = \sigma^2/n were applied blindly here, it would predict

σ2n=52=2.5,\frac{\sigma^2}{n} = \frac{5}{2} = 2.5,

which is wrong — it overstates the true variance (1.667≠2.51.667 \ne 2.5). The correct formula for sampling without replacement includes a finite-population correction factor N−nN−1\frac{N-n}{N-1}:

Var⁡(Xˉ)=σ2n⋅N−nN−1=52⋅4−24−1=52⋅23=53,\operatorname{Var}(\bar X) = \frac{\sigma^2}{n}\cdot\frac{N-n}{N-1} = \frac{5}{2}\cdot\frac{4-2}{4-1} = \frac{5}{2}\cdot\frac{2}{3} = \frac{5}{3},

which matches the direct computation exactly. This shows precisely where independence was used in the theorem's proof (Step 4): without it, Step 3's cross-covariance terms do not vanish, and the simple formula σ2/n\sigma^2/n is invalid.

Scope, edge cases & common traps

SituationWhat holdsWhat can fail
Sampling with replacement, or from an (effectively) infinite populationFull iid theorem applies exactly—
Sampling without replacement from a finite population, nn small relative to NNE[Xˉ]=μE[\bar X]=\mu still holdsVar⁡(Xˉ)=σ2/n\operatorname{Var}(\bar X)=\sigma^2/n fails; needs finite-population correction N−nN−1\frac{N-n}{N-1}
σ2=∞\sigma^2 = \infty (e.g., heavy-tailed population)Xˉ\bar X still unbiased if μ\mu finiteVar⁡(Xˉ)\operatorname{Var}(\bar X) undefined; "shrinks like 1/n1/n" claim is meaningless
Statistic depends on unknown parameters (e.g. (Xi−μ)(X_i-\mu) with unknown μ\mu)—Not a statistic at all, by definition
Sample size n=1n=1Theorem still technically trueVar⁡(Xˉ)=σ2\operatorname{Var}(\bar X)=\sigma^2; no reduction in variability

The mathematician behind it

Origins · 1 minRonald Aylmer Fisher and how this was discovered
RA
Ronald Aylmer Fisher
1890–1962 · Cambridge and Rothamsted Experimental Station, England

Fisher gave statistics its modern mathematical backbone: the precise notions of a statistical population modeled by a parametric distribution, a random sample as a repeatable idealization of an experiment, and a statistic as any function of the data usable for inference. His 1922 paper On the Mathematical Foundations of Theoretical Statistics introduced much of the vocabulary — parameter, statistic, likelihood, sufficiency, consistency — that this very article relies on, and set the agenda for point estimation that the next articles in this chapter carry forward.

History
- 1890: Ronald Fisher born in London.
- 1919: Fisher joins Rothamsted Experimental Station, applying statistics to agricultural field trials, driving the need for rigorous sampling theory.
- 1922: Publishes On the Mathematical Foundations of Theoretical Statistics, formally defining population, sample, statistic, and parameter.
- 1925: Statistical Methods for Research Workers popularizes these ideas for working scientists.
- 1962: Fisher dies in Adelaide, Australia; his framework remains the standard language of inference.
What exactly is the difference between a population and a sample?

A population is the entire distribution of interest — every unit or value you could in principle observe, described mathematically by a probability distribution FF with fixed but often unknown parameters like μ\mu and σ2\sigma^2. A sample is a finite, observable subset drawn from that population, modeled (under the iid assumption) as random variables X1,…,XnX_1,\dots,X_n each distributed as FF and mutually independent. The population is typically infinite or too large to fully examine; the sample is what you actually collect and compute with. Inference is the bridge: using known probabilistic relationships between sample statistics and population parameters (like the theorem in this article) to make justified claims about the population from the sample alone.

Why must a statistic not depend on unknown parameters?

The entire purpose of a statistic is to be computable from data you actually observe. If a function of the sample also involves an unknown parameter — for instance ∑(Xi−μ)2\sum (X_i - \mu)^2 where μ\mu is unknown — then you cannot compute its numerical value once the data are in hand, because μ\mu itself is unknown; that formula still depends on a piece of information nobody has. Excluding unknown parameters from the definition of a statistic keeps the concept useful for its intended job: a statistic is meant to estimate or test something about unknown parameters, so it cannot itself secretly assume knowledge of them. Quantities like ∑(Xi−Xˉ)2\sum (X_i-\bar X)^2, using the sample mean instead, are legitimate statistics.

Does Var⁡(Xˉ)=σ2/n\operatorname{Var}(\bar X)=\sigma^2/n mean Xˉ\bar X converges to μ\mu?

It is strong evidence toward convergence but is not itself a convergence proof. The formula shows the variance of Xˉ\bar X shrinks to zero as n→∞n \to \infty, meaning Xˉ\bar X becomes less and less spread out around μ\mu in a mean-square sense. Combined with Chebyshev's inequality, this does imply convergence in probability of Xˉ\bar X to μ\mu — this chain of reasoning is exactly how one proves the weak Law of Large Numbers under a finite-variance assumption. But that extra logical step (applying Chebyshev, taking a limit of tail probabilities) is separate machinery, built on top of, not identical to, the mean/variance computation given here.

Why is the sample variance divided by n−1n-1 instead of nn?

Dividing by n−1n-1 rather than nn makes S2S^2 an unbiased estimator of σ2\sigma^2, meaning E[S2]=σ2E[S^2] = \sigma^2 exactly for every sample size n≥2n \ge 2. Intuitively, the sample mean Xˉ\bar X is itself estimated from the same data used to measure spread, and Xˉ\bar X is always at least as close to the data (in squared distance) as the true unknown μ\mu would be, which systematically shrinks ∑(Xi−Xˉ)2\sum(X_i-\bar X)^2 relative to ∑(Xi−μ)2\sum (X_i - \mu)^2; dividing by n−1n-1 instead of nn exactly corrects this shrinkage on average. A full proof of E[S2]=σ2E[S^2]=\sigma^2 belongs to the article on Point Estimation and Estimators, since it requires analyzing Xˉ\bar X and S2S^2 jointly.

Is the iid assumption ever exactly true in real data?

Rarely exactly, but often an excellent approximation. Sampling with replacement from any population makes iid exact by construction. Sampling without replacement from a very large population relative to the sample size (say, a national survey with nn in the thousands from a population of hundreds of millions) makes the dependence between draws negligible, since the finite-population correction factor N−nN−1\frac{N-n}{N-1} is extremely close to 11. The iid model breaks down more seriously when the population is small relative to nn, or when observations are naturally sequential and correlated, such as daily stock returns or repeated measurements on the same subject over time — situations requiring the more general dependent-sampling models mentioned in the counterexample.

What is the practical difference between a parameter and a statistic?

A parameter is a fixed (though usually unknown) number describing the population, such as μ\mu or σ2\sigma^2; it does not change no matter how many samples you draw, since it belongs to the population itself, not to any particular sample. A statistic is a random variable computed from a sample, such as Xˉ\bar X or S2S^2; its value changes from sample to sample, and it has its own probability distribution (called its sampling distribution, the subject of the next article in this chapter). The entire logic of estimation is to use a statistic, whose distribution you can compute or approximate, as a data-based stand-in for a parameter you cannot directly observe.

Why does the theorem require finite variance σ2<∞\sigma^2 < \infty?

The proof's Step 3–6 chain manipulates Var⁡(∑Xi)\operatorname{Var}(\sum X_i) and Var⁡(Xˉ)\operatorname{Var}(\bar X) as finite numbers; if σ2=∞\sigma^2 = \infty (as happens for heavy-tailed distributions like the Cauchy distribution, which additionally has no finite mean at all), these quantities are not well-defined finite numbers, and the algebra of Step 6 (Var⁡(cY)=c2Var⁡(Y)\operatorname{Var}(cY)=c^2\operatorname{Var}(Y)) becomes vacuous or undefined. The unbiasedness result E[Xˉ]=μE[\bar X]=\mu only needs E[Xi]=μE[X_i]=\mu to be finite (finite mean), a strictly weaker requirement than finite variance, which is why Step 2 of the proof survives even in some heavy-tailed cases where Steps 3–6 do not apply.

Before this

Where this leads