Point Estimation and Estimators

Suppose a public health lab receives 400 water samples. It wants to know what fraction contain a contaminant above a legal threshold. Suppose a factory produces light bulbs whose lifetime follows an exponential distribution with unknown mean. An engineer must decide, from a batch of 50 bulbs, what that mean actually is. In both cases there is an unknown numerical feature of a population — a proportion, a mean, a rate. Only a finite, random sample exists from which to guess it. Point estimation is the branch of statistical inference that turns this guessing into mathematics. It specifies what a "guess" is allowed to be. It also builds a framework for comparing guesses against each other. The idea matters because almost every quantitative claim in science rests on a single number. That number, computed from data, is offered as a stand-in for an unknown constant of nature. A drug's true cure rate, a species' true population size, a physical constant, a regression coefficient — these are never observed directly, only estimated. Point estimation asks the foundational question of what makes one estimate better than another. Infinitely many formulas could all plausibly claim to estimate the same parameter. Without criteria, the exercise is empty. The big picture is this: an unknown parameter $\theta$ indexes a family of possible probability distributions. The data are believed to arise from this family. A statistic — a function of the observable data alone, containing no unknown quantities — is proposed as an estimator $\hat{\theta}$ of $\theta$ . Because the data are random, $\hat\theta$ is itself a random variable with its own sampling distribution. The entire theory of good estimation is the study of that sampling distribution. It examines where the distribution is centered, how spread out it is, and how it behaves as the sample grows. This article gives the formal definition of an estimator. It develops the two central desiderata — unbiasedness and small variance, unified via mean squared error. It also proves the key comparison and consistency results. It closes by placing point estimation within the wider inferential machinery it feeds: bias diagnostics, the method of moments, and maximum likelihood. Each of these supplies concrete recipes for building estimators. This article's criteria are designed to evaluate the properties of those estimators.

Key takeaways

A parameter $\theta$ is an unknown numerical feature of a population; an estimator $\hat\theta$ is a statistic (a function of the sample only) proposed to approximate it.
Estimators are random variables: they have sampling distributions, and "good estimator" means "good sampling distribution," not "gets today's data exactly right."
Unbiasedness ( $E[\hat\theta]=\theta$ for every $\theta$ ) is one desirable but not sufficient property; low variance is the other.
Mean squared error $\mathrm{MSE}(\hat\theta) = \mathrm{Var}(\hat\theta) + \mathrm{Bias}(\hat\theta)^2$ unifies both criteria into a single number to minimize.
Consistency demands that $\hat\theta_n \to \theta$ in probability as sample size $n\to\infty$ ; the Weak Law of Large Numbers is the standard engine for proving it.
Key idea: an estimator is judged not by any single value it outputs, but by the entire probability distribution of outputs it would produce over repeated sampling.

Where this lives today

Point estimation is the load-bearing wall underneath essentially all quantitative inference. In clinical trials, the estimated cure rate or hazard ratio from a finite trial is a point estimate whose bias and variance properties directly determine sample-size requirements and regulatory conclusions. In econometrics, ordinary least squares coefficients are point estimators of unknown structural parameters, and the Gauss–Markov theorem is precisely a statement that OLS is the minimum-variance linear unbiased estimator — an MSE-style optimality claim in the exact spirit developed here. In machine learning, model parameters (weights, cluster centers, mixture proportions) are estimated from training data, and the bias–variance decomposition of this article is the same decomposition invoked to explain overfitting versus underfitting. In physics and metrology, repeated measurements of a physical constant (the gravitational constant, a particle mass) are combined into a point estimate whose reported uncertainty is literally an estimated standard deviation of its sampling distribution. In cryptography and coding theory, estimating channel error rates or key-generation biases from observed transmissions uses the same proportion-estimation logic as Example 2. And within statistics itself, this framework is the direct staging ground for the systematic study of estimator bias, for moment-matching constructions of estimators, and for likelihood-based estimation, all of which are judged by exactly the unbiasedness, variance, MSE, and consistency criteria developed here.

:::quiz[If $\hat\theta$ satisfies $E[\hat\theta] = \theta + 3$ for all $\theta$ , what is true?]

$\hat\theta$ is unbiased
$\hat\theta$ is consistent regardless of its variance
$\hat\theta$ has constant bias equal to 3, so it is biased ✓
$\hat\theta$ cannot be used as an estimator at all

Ask the tutor: How do you decide, in practice, whether to accept a small amount of bias in exchange for a reduction in variance?

X_1,\dots,X_n

Intuition

Think of an estimator as a machine, not a single output. You feed it a sample of size $n$ ; it returns a number. If you could rerun the experiment — draw a fresh sample of the same size from the same population — the machine would generally return a different number. The collection of all the numbers it could return, weighted by how likely each is, is the sampling distribution of the estimator. Judging an estimator means judging this whole cloud of hypothetical outputs, not the single output you happened to see. Two features of that cloud matter most. First, where is it centered — does it hover around the true $\theta$ , or is it systematically off to one side (biased)? Second, how tightly clustered is it — does it produce wildly different answers from sample to sample (high variance), or does it settle down reliably? A good estimator is centered on the target and tightly clustered. A biased-but-precise estimator reliably misses; an unbiased-but-wild one is right on average but rarely close on any given try. Mean squared error is the single number that penalizes both kinds of failure at once, and it is the quantity that unifies the entire subject.

Figure

Sampling distributions of two estimators of the same

\theta = 5

\hat\theta_1

unbiased but spread out,

\hat\theta_2

biased but tight

Central results

\hat\theta

\theta \in \Theta

\hat\theta

\mathrm{Bias}_\theta(\hat\theta) = 0

This corollary is why, when comparing two unbiased estimators, it suffices to compare their variances directly — the smaller-variance unbiased estimator has smaller MSE at every $\theta$ , and is preferred. When one or both estimators are biased, MSE is the correct arbiter, since it can reveal that a biased estimator outperforms an unbiased one (illustrated below).

X_1,\dots,X_n

E[\bar X] = \frac1n \sum_{i=1}^n E[X_i] = \frac1n \cdot n\mu = \mu.

X_1,\dots,X_n

E[\bar X_n] = \mu

This is precisely an instance of the Weak Law of Large Numbers: consistency of the sample mean is the weak law of large numbers, restated in the language of estimation.

\to 0

\theta \in \Theta

This corollary is the standard tool: to verify consistency in practice, one typically checks unbiasedness (or asymptotic unbiasedness) together with vanishing variance, rather than working with the definition of convergence in probability directly.

Worked examples

Example 1 — Estimating the range parameter of a Uniform distribution, asymmetric sample. Let $X_1,\dots,X_n$ be i.i.d. $\mathrm{Uniform}(0,\theta)$ with $\theta > 0$ unknown; this models, for instance, the maximum possible response time on a device where $\theta$ is an unknown ceiling. Consider a specific sample of size $n = 5$ :

x_1 = 2.1,\ x_2 = 6.4,\ x_3 = 3.0,\ x_4 = 7.8,\ x_5 = 1.5.

Two natural estimators are proposed: Estimator A: $\hat\theta_A = 2\bar X$ (twice the sample mean, motivated by $E[X_i] = \theta/2$ ). Estimator B: $\hat\theta_B = X_{(n)} = \max(X_1,\dots,X_n)$ (the sample maximum).

Point estimates. With the data above, $\bar x = (2.1+6.4+3.0+7.8+1.5)/5 = 20.8/5 = 4.16$ , so $\hat\theta_A = 8.32$ . The maximum is $x_{(5)} = 7.8$ , so $\hat\theta_B = 7.8$ .

Bias of $\hat\theta_A$ . Since $E[X_i] = \theta/2$ for $\mathrm{Uniform}(0,\theta)$ ,

E[\hat\theta_A] = 2E[\bar X] = 2\cdot \frac{\theta}{2} = \theta,

so $\hat\theta_A$ is unbiased. Its variance: $\mathrm{Var}(X_i) = \theta^2/12$ , so $\mathrm{Var}(\bar X) = \theta^2/(12n)$ , and

\mathrm{Var}(\hat\theta_A) = 4\cdot \frac{\theta^2}{12n} = \frac{\theta^2}{3n}.

For $n=5$ : $\mathrm{Var}(\hat\theta_A) = \theta^2/15$ .

Bias of $\hat\theta_B$ . The density of $X_{(n)} = \max(X_1,\dots,X_n)$ for $\mathrm{Uniform}(0,\theta)$ is found from its CDF $F_{(n)}(y) = (y/\theta)^n$ for $0 \le y \le \theta$ , giving density $f_{(n)}(y) = n y^{n-1}/\theta^n$ . Then

E[X_{(n)}] = \int_0^\theta y \cdot \frac{n y^{n-1}}{\theta^n}\,dy = \frac{n}{\theta^n}\int_0^\theta y^n\,dy = \frac{n}{\theta^n}\cdot\frac{\theta^{n+1}}{n+1} = \frac{n}{n+1}\theta.

So $\mathrm{Bias}(\hat\theta_B) = \frac{n}{n+1}\theta - \theta = -\frac{\theta}{n+1}$ : $\hat\theta_B$ systematically underestimates $\theta$ , as one would expect since the true maximum of the population can exceed any observed sample maximum. For $n=5$ , the bias is $-\theta/6$ . Second moment:

E[X_{(n)}^2] = \int_0^\theta y^2\cdot \frac{ny^{n-1}}{\theta^n}\,dy = \frac{n}{n+2}\theta^2,

\mathrm{Var}(X_{(n)}) = \frac{n}{n+2}\theta^2 - \left(\frac{n}{n+1}\theta\right)^2 = \theta^2\left[\frac{n}{n+2} - \frac{n^2}{(n+1)^2}\right].

For $n=5$ : $\frac{5}{7} - \frac{25}{36} = \frac{180 - 175}{252} = \frac{5}{252}$ , so $\mathrm{Var}(\hat\theta_B) = \frac{5}{252}\theta^2 \approx 0.01984\,\theta^2$ .

MSE comparison. By [Mean Squared Error Decomposition],

\mathrm{MSE}(\hat\theta_A) = \mathrm{Var}(\hat\theta_A) = \frac{\theta^2}{15} \approx 0.0667\,\theta^2,

\mathrm{MSE}(\hat\theta_B) = \mathrm{Var}(\hat\theta_B) + \mathrm{Bias}(\hat\theta_B)^2 = \frac{5}{252}\theta^2 + \frac{\theta^2}{36} = \theta^2\left(0.01984 + 0.02778\right) \approx 0.0476\,\theta^2.

So despite being biased, $\hat\theta_B$ has strictly smaller MSE than the unbiased $\hat\theta_A$ for every $\theta$ , at $n=5$ — a concrete demonstration that unbiasedness alone does not guarantee good estimation, and MSE is the right unifying criterion. (The usual fix, taking $\hat\theta_C = \frac{n+1}{n}X_{(n)}$ , restores unbiasedness while retaining most of the variance reduction; its MSE at $n=5$ is $\mathrm{Var}(\hat\theta_C) = (6/5)^2\cdot\frac{5}{252}\theta^2 \approx 0.0286\,\theta^2$ , better still.)

Example 2 — Estimating a Bernoulli proportion with a biased "add-two" estimator, and checking consistency. Let $X_1,\dots,X_n$ be i.i.d. $\mathrm{Bernoulli}(p)$ , e.g. $X_i = 1$ if the $i$ -th sampled water sample is contaminated, $0$ otherwise, with true $p \in (0,1)$ unknown. Suppose a lab of $n = 40$ samples finds $17$ contaminated ( $\sum x_i = 17$ ). Estimator A (sample proportion): $\hat p_A = \bar X = \frac1n\sum X_i$ . Here $\hat p_A = 17/40 = 0.425$ . Estimator B ("Laplace add-two" smoothing, common in small-sample settings to avoid estimates of exactly $0$ or $1$ ):

\hat p_B = \frac{\sum_{i=1}^n X_i + 1}{n+2}.

Here $\hat p_B = (17+1)/(42) = 18/42 = 3/7 \approx 0.4286$ .

Unbiasedness of $\hat p_A$ . Since $E[X_i] = p$ , linearity gives $E[\hat p_A] = p$ exactly, for every $n$ and every $p \in (0,1)$ : $\hat p_A$ is unbiased. Its variance is $\mathrm{Var}(X_i)/n = p(1-p)/n$ ; at $n=40$ this is $p(1-p)/40$ .

Bias of $\hat p_B$ . Write $S = \sum X_i \sim \mathrm{Binomial}(n,p)$ , so $E[S] = np$ . Then

E[\hat p_B] = \frac{E[S]+1}{n+2} = \frac{np+1}{n+2}.

The bias is

\mathrm{Bias}(\hat p_B) = \frac{np+1}{n+2} - p = \frac{np + 1 - p(n+2)}{n+2} = \frac{1 - 2p}{n+2}.

This is nonzero unless $p = 1/2$ , so $\hat p_B$ is biased for general $p$ — but the bias is $O(1/n)$ and shrinks as $n$ grows. At $n=40$ , if the true $p$ were, say, $0.4$ , the bias would be $(1-0.8)/42 = 0.2/42 \approx 0.00476$ : small, but present for every finite $n$ .

Variance of $\hat p_B$ . Since $\mathrm{Var}(S) = np(1-p)$ ,

\mathrm{Var}(\hat p_B) = \frac{\mathrm{Var}(S)}{(n+2)^2} = \frac{np(1-p)}{(n+2)^2}.

Compare to $\mathrm{Var}(\hat p_A) = p(1-p)/n = \frac{np(1-p)}{n^2}$ . Since $(n+2)^2 > n^2$ , we have $\mathrm{Var}(\hat p_B) < \mathrm{Var}(\hat p_A)$ for every $n$ and every $p \in (0,1)$ — the smoothing trades a small, shrinking bias for a uniformly smaller variance, again illustrating the bias–variance tradeoff central to MSE comparisons.

Consistency check for both. For $\hat p_A$ : it is exactly unbiased for every $n$ , and $\mathrm{Var}(\hat p_A) = p(1-p)/n \to 0$ as $n\to\infty$ (since $p(1-p) \le 1/4$ is bounded), so by the Corollary to [Consistency of the Sample Mean], $\hat p_A$ is consistent. For $\hat p_B$ : $\mathrm{Bias}(\hat p_B) = (1-2p)/(n+2) \to 0$ and $\mathrm{Var}(\hat p_B) = np(1-p)/(n+2)^2 \to 0$ as $n \to \infty$ (numerator grows linearly, denominator quadratically), so by [Mean Squared Error Decomposition], $\mathrm{MSE}(\hat p_B) \to 0$ for every fixed $p$ . Since $\mathrm{MSE}$ bounds tail probability via Markov's inequality applied to $(\hat p_B - p)^2$ ,

P(|\hat p_B - p| > \varepsilon) = P\big((\hat p_B - p)^2 > \varepsilon^2\big) \le \frac{\mathrm{MSE}(\hat p_B)}{\varepsilon^2} \to 0,

so $\hat p_B$ is also consistent, despite being biased at every finite $n$ . This shows consistency is a large-sample property and is compatible with finite-sample bias, as long as that bias vanishes in the limit.

+

Scope, edge cases & common traps

Situation	What to check	Common pitfall
Comparing two estimators	Compute MSE at each $\theta$ , not just one value	Comparing only variances when one estimator is biased
Small sample, extreme statistic (e.g. max, min)	Bias often present and computable in closed form	Assuming "natural-looking" estimators (like the max) are automatically unbiased
Large-sample claims	Consistency requires a limit statement, not a fixed- $n$ property	Confusing "unbiased" with "consistent" — neither implies the other in general
Heavy-tailed populations	Verify $E[X_i^2] < \infty$ before invoking Chebyshev-type bounds	Applying variance-based consistency arguments to Cauchy-like data
Vector-valued $\theta$	MSE becomes a matrix (covariance $+$ bias-outer-product); "smaller" needs a matrix order	Silently treating multivariate MSE as an ordinary number

The mathematician behind it

Origins · 1 minRonald Aylmer Fisher and how this was discovered

Ronald Aylmer Fisher

1890–1962 · Rothamsted Experimental Station and University College London

Fisher, working largely at the Rothamsted agricultural research station in the 1910s–1920s, built much of the vocabulary and machinery still used to evaluate estimators today: he coined the terms consistency, efficiency, and sufficiency as formal criteria for comparing estimators, and introduced maximum likelihood estimation as a general recipe for constructing them. His 1922 paper "On the Mathematical Foundations of Theoretical Statistics" is widely regarded as founding modern parametric estimation theory, framing the parameter-and-statistic language this article uses. Later work by Jerzy Neyman, Egon Pearson, and others sharpened the variance and mean-squared-error criteria into the comparative framework — unbiased minimum-variance estimation, efficiency bounds — that completes the picture Fisher began.

History

- 1890: Fisher born in London.

- 1922: "On the Mathematical Foundations of Theoretical Statistics" introduces consistency, efficiency, sufficiency, and maximum likelihood as formal criteria.

- 1925: Fisher's Statistical Methods for Research Workers popularizes estimation theory for practicing scientists.

- 1930s–1940s: Neyman–Pearson and later Cramér–Rao developments formalize variance lower bounds and minimum-variance unbiased estimation.

- 1962: Fisher dies in Adelaide, Australia, having shaped the language of point estimation used throughout statistics.

What exactly distinguishes an "estimator" from an "estimate"?

An estimator is a formula or rule — a statistic $\hat\theta(X_1,\dots,X_n)$ — defined before data are observed, and it is a random variable because the data are random. An estimate is the specific number you get when you plug in observed data $x_1,\dots,x_n$ into that formula. For example, $\bar X = \frac1n\sum X_i$ is an estimator of the population mean; if a particular sample gives $\bar x = 4.16$ , then $4.16$ is the estimate. All the probabilistic properties discussed in this article — bias, variance, MSE, consistency — are properties of the estimator (its sampling distribution over hypothetical repeated samples), never of a single realized estimate, which is just one number with no distribution of its own.

Can an estimator be both biased and consistent?

Yes, and Example 2's "add-two" estimator $\hat p_B$ demonstrates this explicitly: its bias $(1-2p)/(n+2)$ is nonzero for every finite $n$ (unless $p=1/2$ ), yet it vanishes as $n\to\infty$ , and its variance also vanishes, so its MSE tends to zero and it is consistent. Consistency is purely an asymptotic ( $n\to\infty$ ) statement, while unbiasedness is a finite- $n$ statement required to hold for every $n$ . An estimator can fail the finite- $n$ requirement while still satisfying the asymptotic one, as long as bias and variance both shrink to zero in the limit.

Does small variance always mean a good estimator?

No — variance alone ignores where the sampling distribution is centered. An estimator with tiny variance but centered far from the true $\theta$ (large, persistent bias) will have large MSE and will reliably miss the target with great precision. The correct combined criterion is MSE, which by the [Mean Squared Error Decomposition] theorem equals variance plus squared bias; only when comparing estimators that share the same bias (in particular, both zero) does comparing variances alone suffice to rank them by MSE.

Why is consistency considered a minimal requirement rather than a strong guarantee?

Consistency only guarantees that, given enough data, the estimator's sampling distribution collapses onto the true parameter value; it says nothing about how fast this happens, nor about performance at any fixed, finite sample size actually available in practice. Two consistent estimators can have wildly different finite-sample MSE, and an experimenter with $n=30$ observations cares about that finite-sample behavior, not the $n\to\infty$ limit. Consistency functions as a sanity check — an estimator that is not consistent is almost never usable — rather than as a positive endorsement of practical performance.

How does this connect to maximum likelihood estimation and the method of moments?

Point estimation as developed here supplies the criteria — unbiasedness, variance, MSE, consistency — against which any proposed estimator is judged; it does not, by itself, tell you how to construct an estimator. The method of moments constructs estimators by equating sample moments to population moments and solving for $\theta$ , while maximum likelihood estimation constructs estimators by maximizing the likelihood of the observed data over $\theta$ . Both are estimator-construction recipes whose resulting statistics are then evaluated exactly by the machinery in this article: is the moment estimator unbiased, what is its variance, is the MLE consistent, and so on.

Is an unbiased estimator with the smallest possible variance always the "best" choice?

Among unbiased estimators, the one with uniformly minimum variance (a UMVUE, when it exists) is optimal by the MSE criterion restricted to that class, since MSE equals variance there. But restricting to unbiased estimators is itself a choice, not a logical necessity: Example 1 shows a biased estimator can have strictly smaller MSE than the best unbiased competitor at a given sample size. So "best unbiased" and "best MSE overall" can differ; whether unbiasedness is required is a modeling decision, often driven by other considerations such as interpretability or downstream use, not derivable from MSE alone.

Why must bias and variance be evaluated "for every

\theta

" rather than at one value?

Because $\theta$ is unknown, a criterion that only worked for one particular true value of $\theta$ would be useless in practice — one could not know in advance which value happened to hold. Unbiasedness is defined as $E_\theta[\hat\theta]=\theta$ holding for all $\theta \in \Theta$ precisely so it is a property of the estimator's formula, verifiable in advance without knowing the truth. The same universal quantification applies to consistency and to MSE comparisons: an estimator with lower MSE at one specific $\theta$ but higher MSE at another is not uniformly better, and such estimators require additional criteria (like minimax risk) to compare, beyond the scope of a single- $\theta$ check.