Sufficiency and the Factorization Theorem

Sufficiency is the conceptual backbone of the Rao-Blackwell Theorem, which shows that averaging any estimator over the conditional distribution given a sufficient statistic can only improve (or leave unchanged) its mean squared error — turning the abstract "no information lost" idea into an active tool for constructing better estimators. It underlies the Lehmann–Scheffé theorem for uniformly minimum-variance unbiased estimation, and the very definition of an exponential family is built around statistics that remain jointly sufficient regardless of sample size. In Bayesian statistics, sufficiency justifies updating a prior using only $T(X)$ rather than the full sample when $T$ is sufficient, since the likelihood factors through $T$. In data compression and information theory, sufficiency formalizes the idea of a "minimal lossless summary" of data relative to a parametric question, closely related to the data-processing inequality. In machine learning, sufficient statistics appear explicitly in exponential-family models (e.g. the natural parameters and sufficient statistics of a Gaussian or a categorical distribution used in graphical models and variational inference), and in the design of efficient online algorithms that update only a compressed summary of streaming data rather than storing every observation.

Ask the tutor: How do I show that $T(X)=(\sum X_i,\sum X_i^2)$ is jointly sufficient—but neither component is sufficient alone—for a Normal$(\mu,\sigma^2)$ sample?

Think of the data $X$ as containing two logically separable pieces: the part relevant to estimating $\theta$, and the part that is "just noise," analogous to the arbitrary order in which independent trials happen to arrive. Sufficiency formalizes the claim that a statistic $T(X)$ has extracted all the relevant piece, so that after conditioning on $T(X)=t$, whatever remains random about $X$ behaves identically regardless of what $\theta$ actually is — you could simulate the leftover randomness yourself, without knowing $\theta$, using only a random number generator and the value $t$. A vivid way to picture this: imagine the sample space sliced into level sets $\{x : T(x) = t\}$, one slice per value of $t$. Sufficiency says that within each slice, the relative likelihood of different $x$'s is a fixed, $\theta$-independent shape; only the overall mass assigned to the whole slice depends on $\theta$. All the $\theta$-dependence "lives" at the level of slices, not within them.

This picture also explains why sufficiency is about compression without loss: reporting $T(X)$ alone lets anyone reconstruct — in a stochastic sense — the same joint behavior of $X$, because the "extra randomness" beyond $T$ is a known, parameter-free recipe.

Example 1 — Sufficiency for a Poisson rate with unequal exposures. Suppose $X_1,\dots,X_n$ are independent counts, where $X_i \sim \text{Poisson}(\lambda w_i)$, with $w_1,\dots,w_n > 0$ known, fixed "exposure" constants (e.g. observation times or population sizes) that differ across units — this is the realistic asymmetric version of the usual i.i.d. Poisson sample. The parameter of interest is the rate $\lambda > 0$. The joint probability mass function is

Group the factors: let

Then $f_\lambda(x) = g_\lambda(T(x)) h(x)$, where $g_\lambda$ depends on $x$ only through $t = \sum x_i$, and $h(x)$ does not involve $\lambda$ at all (the $w_i$'s are known constants, not parameters). By the Factorization Theorem, $T(X) = \sum_i X_i$ is sufficient for $\lambda$ — even though the exposures $w_i$ are unequal, so the individual $X_i$ are not identically distributed. This is a genuinely non-symmetric example: sufficiency of the total count survives heterogeneity in exposure, a fact that is not obvious without the algebraic criterion. As a direct check via the definition, condition on $T = t$. It is a classical fact (provable by the moment generating function or direct combinatorics) that given $\sum_i X_i = t$, the vector $(X_1,\dots,X_n)$ is Multinomial$\left(t;\ \frac{w_1}{W},\dots,\frac{w_n}{W}\right)$, whose probabilities $\binom{t}{x_1,\dots,x_n}\prod_i (w_i/W)^{x_i}$ involve only the known constants $w_i/W$, not $\lambda$ — confirming sufficiency directly, consistent with the factorization argument.

Example 2 — Sufficiency for a uniform upper endpoint (a non-exponential-family case). Let $X_1, \dots, X_n$ be i.i.d. $\text{Uniform}(0, \theta)$, $\theta > 0$ unknown, but suppose additionally we append a fixed, known "calibration" observation: $X_0 \equiv c$ for a known constant $c>0$ unrelated to $\theta$ (this makes the example asymmetric and forces care about what "the data" is). We seek a sufficient statistic for $\theta$ based on $(X_0, X_1,\dots,X_n)$. The joint density (with respect to Lebesgue measure on $\mathbb R^n$, treating $X_0=c$ as deterministic so effectively we work with the density of $(X_1,\dots,X_n)$) is

Let $T(x) = \max(x_1,\dots,x_n) =: x_{(n)}$. Write

Then $f_\theta(x) = g_\theta(T(x))\,h(x)$ for all $x$, and $h$ is free of $\theta$. By the Factorization Theorem, $T(X) = X_{(n)} = \max_i X_i$ is sufficient for $\theta$. Notice the calibration constant $c$ plays no role: since it does not depend on $\theta$, it is absorbed entirely into the $\theta$-free part $h$, illustrating that sufficiency concerns only the $\theta$-dependent structure of the likelihood, and constants (however included in "the data") never affect which statistic is sufficient. As a sanity check by direct computation: given $X_{(n)} = t \le \theta$, the conditional joint density of the order statistics is that of $n-1$ i.i.d. Uniform$(0,t)$ variables together with the known maximum $t$ — a distribution that manifestly does not involve $\theta$, confirming sufficiency without appeal to factorization.