The Rao-Blackwell Theorem

The Rao–Blackwell principle recurs across statistics and computation wherever an estimator can be improved "for free" by exploiting distributional structure. In classical parametric inference, it is the standard route to building good unbiased estimators once a sufficient statistic (via the Factorization Theorem) has been identified, and it is the conceptual predecessor to the Lehmann–Scheffé theorem for uniformly minimum variance unbiased estimation. In Monte Carlo statistics, Rao–Blackwellization is an explicit variance-reduction technique: if a Monte Carlo estimator can be written as $E[h(X,Y)]$ and $E[h(X,Y)\mid X]$ has closed form, replacing simulated draws of $h(X,Y)$ by the analytically averaged $E[h(X,Y)\mid X]$ strictly reduces simulation variance for the same computational budget — a direct descendant of the theorem's inequality, now used in Gibbs samplers and importance sampling. In sequential Monte Carlo (particle filtering) for state-space models, "Rao–Blackwellized particle filters" analytically marginalize out a linear-Gaussian sub-state (via a Kalman filter) and use particles only for the nonlinear remainder, dramatically reducing the number of particles needed for a target accuracy — a computational-statistics workhorse in robotics (SLAM — simultaneous localization and mapping) and signal processing. In Bayesian statistics, the same conditioning idea underlies Rao–Blackwellized estimators of posterior functionals, where mixture representations allow closed-form conditional expectations to replace noisy posterior samples. In information theory and estimation theory more broadly, the theorem sits alongside the Cramer-Rao Lower Bound as one of the two central classical tools — one gives an achievable improvement path, the other gives a universal lower bound — for understanding how good an unbiased estimator can possibly be.

:::quiz[If $\delta(X)$ is already a function of the sufficient statistic $T$, what is $\delta^*(T) = E[\delta(X)\mid T]$?]

• Always strictly smaller in MSE than $\delta$
• Equal to $\delta(X)$ almost surely, with no improvement ✓
• Undefined, since $\delta$ has no residual randomness
• Equal to $\tau(\theta)$ exactly

Ask the tutor: Can you show how Rao–Blackwellization combined with a complete sufficient statistic pins down the unique minimum-variance unbiased estimator via Lehmann–Scheffé?

Think of the data $X$ as containing two ingredients: the part that is informative about $\theta$, and pure noise that is irrelevant to $\theta$ once you know $T(X)$. Sufficiency is precisely the statement that, conditional on $T$, the remaining randomness in $X$ carries no further information about $\theta$ — its distribution is the same no matter what $\theta$ truly is. Now imagine you have an unbiased estimator $\delta(X)$ that uses the whole dataset, but perhaps clumsily — it might use only $X_1$, say, ignoring $X_2,\dots,X_n$ entirely. If you replace $\delta(X)$ by its average over all the data configurations compatible with the observed value of $T$, you are averaging out exactly the "irrelevant noise" component, because that component's distribution does not depend on $\theta$. Averaging cannot change what you are estimating (bias is preserved), but averaging over a family of numbers always produces something with variance at most the variance of a randomly chosen member of the family — this is nothing but Jensen's inequality applied to conditional expectation, or equivalently the law of total variance. The picture below shows a single unbiased but noisy estimator's sampling distribution (wide, gray) alongside the conditional-mean estimator's sampling distribution (narrow, colored), both centered at the true value $\tau(\theta)$.

Example 1: Bernoulli trials, a deliberately wasteful estimator. Let $X_1,\dots,X_n$ be i.i.d. $\mathrm{Bernoulli}(p)$, $p\in(0,1)$, and suppose we want to estimate $p$. By the Factorization Theorem, since the joint pmf is

the statistic $T = \sum_{i=1}^n X_i$ is sufficient for $p$ (the factorization holds with $g_p(t) = (p/(1-p))^t (1-p)^n$ and $h(x)\equiv 1$). This matches Sufficiency and the Factorization Theorem. Take the deliberately crude, wasteful estimator $\delta(X) = X_1$ — using only the first observation. It is unbiased: $E_p[X_1] = p$. Its MSE is

Now compute the Rao–Blackwell estimator $\delta^*(T) = E[X_1 \mid T]$ where $T=\sum_i X_i$. By exchangeability of $(X_1,\dots,X_n)$ (they are i.i.d.), conditional on $T=t$ each of the $\binom{n}{t}$ arrangements of $t$ ones among $n$ slots is equally likely, so

(Formally: $\sum_{i=1}^n E[X_i\mid T] = E[T\mid T] = T$, and by symmetry each $E[X_i\mid T]$ is the same function of $T$, so each equals $T/n$.) Thus $\delta^*(T) = T/n = \bar X$, the sample mean — the familiar MLE/moment estimator. Its MSE is

Indeed $\mathrm{MSE}_p(\bar X) = \mathrm{MSE}_p(\delta)/n \le \mathrm{MSE}_p(\delta)$ for all $n\ge 1$, with strict inequality whenever $n\ge 2$ — exactly as guaranteed, since for $n\ge 2$, $X_1 \ne \bar X$ with positive probability under $P_p$ (e.g. if $X_1=1,X_2=0$). This is a clean numerical illustration: for $n=4$, $p=0.3$, $\mathrm{MSE}(\delta) = 0.21$ while $\mathrm{MSE}(\delta^*) = 0.0525$, a fourfold reduction, matching $\mathrm{MSE}(\delta)/n$.

Example 2: Poisson data, estimating a nonlinear functional $\tau(\lambda) = P(X_1 = 0) = e^{-\lambda}$. Let $X_1,\dots,X_n$ be i.i.d. $\mathrm{Poisson}(\lambda)$, $\lambda>0$. We want to estimate $\tau(\lambda) = e^{-\lambda} = P_\lambda(X_1=0)$ — a genuinely nonlinear function of $\lambda$, chosen to show the theorem's reach beyond estimating $\lambda$ itself. The joint pmf is $f_\lambda(x) = e^{-n\lambda}\lambda^{\sum x_i} / \prod_i x_i!$, which factors as a function of $T=\sum_i X_i$ times a $\lambda$-free term; hence $T=\sum_i X_i \sim \mathrm{Poisson}(n\lambda)$ is sufficient. A simple, crude, unbiased estimator of $\tau(\lambda)=e^{-\lambda}$ is

since $E_\lambda[\mathbf 1\{X_1=0\}] = P_\lambda(X_1=0) = e^{-\lambda}$. Its MSE (it's an indicator, so variance is $\tau(1-\tau)$):

Rao–Blackwellize: $\delta^*(T) = E[\mathbf 1\{X_1=0\} \mid T = t] = P(X_1 = 0 \mid T=t)$. Given $T=t$, the conditional distribution of $(X_1,\dots,X_n)$ given their sum is $t$ is multinomial: each unit of "count" is independently assigned to one of the $n$ observations with equal probability $1/n$ (a standard fact for independent Poisson variables conditioned on their sum). Hence, conditionally, $X_1 \mid T=t \sim \mathrm{Binomial}(t, 1/n)$, so

Therefore the Rao–Blackwell estimator is

This is exactly the uniformly minimum variance unbiased estimator (UMVUE) of $e^{-\lambda}$ for the Poisson family — a fact usually derived via Lehmann–Scheffé, but here we obtain a genuinely improved unbiased estimator directly from Rao–Blackwellizing the crude indicator, with no need for completeness to see the improvement (only to see that it cannot be improved further).

Numerical check, $n=5,\ \lambda = 2$ (so $\tau(\lambda) = e^{-2} \approx 0.1353$):

• $\mathrm{MSE}_\lambda(\delta) = e^{-2}(1-e^{-2}) \approx 0.1353 \times 0.8647 \approx 0.11702$.
• $E_\lambda[\delta^*(T)] = E_\lambda[(4/5)^T]$. Since $T\sim \mathrm{Poisson}(n\lambda)=\mathrm{Poisson}(10)$, the probability generating function gives $E[s^T] = e^{n\lambda(s-1)}$; at $s=4/5$: $E[(4/5)^T] = e^{10(4/5 - 1)} = e^{-2} \approx 0.1353$ — confirming unbiasedness exactly, as the theorem guarantees.
• To see the variance shrink numerically: $E_\lambda[\delta^{*2}] = E[(16/25)^T] = e^{10(16/25-1)} = e^{-3.6}\approx 0.02732$, so $\mathrm{Var}(\delta^*) = 0.02732 - (0.1353)^2 \approx 0.02732 - 0.01831 = 0.00901$.

So $\mathrm{MSE}(\delta^*) \approx 0.00901 \ll \mathrm{MSE}(\delta)\approx 0.11702$: roughly a thirteen-fold variance reduction, illustrating that Rao–Blackwellization can produce dramatic gains, not just marginal ones, especially when the naive estimator is very wasteful (using one data point out of five).