Latent Variable Models and the Evidence Lower Bound

A latent variable model describes data through a joint distribution $p_\theta(x, z) = p(z)\, p_\theta(x \mid z)$, where $x$ is observed, $z$ is an unobserved (latent) variable, $p(z)$ is a prior over latents, and $p_\theta(x\mid z)$ is the generative or "decoder" distribution parameterized by $\theta$. The model explains data by imagining that nature first draws a hidden cause $z$, then generates the visible $x$ conditioned on it. Fitting the model means choosing $\theta$ to make observed data likely under this story, i.e., maximizing the marginal likelihood (also called the evidence):

For discrete $z$ this is a sum instead of an integral, but the difficulty is the same: the integral marginalizes out every possible explanation $z$ for the data, weighted by how plausible that explanation is. When $p_\theta(x \mid z)$ is a simple conjugate distribution (e.g., linear-Gaussian), this integral can sometimes be solved in closed form. But when $p_\theta(x \mid z)$ is parameterized by a neural network, the integral has no analytic solution, and its dimension (over all possible $z$) makes numerical quadrature or naive Monte Carlo hopelessly slow — this is precisely the intractable marginal likelihood problem. A related, equally intractable object is the posterior over latents given data, $p_\theta(z \mid x) = p_\theta(x, z) / p_\theta(x)$, which tells you which hidden causes plausibly generated a specific observation. Since it is defined using $p_\theta(x)$, if the evidence is intractable, so is the exact posterior. Variational inference solves both problems at once: introduce a tractable family of distributions $q_\phi(z \mid x)$, indexed by parameters $\phi$, meant to approximate the true posterior. Then derive a bound on $\log p_\theta(x)$ that only requires expectations under $q_\phi$, never under the true posterior. That bound is the Evidence Lower BOund, or ELBO.

Deriving the ELBO. Start from the log marginal likelihood and introduce $q_\phi(z\mid x)$ by multiplying and dividing inside the integral:

Because $\log$ is concave, Jensen's inequality lets us move the log inside the expectation, only decreasing the value:

This quantity $\mathcal{L}(\theta,\phi;x)$ is the ELBO. It is fully tractable whenever we can sample from $q_\phi(z\mid x)$, evaluate its density, and evaluate $p(z)$ and $p_\theta(x\mid z)$ — all mild requirements satisfied by simple parametric families like Gaussians. Two equivalent rewritings expose its meaning:

which balances explaining $x$ well against staying close to the prior, and

which reveals the exact gap between the true evidence and the bound: it is the KL divergence from our approximate posterior to the true posterior. Since KL divergence is always non-negative, this immediately re-derives the inequality and, more importantly, tells us how to close the gap: choose $\phi$ to make $q_\phi(z\mid x)$ as close as possible to $p_\theta(z\mid x)$.

The practical recipe alternates — or jointly performs via gradient ascent — two goals: improve the generative model $\theta$ and improve the approximate posterior $\phi$, both by maximizing the same scalar objective, the ELBO, summed (or averaged) over the dataset.

Step-by-step mechanism: 1. Choose a prior $p(z)$, typically simple (e.g., standard normal $\mathcal{N}(0, I)$), fixed or learned. 2. Choose a generative (decoder) family $p_\theta(x\mid z)$, e.g., a neural network mapping $z$ to the parameters of a distribution over $x$ (Gaussian mean/variance for continuous $x$, Bernoulli logits for binary $x$). 3. Choose a variational (encoder) family $q_\phi(z\mid x)$, e.g., a neural network mapping $x$ to the mean and variance of a diagonal Gaussian over $z$. 4. For each data point (or minibatch), estimate the ELBO:

• Sample $z \sim q_\phi(z\mid x)$ (or use a closed-form expectation when possible).
• Compute $\log p_\theta(x\mid z)$ — the reconstruction log-likelihood.
• Compute $D_{KL}(q_\phi(z\mid x)\,\|\,p(z))$ — closed-form when both are Gaussian.
• Combine: $\hat{\mathcal{L}} = \log p_\theta(x\mid z) - D_{KL}(q_\phi(z\mid x)\,\|\,p(z))$.

5. Ascend the gradient of $\hat{\mathcal{L}}$ with respect to both $\theta$ and $\phi$ (this requires differentiating through the sampling step — the reparameterization trick, foreshadowing the Variational Autoencoder). 6. Repeat over the dataset until convergence; at convergence, $\theta$ defines a good generative model and $q_\phi(z\mid x)$ approximates the true posterior for any $x$. This is the modern, amortized, gradient-based version. A classical, non-amortized ancestor is Expectation-Maximization (EM), which alternates fully solving for $q$ (the E-step, setting $q = p_\theta(z\mid x)$ exactly when tractable) and then maximizing over $\theta$ (the M-step) — both steps are coordinate ascent on the exact same ELBO objective. ``` ALGORITHM: ELBO-based training of a latent variable model (variational EM, amortized) Input: dataset D = {x_1, ..., x_N} prior p(z) decoder p_theta(x | z) # neural network, params theta encoder q_phi(z | x) # neural network, params phi learning rate eta, number of steps T Initialize theta, phi randomly for step = 1 to T: sample minibatch B = {x_1, ..., x_m} from D grad_theta = 0 grad_phi = 0 for each x in B:

# --- variational step: get an approximate posterior sample --- (mu, sigma) = encoder_forward(phi, x) # e.g. Gaussian params epsilon ~ N(0, I) # reparameterization noise z = mu + sigma * epsilon # differentiable sample

# --- evaluate the ELBO for this point --- recon_ll = log p_theta(x | z) # decoder log-likelihood kl_term = KL( N(mu, sigma^2) || p(z) ) # closed form if p(z)=N(0,I) elbo_x = recon_ll - kl_term

# --- backpropagate through both terms --- grad_theta += d(elbo_x) / d(theta) grad_phi += d(elbo_x) / d(phi) # flows through z via reparameterization theta = theta + eta grad_theta / m # gradient ASCENT on ELBO phi = phi + eta grad_phi / m return theta, phi ``` The key subtlety in this pseudocode is that $z$ is drawn via $z = \mu + \sigma \odot \epsilon$ with $\epsilon \sim \mathcal{N}(0,I)$ fixed as external randomness, rather than sampled directly as $z \sim \mathcal{N}(\mu, \sigma^2)$. This reparameterization makes $z$ a deterministic, differentiable function of $\phi$ and noise, so gradients can flow from the ELBO back into the encoder — without it, `grad_phi` could not be computed by ordinary backpropagation, only by higher-variance estimators like REINFORCE.

Correctness — why the inequality holds and what "tight" means. The derivation used only two facts: the algebraic identity $p_\theta(x,z) = q_\phi(z\mid x)\cdot \frac{p_\theta(x,z)}{q_\phi(z\mid x)}$ (valid wherever $q_\phi(z\mid x) > 0$), and Jensen's inequality for the concave function $\log$, which states $\log \mathbb{E}[Y] \ge \mathbb{E}[\log Y]$ for any positive random variable $Y$, with equality iff $Y$ is almost surely constant. Applying this with $Y = p_\theta(x,z)/q_\phi(z\mid x)$ gives the ELBO inequality, and equality holds exactly when $p_\theta(x,z)/q_\phi(z\mid x)$ is constant in $z$ — which happens precisely when $q_\phi(z\mid x) \propto p_\theta(x,z)$, i.e. when $q_\phi(z\mid x) = p_\theta(z\mid x)$ (the true posterior). This matches the second derivation exactly: the gap is $D_{KL}(q_\phi(z\mid x)\,\|\,p_\theta(z\mid x)) \ge 0$, with equality iff the two distributions are identical (KL divergence is zero iff its two arguments coincide almost everywhere). So the ELBO is not an ad hoc approximation — it is provably a lower bound for any valid probability distribution $q_\phi$, and the slack is a well-understood, non-negative quantity that vanishes exactly when inference is perfect. This also explains why maximizing the ELBO over $\phi$ alone (holding $\theta$ fixed) performs correct approximate inference: since $\log p_\theta(x)$ does not depend on $\phi$, increasing $\mathcal{L}(\theta,\phi;x)$ can only be achieved by decreasing the KL gap. And maximizing over $\theta$ pushes up a valid lower bound on the true log-likelihood, so (unlike, say, maximizing a heuristic surrogate) improving the ELBO is guaranteed to improve a certified floor on how well the model explains the data.

Complexity. Let $d$ be the dimensionality of $z$, and suppose the encoder and decoder are neural networks with $O(P)$ parameters and $O(F)$ floating-point operations per forward pass.

• Exact marginal likelihood (baseline, for contrast): computing $p_\theta(x) = \int p(z) p_\theta(x\mid z)\,dz$ exactly by numerical integration over a grid costs $O(k^d)$ for $k$ grid points per dimension — exponential in $d$, intractable for $d$ beyond a handful of dimensions. This is the problem the ELBO exists to avoid.
• ELBO estimation (Monte Carlo, one sample): $O(F)$ — one encoder forward pass to get $q_\phi(z\mid x)$'s parameters, one sample of $z$ (cost $O(d)$, negligible next to $F$), one decoder forward pass to get $\log p_\theta(x\mid z)$, and, when $q$ and the prior are both Gaussian, a closed-form KL term costing only $O(d)$ arithmetic operations (no sampling needed for that piece). Crucially this cost does not grow with $d$ exponentially — it is linear in network size, independent of how the exponential-integral difficulty would have scaled.
• Gradient computation: backpropagation through encoder and decoder costs the same order as the forward pass, $O(F)$, by the standard reverse-mode differentiation argument (each op is differentiated once).
• Per training step overall: $O(m F)$ for a minibatch of size $m$; memory: $O(P + mF_{\text{act}})$ for parameters plus stored activations for backpropagation, exactly as in standard supervised deep learning.
• Variance of the estimate: because $z$ is stochastic, the ELBO estimate from a single sample is a Monte Carlo estimator with some variance; more samples of $z$ reduce variance at cost $O(kF)$ for $k$ samples, trading compute for lower-variance gradients — a tunable, not a fundamental, tradeoff.

Honest tradeoffs. The ELBO purchases tractability at the price of an approximation gap that is rarely known exactly in practice (we cannot compute the true posterior to check how tight our bound is). A poorly chosen $q_\phi$ family — e.g., a diagonal Gaussian approximating a true posterior that is highly multimodal or correlated — can leave a large, uncorrectable gap no matter how well $\phi$ is optimized within that family; this is a model-mismatch limitation, not an optimization failure. Richer variational families (normalizing flows, hierarchical $q$) shrink the gap but increase computational cost per step.

Consider a toy one-dimensional latent variable model so every term can be computed by hand. Let the prior be standard normal, $p(z) = \mathcal{N}(0, 1)$. Let the decoder be a fixed linear-Gaussian model, $p(x \mid z) = \mathcal{N}(x; \,2z,\, 1)$ (mean $2z$, variance $1$) — chosen simple so we can also compute the true posterior for comparison, to see the ELBO gap concretely. Suppose we observe $x = 3$.

Step 1 — the true posterior (ground truth, for checking). Since prior and likelihood are conjugate Gaussians, $p(z\mid x) \propto p(z)\,p(x\mid z) \propto \exp\!\left(-\tfrac{z^2}{2}\right)\exp\!\left(-\tfrac{(x-2z)^2}{2}\right)$. Completing the square in $z$ gives a Gaussian posterior with

So the true posterior is $p(z\mid x=3) = \mathcal{N}(1.2,\ 0.2)$. (In a real deep model we would not have this in closed form — this is exactly why we need $q_\phi$.)

Step 2 — suppose our variational family is Gaussian $q_\phi(z\mid x) = \mathcal{N}(\mu, \sigma^2)$ and, before training, the encoder outputs a poor guess $\mu = 0.5,\ \sigma^2 = 1$ (close to the prior, badly off the true posterior).

Step 3 — compute the ELBO terms with this bad guess. Draw $z \sim \mathcal{N}(0.5, 1)$; take the mean-value estimate $z = 0.5$ for illustration (in practice we'd sample and/or use closed forms where possible).

• Reconstruction term: $\log p(x\mid z) = \log \mathcal{N}(3;\, 2(0.5),\, 1) = \log \mathcal{N}(3; 1, 1)$.

• KL term (closed form for two Gaussians $\mathcal{N}(\mu,\sigma^2)$ vs $\mathcal{N}(0,1)$):

• ELBO estimate: $\mathcal{L} \approx -2.919 - 0.125 = -3.044$.

Step 4 — repeat with a better guess, $\mu = 1.2,\ \sigma^2 = 0.2$ (matching the true posterior exactly), using $z = 1.2$:

• Reconstruction term: $\log \mathcal{N}(3; 2(1.2), 1) = \log \mathcal{N}(3; 2.4, 1) = -0.919 - \tfrac{(0.6)^2}{2} = -0.919 - 0.18 = -1.099$.
• KL term: $\tfrac12(0.2 + 1.44 - 1 - \log 0.2) = \tfrac12(0.2+1.44-1+1.609) = \tfrac12(2.249) = 1.125$.
• ELBO estimate: $\mathcal{L} \approx -1.099 - 1.125 = -2.224$.

Step 5 — check against the true log-evidence. For this linear-Gaussian model, $p(x) = \mathcal{N}(x; 0, \,4\cdot 1 + 1) = \mathcal{N}(3; 0, 5)$ (since $x = 2z + \text{noise}$, $z\sim\mathcal N(0,1)$ gives $\text{Var}(x)=4+1=5$):

(Note: because we used single point evaluations at the mean instead of a proper Monte Carlo expectation, the "bad $q$" row's ELBO estimate of $-3.044$ exceeding $-3.318$ reflects estimator noise/bias from skipping the expectation, not a violation of the theorem — with true expectations taken analytically, both values are guaranteed $\le -3.318$, and the good-$q$ case, using the exact posterior, gives $\mathcal L = \log p(x)$ exactly, since the KL gap is exactly zero there.) This subtlety itself is instructive: it shows why in practice we must be careful about Monte Carlo variance when reporting or comparing ELBO values, and why using the exact posterior as $q$ collapses the bound to equality — exactly as the correctness argument predicted.

The mathematics of bounding an intractable integral via a variational distribution has roots in statistical physics — mean-field approximations for the free energy of interacting systems — and was imported into statistics and machine learning through the lens of variational inference and the closely related Expectation-Maximization algorithm.

The ELBO is the training objective behind Variational Autoencoders, which power generative image models, molecular design pipelines, anomaly detection systems, and representation-learning backbones across industry. It underlies topic models like Latent Dirichlet Allocation, where variational inference (an ELBO maximization) is the standard scalable fitting algorithm for millions of documents. In Bayesian deep learning, ELBO-style objectives approximate posteriors over network weights for uncertainty quantification in medical imaging and autonomous systems. Probabilistic programming languages (Pyro, Stan's variational mode, TensorFlow Probability) implement generic ELBO optimizers so practitioners can plug in custom generative models without deriving inference by hand. The same bound, generalized to sequences of latents, appears in state-space models for time series and in diffusion-adjacent hierarchical latent variable models, making the ELBO one of the most reused mathematical objects connecting classical statistics to modern deep generative modeling, alongside foundational tools like KL Divergence and Bayes' Rule.

Ask the tutor: How does the reparameterization trick concretely enable gradient-based optimization of the ELBO in a Variational Autoencoder?