Generative Modeling: Learning Data Distributions

A generative model is a probabilistic model $p_\theta(x)$, parameterized by $\theta$ (e.g., neural network weights), trained on samples $x_1, \dots, x_n \overset{\text{iid}}{\sim} p_{\text{data}}$ so that $p_\theta$ approximates $p_{\text{data}}$. This contrasts with discriminative models, which learn $p(y \mid x)$ directly and are indifferent to the marginal structure of $x$. Generative models can be used for three related but distinct tasks: density estimation (compute or bound $p_\theta(x)$ for a given $x$), sampling (draw new $x \sim p_\theta$), and representation learning (extract a low-dimensional code $z$ that explains $x$). The taxonomy in this article rests on one axis: how the model relates to the likelihood function $p_\theta(x)$.

Explicit density, tractable likelihood. The model defines $p_\theta(x)$ via a formula that can be evaluated exactly in closed form (or with a cheap forward pass). Two major families achieve this:

• Autoregressive models factor the joint distribution using the chain rule of probability,

and parameterize each conditional with a neural network (e.g., PixelRNN, PixelCNN, GPT-style transformers over tokens). Every factor is a normalized distribution over one variable, so the product is automatically normalized — no partition function to worry about.

• Normalizing flows transform a simple base density $p_Z(z)$ (e.g., a standard Gaussian) through an invertible, differentiable map $x = f_\theta(z)$, and use the change-of-variables formula

Both families allow exact log-likelihood computation, so training reduces to plain maximum likelihood.

Explicit density, approximate likelihood. The model posits latent variables $z$ with $p_\theta(x) = \int p_\theta(x \mid z)\, p(z)\, dz$, but this integral is intractable to compute exactly for expressive decoders $p_\theta(x\mid z)$. Instead of the true likelihood, training maximizes a tractable lower bound on it — the evidence lower bound (ELBO) in Variational Autoencoders, or a denoising-based bound in diffusion models. You never compute $p_\theta(x)$ exactly, only a certified bound on $\log p_\theta(x)$.

Implicit density. The model provides only a sampling procedure — typically $x = G_\theta(z)$ for $z \sim p(z)$ a simple noise distribution and $G_\theta$ a neural network — with no tractable expression for $p_\theta(x)$ at all. Generative Adversarial Networks are the canonical example: a generator $G_\theta$ and discriminator $D_\phi$ are trained jointly so that $G_\theta$'s samples become indistinguishable from real data according to $D_\phi$, an objective that requires no likelihood evaluation whatsoever. The training principle splits along the same lines. Likelihood-based training (MLE or its bound) directly optimizes a probabilistic objective and gives you calibrated density values as a byproduct. Sampling-based / adversarial training optimizes an objective defined purely in terms of samples — often a divergence estimated implicitly through an auxiliary discriminator or a two-sample test — and never yields a usable $p_\theta(x)$.

Regardless of family, generative model training follows a common template: define a parametric object that can produce or score samples, define a loss connecting model and data, and optimize by gradient descent. Below are the two canonical mechanisms — MLE for explicit models, and adversarial training for implicit ones — laid out as algorithms, followed by a diagram placing all four families on the same map.

Mechanism 1 — Maximum Likelihood Training (explicit density). The population objective is to minimize the Kullback–Leibler divergence from data to model,

which is estimated with the empirical average over the training set — this is maximum likelihood estimation. ``` Algorithm: Maximum-Likelihood Training of an Explicit Density Model Input: dataset D = {x_1, ..., x_n}, model p_theta with tractable log p_theta(x) Output: fitted parameters theta 1. Initialize theta randomly 2. repeat until convergence: 3. sample a minibatch B ⊂ D of size m 4. compute batch log-likelihood: L(theta) = (1/m) sum_{x in B} log p_theta(x) 5. compute gradient g = ∇_theta L(theta) # exact if likelihood is tractable 6. theta ← theta + eta g # gradient ASCENT on log-likelihood 7. return theta

# For autoregressive models: log p_theta(x) = sum_i log p_theta(x_i | x_

# in one forward pass with a causal mask (teacher forcing).

# For normalizing flows: log p_theta(x) = log p_Z(f_theta^{-1}(x)) + log|det J_{f_theta^{-1}}(x)|,

# requiring f_theta invertible with a tractable Jacobian determinant.

# For latent-variable models (VAE/diffusion): replace step 4 with a lower bound ELBO(theta, phi) ≤ log p_theta(x),

# introducing an auxiliary inference network with parameters phi, and ascend the bound instead. ```

Mechanism 2 — Adversarial (sampling-based) training (implicit density). ``` Algorithm: Adversarial Training of an Implicit Generative Model Input: dataset D of real samples, noise prior p(z), generator G_theta, discriminator D_phi Output: fitted generator parameters theta 1. Initialize theta, phi randomly 2. repeat until convergence: 3. sample minibatch of real data {x_1,...,x_m} ~ D 4. sample minibatch of noise {z_1,...,z_m} ~ p(z) 5. fake samples: x̂_j = G_theta(z_j) 6. # --- discriminator step: push D toward distinguishing real vs. fake --- 7. phi ← phi + eta_D ∇_phi [ (1/m) sum_i log D_phi(x_i) + (1/m) sum_j log(1 - D_phi(x̂_j)) ] 8. # --- generator step: push G to fool the (updated) discriminator --- 9. re-sample noise, recompute x̂_j = G_theta(z_j) 10. theta ← theta + eta_G ∇_theta [ (1/m) sum_j log D_phi(x̂_j) ] # non-saturating variant 11. return theta

# No log p_theta(x) is ever computed. D_phi acts as a learned, implicit density-ratio estimator:

# at the Nash equilibrium, D*(x) = p_data(x) / (p_data(x) + p_theta(x)), which is exactly 1/2 everywhere

# iff p_theta = p_data, tying the adversarial game to Jensen–Shannon divergence minimization. ``` The diagram below places the four families on a single map along the explicit/implicit and tractable/approximate axes, showing how each foreshadowed architecture fits.

Time and space complexity by family. Autoregressive models. Training likelihood for a $d$-dimensional input costs $O(d)$ sequential conditional evaluations, but with masked architectures (causal convolutions, causal attention) all $d$ conditionals are computed in parallel in one forward pass, giving $O(d)$ work but $O(1)$ sequential depth relative to $d$ during training. Sampling, however, is inherently sequential: generating $x_i$ requires $x_1,\dots,x_{i-1}$, so sampling costs $O(d)$ sequential network evaluations — this asymmetry (fast parallel training, slow sequential sampling) is a defining, and honest, limitation of autoregressive models. Normalizing flows. Density evaluation costs one forward (or inverse) pass through $f_\theta$ plus the log-determinant of a $d \times d$ Jacobian, which is $O(d^3)$ in general. Practical flow architectures (coupling layers, autoregressive flows) constrain $f_\theta$ so the Jacobian is triangular, reducing the determinant to $O(d)$ — the product of diagonal entries. This architectural constraint is what makes flows tractable at all; it is a real restriction on expressiveness, not a free lunch. VAEs. One forward pass through encoder and decoder costs $O(d)$ (network-dependent), and the ELBO is computed with a single Monte Carlo sample of $z$ via the reparameterization trick, adding negligible overhead. Sampling is a single decoder forward pass — $O(1)$ network evaluations, much faster than autoregressive sampling. GANs. One generator forward pass produces a sample, $O(1)$ network evaluations — the fastest sampling of any family discussed. Training cost per step is one discriminator pass plus one generator pass, but the number of steps to convergence is empirically much less predictable, since there is no monotone loss to track.

Why maximum likelihood is the correct objective (explicit models). Minimizing $D_{\mathrm{KL}}(p_{\text{data}} \| p_\theta)$ over $\theta$ is equivalent to maximizing $\mathbb{E}_{p_{\text{data}}}[\log p_\theta(x)]$, since

and the first term does not depend on $\theta$. Because $D_{\mathrm{KL}} \geq 0$ with equality iff $p_{\text{data}} = p_\theta$ almost everywhere (Gibbs' inequality), driving the KL to its minimum value of zero — which MLE does asymptotically, as $n \to \infty$, under standard regularity conditions and a sufficiently expressive model class — is provably the right target. The empirical average $\frac{1}{n}\sum_i \log p_\theta(x_i)$ is an unbiased Monte Carlo estimator of $\mathbb{E}_{p_{\text{data}}}[\log p_\theta(x)]$, so gradient ascent on the sample average is a consistent estimator of the population optimum by the law of large numbers.

Why the ELBO is a valid surrogate (approximate-likelihood models). For any latent-variable model with variational posterior $q_\phi(z\mid x)$,

Since the KL term is non-negative, the ELBO is a certified lower bound on the true log-likelihood for every $x$; maximizing it maximizes a guaranteed floor on the quantity we actually care about, and the gap closes as $q_\phi$ approaches the true posterior.

Why the adversarial game targets the right divergence. At a fixed generator, the optimal discriminator is $D^*(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_\theta(x)}$ (this is the Bayes-optimal classifier between the two distributions, derivable by maximizing the discriminator's expected log-likelihood pointwise). Substituting $D^*$ back into the value function shows the generator's objective becomes, up to a constant, $2\, D_{\mathrm{JS}}(p_{\text{data}} \,\|\, p_\theta) - \log 4$, the Jensen–Shannon divergence. Since $D_{\mathrm{JS}} \geq 0$ with equality iff the distributions coincide, the global minimum of the generator's loss (given an optimal discriminator) is achieved exactly when $p_\theta = p_{\text{data}}$ — this is the theoretical justification for adversarial training, though it assumes an idealized, fully-optimized discriminator at every step, which practical alternating gradient descent only approximates.

Honest tradeoffs. Tractable-likelihood models give calibrated probabilities and stable, well-understood training (plain gradient ascent on a scalar objective) but pay for it — autoregressive sampling is slow, and flows require restrictive invertibility constraints that can limit expressiveness. Approximate-likelihood models relax the invertibility constraint but only optimize a bound, so the model may be penalized less for missing modes than a true likelihood would penalize it (a known failure mode of posterior-collapse in VAEs). Implicit models sample fastest and can produce the sharpest, most realistic outputs, but they lose density evaluation entirely, offer no direct measure of overfitting or mode coverage, and their non-convex minimax training can oscillate or collapse (mode collapse) rather than converge.

Consider a toy dataset of 2-D points and three candidate ways to model $p(x_1, x_2)$: an autoregressive factorization, a flow, and an implicit sampler. Suppose we observe $n = 4$ points:

Step 1 — Autoregressive model. Factor $p_\theta(x_1, x_2) = p_\theta(x_1)\, p_\theta(x_2 \mid x_1)$. Suppose after training, the model assigns

The exact joint log-likelihood of the dataset is computed by simple multiplication and summation — no integral, no approximation:

This is an exact number — the defining feature of the autoregressive/flow family. Gradient ascent adjusts the conditional tables (or network weights producing them) directly on this quantity.

Step 2 — Latent-variable model (VAE-style). Now suppose instead $p_\theta(x) = \int p_\theta(x\mid z)\,p(z)\,dz$ with a 1-D Gaussian latent $z$. We cannot integrate this exactly for a neural decoder, so we instead compute the ELBO for point $(1,2)$ using an encoder $q_\phi(z \mid x) = \mathcal{N}(\mu = 0.3, \sigma^2 = 0.8)$ and prior $p(z) = \mathcal{N}(0,1)$:

The KL term between two Gaussians has a closed form:

If the reconstruction term (estimated with one sampled $z$ via reparameterization, $z = \mu + \sigma\epsilon$) evaluates to, say, $-1.10$, then $\text{ELBO}(x) \approx -1.10 - 0.057 = -1.157$. This is not $\log p_\theta(x)$ — it is guaranteed to be less than or equal to the true (uncomputable) log-likelihood. Training maximizes this quantity, trusting the bound's tightness rather than exactness.

Step 3 — Implicit model (GAN-style). Here there is no likelihood step at all. A generator maps $z \sim \mathcal{N}(0, I)$ to $\hat x = G_\theta(z)$; suppose $z = 0.5$ maps to $\hat x = (1.1, 1.9)$. We never compute $p_\theta((1.1,1.9))$. Instead, the discriminator scores it: $D_\phi(1.1, 1.9) = 0.42$ (closer to 0 = "looks fake," closer to 1 = "looks real"). Training updates $\theta$ to push this score toward $1$ and updates $\phi$ to keep distinguishing real points (which score near $1$) from generated ones — the entire signal is comparative, sample-vs-sample, never a probability of the data. The contrast is the whole lesson: identical dataset, three fundamentally different notions of what "fitting the distribution" even means numerically.

The generative modeling paradigm draws on a century of probability and statistics, crystallized into deep learning through a sequence of foundational papers connecting neural networks to explicit and implicit density estimation.

Explicit autoregressive models power large language models: every modern text generator (GPT-family and successors) is, at its core, an autoregressive density model over tokens trained by maximum likelihood, exactly the mechanism described above. Normalizing flows are used in scientific simulation (lattice field theory, molecular conformer generation) where exact density evaluation is required for downstream physics calculations, and in some fast, invertible audio vocoders. Variational Autoencoders and their descendants underlie representation learning pipelines, anomaly detection systems, and are a core building block inside modern latent diffusion image generators (compressing images into a latent space before the diffusion process operates). Generative Adversarial Networks, despite training instability, remain widely used for fast, high-resolution image synthesis, super-resolution, image-to-image translation, and data augmentation in domains such as medical imaging where labeled data is scarce. Diffusion models, which combine the variational/approximate-likelihood viewpoint with an iterative sampling procedure, currently dominate state-of-the-art image, video, and audio generation systems (e.g., large-scale text-to-image and text-to-video systems), and represent the natural next architecture built directly on the concepts introduced here.

Ask the tutor: How does the reparameterization trick make the ELBO differentiable with respect to the encoder's parameters?