Image Generation Fundamentals

A grayscale image of size $H \times W$ can be viewed as a single point $x \in \mathbb{R}^{HW}$. A dataset of images is then a finite set of samples drawn (we assume) i.i.d. from some unknown, fixed distribution $p_{\text{data}}(x)$ over that space. Generative modeling is the task of constructing a parametric distribution $p_\theta(x)$, using only these samples, such that $p_\theta$ is (a) close to $p_{\text{data}}$ in some useful sense and (b) easy to draw new samples from. This is fundamentally different from Convolutional Neural Networks used for discriminative tasks like classification, where the goal is $p(y \mid x)$ for a small label set $y$; here the "label" space is the entire image itself. Because $p_{\text{data}}$ is supported on a thin, curved manifold inside a huge ambient space — most points in $\mathbb{R}^{HW}$ look like television static, not photographs — directly parameterizing $p_\theta(x)$ with a simple distribution (like a single high-dimensional Gaussian) fails badly: it cannot represent the sharp, multi-modal, low-dimensional structure of real images. The standard fix is to introduce a latent variable $z \in \mathbb{R}^d$, with $d \ll HW$, and to define the image distribution indirectly:

so that $p_\theta(x) = \int p_\theta(x \mid z)\, p(z)\, dz$. The prior $p(z)$ is chosen simple (typically an isotropic Gaussian, $z \sim \mathcal{N}(0, I)$), and all of the complexity of turning "a random low-dimensional code" into "a photorealistic image" is pushed into a learned decoder network $g_\theta: z \mapsto x$. Sampling then reduces to drawing an easy Gaussian vector and pushing it through a neural network — trivial once trained. Two competing families answer the question "how do we fit $\theta$ so this actually works?" Autoregressive models avoid latent variables entirely and instead factor the image likelihood exactly via the chain rule of probability, pixel by pixel. Latent variable models such as the Variational Autoencoder introduce an auxiliary encoder $q_\phi(z\mid x)$ that infers a plausible $z$ for each training image, and train encoder and decoder jointly by maximizing a tractable lower bound on the log-likelihood — the Evidence Lower Bound (ELBO). Both give an explicit, trainable density; both are the direct ancestors of the implicit-density approaches (GANs) and iterative-denoising approaches (diffusion) covered elsewhere.

Autoregressive factorization

Any joint distribution over pixels $x = (x_1, \dots, x_n)$ (raster-scan order, say) can be factored exactly, with no approximation, using the chain rule:

A model like PixelCNN/PixelRNN parameterizes each conditional $p_\theta(x_i \mid x_{<i})$ with a neural network — masked convolutions ensure pixel $i$'s prediction only looks at pixels already generated. Training maximizes the exact log-likelihood $\sum_i \log p_\theta(x_i \mid x_{<i})$ by teacher-forcing (feeding true pixel values as context, so training is one parallel forward pass). Sampling, however, must generate pixels strictly in order — $n$ sequential network evaluations per image. ```

# Training: exact log-likelihood, fully parallel over pixels function TRAIN_AUTOREGRESSIVE(images, theta): for batch in images: loss = 0 for i in 1..n: # can be vectorized, no sequential dependency context = batch[:, 1..i-1] # true pixels, teacher-forced p_i = NETWORK(context; theta) # predicted distribution over x_i loss += -log p_i[batch[:, i]] backprop(loss); update(theta)

# Sampling: inherently sequential, one pixel unlocks the next function SAMPLE_AUTOREGRESSIVE(theta, n): x = zeros(n) for i in 1..n: p_i = NETWORK(x[1..i-1]; theta) x[i] = draw_from(p_i) return x ```

Latent-variable models and the ELBO

For a latent-variable model, the marginal likelihood $p_\theta(x) = \int p_\theta(x\mid z) p(z)\, dz$ is an intractable integral over all of $z$-space. We cannot compute it directly, and gradient ascent on an unknown quantity is impossible. The fix is to introduce an approximate posterior $q_\phi(z \mid x)$ — an encoder network predicting, for each image $x$, a distribution over plausible latents — and to derive a computable lower bound. Start from the (trivial) identity, for any distribution $q_\phi(z\mid x)$:

The KL term is $\ge 0$ (KL divergence is always non-negative), so the first term is a lower bound on $\log p_\theta(x)$ — the ELBO:

Maximizing $\mathcal{L}$ simultaneously increases the true likelihood (pushing the bound up) and tightens the bound (making the KL gap between $q_\phi$ and the true posterior $p_\theta(z\mid x)$ smaller). Both terms are now computable: the reconstruction term is estimated by sampling $z \sim q_\phi(z\mid x)$ and decoding; the KL term has a closed form when $q_\phi$ and $p(z)$ are both Gaussian. The remaining obstacle is that $z$ is sampled, and backpropagating through a random sampling operation is undefined. The reparameterization trick rewrites the sample as a deterministic, differentiable function of $\phi$ plus external noise:

so gradients flow through $\mu_\phi, \sigma_\phi$ normally, and the randomness is isolated in $\epsilon$, which requires no gradient. ```

# VAE training step function TRAIN_VAE_STEP(x, phi, theta): mu, log_sigma2 = ENCODER(x; phi) # q_phi(z|x) = N(mu, diag(sigma^2)) epsilon = sample_standard_normal(dim(mu)) z = mu + exp(0.5 log_sigma2) epsilon # reparameterization trick x_hat = DECODER(z; theta) # p_theta(x|z) recon_loss = RECONSTRUCTION_ERROR(x, x_hat) # e.g. Gaussian NLL or BCE per pixel kl_loss = -0.5 * sum(1 + log_sigma2 - mu^2 - exp(log_sigma2)) # closed-form N||N KL loss = recon_loss + kl_loss # negative ELBO (minimize this) backprop(loss) update(phi, theta)

# Sampling new images after training: ignore the encoder entirely function SAMPLE_VAE(theta, d): z = sample_standard_normal(d) x = DECODER(z; theta) return x ```

Autoregressive models. Training cost per image is $O(n)$ conditional density evaluations, but because teacher forcing supplies ground-truth context, all $n$ predictions can be computed in a single masked-convolution forward pass — training time is $O(n)$ work but effectively one parallel pass, i.e. wall-clock roughly $O(1)$ sequential steps given enough parallel hardware. Sampling is fundamentally sequential: pixel $i$ needs pixels $1,\dots,i-1$, so generating one image costs $O(n)$ sequential network evaluations, each itself $O(n)$ or more work depending on receptive field — a real bottleneck for large images (megapixel images can take seconds to minutes to sample). Space is $O(n)$ for the running image buffer plus model parameters. Correctness is exact: the chain-rule factorization is an identity, not an approximation, so a perfectly trained autoregressive model recovers $p_{\text{data}}$ exactly in the limit of infinite capacity and data — there is no approximation gap, only estimation error from finite data and finite model capacity.

VAEs. Both training and sampling are $O(1)$ network forward passes per image (one encoder pass, one decoder pass), independent of pixel count in terms of sequential steps — a huge sampling-speed advantage over autoregressive models. Space is dominated by the encoder/decoder parameters, $O(P)$, plus $O(d)$ for the latent code, $d \ll n$. The correctness argument for VAEs is the ELBO derivation itself, which is worth restating as an invariant: for any choice of $\phi$, at any point during training,

This is not an approximation that might fail; it is an algebraic identity plus the fact that KL divergence is non-negative (itself a consequence of Jensen's inequality applied to the concave $\log$ function). Gradient ascent on $\mathcal{L}$ with respect to $\theta$ therefore always pushes a true lower bound on the data likelihood upward. Gradient ascent with respect to $\phi$ tightens the bound by shrinking the KL gap. The tradeoff is that maximizing a lower bound does not guarantee maximizing the true likelihood: if $q_\phi$ is a poor approximating family (e.g., forced to be Gaussian when the true posterior is multi-modal), the gap $D_{KL}(q_\phi\|p_\theta(z\mid x))$ can remain large no matter how well $\theta,\phi$ are optimized, and samples tend to look characteristically blurry — the model hedges over multiple plausible modes rather than committing to one sharp explanation, because averaging over uncertainty in $z$ under a unimodal Gaussian encoder smooths the decoder's output distribution. Both families are honest densities: you can evaluate (or lower-bound) $\log p_\theta(x)$ for a given image, unlike GANs, which give no likelihood at all. This is the key trade axis in generative modeling — exact/tractable likelihood versus sample sharpness/speed — that later architectures each resolve differently.

Trace a tiny 1-D toy version of the VAE math on a single training image so every number is checkable by hand. Suppose the encoder for image $x$ outputs $\mu_\phi(x) = 0.6$, $\log\sigma^2_\phi(x) = -0.5$ (so $\sigma_\phi(x) = e^{-0.25} \approx 0.7788$), and the prior is standard normal, $p(z) = \mathcal{N}(0,1)$.

Step 1 — KL term (closed form for two Gaussians):

Step 2 — reparameterized sample. Draw $\epsilon = 0.30$ (fixed for reproducibility of this trace):

Step 3 — decode. Suppose the (linear, toy) decoder is $\hat{x} = \tanh(1.2z) = \tanh(1.0003) \approx 0.7617$, and the true pixel value (normalized to $[-1,1]$) is $x = 0.80$.

Step 4 — reconstruction loss (mean squared error as a stand-in for Gaussian NLL):

Step 5 — total loss (negative ELBO) and gradient direction.

Here the reconstruction is already good, but the KL term dominates: the encoder's posterior is fairly confident ($\sigma \approx 0.78$, close to the prior's $\sigma=1$) but its mean $\mu = 0.6$ is noticeably off-center from the prior's $\mu=0$, and that mismatch is exactly what the KL penalizes. Backpropagating this loss pushes $\mu_\phi(x)$ slightly toward $0$ (reducing KL) while also adjusting decoder weights to keep $\hat{x}$ near $0.80$ (reducing reconstruction error) — the two loss terms compete, which is precisely the regularization/reconstruction tension that keeps the latent space smooth and sample-able rather than a lookup table of memorized codes.

At sampling time, after training, we discard the encoder entirely: draw $z \sim \mathcal{N}(0,1)$ fresh (say $z = -1.2$), decode $\hat{x} = \tanh(1.2 \times -1.2) = \tanh(-1.44) \approx -0.894$, and that is a brand-new synthetic pixel value produced with no reference image at all — the whole point of the exercise.

The mathematical machinery here — variational inference, the ELBO, and its use for training deep generative models with the reparameterization trick — was assembled from decades of Bayesian statistics but crystallized into the modern deep-learning form in the early 2010s.

Autoencoder-style latent bottlenecks are the backbone of modern image compression research, anomaly detection (an image that reconstructs poorly is likely out-of-distribution), and representation learning, where the latent code $z$ serves as a compact, semantically organized feature for downstream tasks. VAEs specifically power controllable image editing tools, where interpolating in latent space produces smooth morphs between images, and are used inside larger pipelines — notably as the compression stage of latent diffusion models, where a VAE encoder first shrinks an image into a smaller latent grid so the (much more expensive) diffusion process runs there instead of on raw pixels. Autoregressive pixel models inform speech synthesis (WaveNet) and are the direct conceptual ancestor of modern autoregressive text-to-image and token-based image generators, which model images as sequences of discrete visual tokens. Every large-scale text-to-image system in production today — used in design tools, game asset pipelines, synthetic data generation for robotics and autonomous driving, and creative software — traces its core sampling logic back to this same "sample $z$, decode to $x$" or "condition on pixels-so-far" template introduced here, elaborated on by Generative Adversarial Networks for Images (which replace explicit likelihood with an adversarial discriminator) and Diffusion Models for Images (which replace one-shot decoding with many small iterative denoising steps).

Ask the tutor: How does the ELBO objective change when the decoder likelihood $p_\theta(x\mid z)$ is modeled as a mixture distribution instead of a single Gaussian, and why might that fix the blurriness problem?