Diffusion Models for Images

A diffusion model is a generative model built from two coupled Markov chains over a sequence of latent variables $x_0, x_1, \dots, x_T$, where $x_0$ is a real image and $x_T$ is (approximately) pure Gaussian noise. The forward diffusion process $q(x_t \mid x_{t-1})$ is fixed, not learned: it adds a small amount of Gaussian noise at every step according to a variance schedule $\beta_1, \dots, \beta_T$. Because each step only adds noise, this process has a closed form that lets us jump directly from $x_0$ to any $x_t$ without simulating every intermediate step — a critical efficiency property exploited during training. The reverse diffusion process $p_\theta(x_{t-1} \mid x_t)$ is what we actually learn. It is parameterized as a Gaussian whose mean is predicted by a neural network, typically a Convolutional Neural Network-based U-Net augmented with self-attention and a timestep embedding. Rather than predicting the mean directly, the standard parameterization (from Ho, Jain, and Abbeel's Denoising Diffusion Probabilistic Models, DDPM) trains the network $\epsilon_\theta(x_t, t)$ to predict the noise $\epsilon$ that was added to produce $x_t$ from $x_0$. This reframing turns a complex generative modeling problem into a simple supervised denoising regression, closely related to Image Generation Fundamentals ideas of learning data distributions through tractable surrogate objectives. Two properties make the framework tractable. First, the forward process has a fixed, non-learned form, so we never need to infer it — we simply construct noisy training pairs on the fly. Second, because both the forward and idealized reverse transitions are Gaussian for small step sizes, the reverse process can be well-approximated by a Gaussian too, which is what justifies parameterizing $p_\theta$ as Gaussian in the first place. This connects diffusion models to Probability distributions machinery: everything is built from conditional Gaussians chained together, with the entire model trained by maximizing a variational lower bound on the data likelihood — the same evidence lower bound (ELBO) logic underlying variational autoencoders, but applied across a much longer, smoother chain of latent variables.

Forward process. Define a variance schedule $\beta_1 < \beta_2 < \cdots < \beta_T$, typically small ($\beta_1 \approx 10^{-4}$) at the start and larger ($\beta_T \approx 0.02$) near the end, over $T \approx 1000$ steps. The forward transition is

Let $\alpha_t = 1 - \beta_t$ and $\bar\alpha_t = \prod_{s=1}^t \alpha_s$. Because the composition of Gaussians is Gaussian, one can show by induction that

which lets us sample any $x_t$ in one step:

As $t \to T$, $\bar\alpha_t \to 0$, so $x_T$ converges to pure standard Gaussian noise regardless of $x_0$.

Reverse process. We want $p_\theta(x_{t-1} \mid x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \Sigma_t)$. Using Bayes' rule, the true (but intractable to sample without knowing $x_0$) posterior $q(x_{t-1} \mid x_t, x_0)$ is itself Gaussian with a known mean that depends linearly on $x_0$ and $x_t$. Substituting the noise-based expression for $x_0$ in terms of $x_t$ and $\epsilon$ yields a mean that depends only on $x_t$, $t$, and $\epsilon$ — exactly what the network predicts. The result, after Ho et al.'s simplification, is:

The variance $\Sigma_t$ is usually fixed to $\beta_t I$ or a related closed-form value, though later work (e.g., improved DDPM) learns it too.

Training. Sample a real image $x_0$, a random timestep $t$, and Gaussian noise $\epsilon$; construct $x_t$ in closed form; train the network to predict $\epsilon$ from $(x_t, t)$ using simple mean-squared error. This is the "simple" DDPM loss, a tight and empirically effective surrogate for the full variational bound. ```

# TRAINING function TRAIN(dataset, T, beta_schedule, network eps_theta): precompute alpha_t, alpha_bar_t for t = 1..T from beta_schedule repeat until convergence: x0 <- sample a real image from dataset t <- sample uniformly from {1, ..., T} eps <- sample from Normal(0, I), same shape as x0 xt <- sqrt(alpha_bar_t[t]) x0 + sqrt(1 - alpha_bar_t[t]) eps eps_pred <- eps_theta(xt, t) loss <- MeanSquaredError(eps, eps_pred) update network parameters theta by gradient descent on loss return eps_theta

# SAMPLING (ancestral sampling, DDPM) function SAMPLE(T, beta_schedule, eps_theta): x_T <- sample from Normal(0, I) for t = T down to 1: z <- sample from Normal(0, I) if t > 1 else 0 eps_pred <- eps_theta(x_t, t) mu <- (1 / sqrt(alpha_t[t])) (x_t - (beta_t[t] / sqrt(1 - alpha_bar_t[t])) eps_pred) sigma_t <- sqrt(beta_t[t]) # or a learned/tuned variance x_{t-1} <- mu + sigma_t * z return x_0 ``` The timestep $t$ is fed to the network via a sinusoidal or learned embedding, analogous to positional encodings in transformers, so a single network handles all noise levels. The U-Net's multi-scale convolutional structure with skip connections lets it reason about both fine texture and global structure at every noise level.

Training time and space. Each training step performs one forward and one backward pass of the U-Net, so per-step cost is $O(C)$ where $C$ is the cost of a single network evaluation — dominated by the convolutional and attention layers, typically $O(HW \cdot k^2 \cdot c_{in} c_{out})$ per convolutional layer for images of resolution $H \times W$, plus $O((HW)^2)$ for any full self-attention layers at low resolution. Because the closed-form shortcut $x_t = \sqrt{\bar\alpha_t}x_0 + \sqrt{1-\bar\alpha_t}\epsilon$ lets us sample any $t$ directly, training needs exactly one network evaluation per gradient step, independent of $T$ — a crucial efficiency win over naively simulating the whole chain. Memory is $O(\text{network parameters} + \text{batch} \times HW \times \text{channels})$ for activations, same as training any comparable image CNN.

Sampling time. Naive ancestral sampling (the DDPM sampler above) requires one full network evaluation per timestep, so generating one image costs $O(T \cdot C)$ — with $T$ often 1000, this is the well-known weakness of diffusion models relative to GANs' single forward pass. Deterministic or higher-order samplers (DDIM, DPM-Solver) exploit the fact that the reverse process can be viewed as integrating an ordinary differential equation, allowing accurate sampling in $T' \ll T$ steps (often 20–50), trading a small amount of quality for a $20$–$50\times$ speedup. Space at sampling time is $O(HW \times \text{channels})$ for the single image being iteratively refined, plus the fixed network parameter memory.

Correctness argument. The training objective is derived, not merely asserted, from a variational bound on the log-likelihood of the data. Writing $p_\theta(x_0) = \int p_\theta(x_{0:T})\, dx_{1:T}$ and applying Jensen's inequality with the fixed forward process $q$ as the variational posterior gives:

Because both $q$ and $p_\theta$ factor as products of Gaussian transitions across the chain, $L_{VLB}$ decomposes into a sum of KL divergences between Gaussians at each timestep, each of which has a closed-form expression. Ho et al. show that reparameterizing the mean prediction as a noise prediction, and dropping certain reweighting constants, produces the simple objective:

which upper-bounds (up to reweighting) the true negative log-likelihood. Minimizing this loss therefore directly improves a bound on how well the model explains real image data — this is the formal justification that training actually pushes $p_\theta$ toward the true data distribution, not merely a heuristic. As $T \to \infty$ with correspondingly small steps, the discrete chain converges to a continuous-time diffusion process (a stochastic differential equation), and the reverse process is provably the time-reversal of that diffusion under mild regularity conditions — this is the theoretical bedrock (Anderson's theorem on time-reversed diffusions) beneath the entire framework.

Tradeoffs. The dominant cost is the many-step sampling loop; likelihood estimates from diffusion models are usually competitive but not always state-of-the-art compared to autoregressive models; and the fixed, non-learned forward process, while mathematically convenient, is not necessarily the most efficient possible noising schedule — an active area of follow-up research (learned schedules, flow matching, consistency models) aims squarely at this bottleneck.

Consider a toy one-dimensional "image" — a single pixel intensity $x_0 = 0.8$ (on a $[-1, 1]$ scale) — diffused with a schedule of $T = 4$ steps and $\beta_t = 0.1, 0.2, 0.3, 0.4$. We trace the forward process and one denoising update with an idealized, already-trained network.

Forward pass, drawing one fixed noise sample $\epsilon = 0.5$ at each $t$ for illustration (in practice a fresh $\epsilon$ is drawn per sample; here we hold it fixed to make the arithmetic traceable):

• $x_1 = 0.949(0.8) + 0.316(0.5) = 0.759 + 0.158 = 0.917$
• $x_2 = 0.849(0.8) + 0.529(0.5) = 0.679 + 0.265 = 0.944$
• $x_3 = 0.710(0.8) + 0.704(0.5) = 0.568 + 0.352 = 0.920$
• $x_4 = 0.550(0.8) + 0.836(0.5) = 0.440 + 0.418 = 0.858$

Notice $x_4$ is not simply "more noise piled on $x_3$" — it is recomputed directly from $x_0$ using the closed-form shortcut, exactly as the training loop does; this is why we can train on any $t$ without simulating $t-1$ intermediate steps first.

Reverse pass at $t = 4$. Suppose the trained network, given $(x_4, 4) = (0.858, 4)$, correctly predicts $\hat\epsilon = \epsilon_\theta(x_4, 4) \approx 0.5$ (matching the true noise used above, since we assumed an idealized network). Then:

Sampling noise $z \sim \mathcal{N}(0,1)$, say $z = -0.1$, and $\sigma_4 = \sqrt{0.4} = 0.632$:

Compare to the true forward $x_3 = 0.920$: the discrepancy is expected because $z$ injects fresh stochastic variation (diffusion sampling is not deterministic reconstruction, it is a stochastic reverse walk), and because our toy network and schedule are illustrative, not trained on real data. Repeating this update from $t=4$ down to $t=1$ eventually yields a reconstructed $x_0$ estimate; in a real high-resolution image, this same arithmetic is applied per-pixel-channel, simultaneously, at every spatial location, with the network's convolutional structure coupling neighboring pixels so that the recovered image is spatially coherent rather than independent per-pixel noise removal.

Diffusion models draw on a much older idea in statistical physics: nonequilibrium thermodynamics describes how a system relaxes from an ordered state toward disorder, and this relaxation can in principle be reversed given the right dynamics. Sohl-Dickstein and collaborators first proposed applying this physical intuition to generative modeling of images in 2015, framing generation as learning to reverse a diffusion process. The idea lay relatively dormant until Jonathan Ho, Ajay Jain, and Pieter Abbeel's 2020 Denoising Diffusion Probabilistic Models paper showed that a simplified noise-prediction objective could match or beat GANs on image sample quality, triggering rapid adoption across the field.

Diffusion models now underpin most leading image-generation systems in production. Stable Diffusion runs the diffusion process in a compressed latent space (learned by an autoencoder) rather than raw pixels, dramatically cutting compute while preserving fidelity — a direct extension of the mechanism described here. DALL·E 2 and Imagen condition the reverse process on text embeddings, often derived from CLIP, so that the denoising network is steered toward images matching a caption; this is the natural next step beyond this article, covered under Text-to-Image with CLIP and Latent Diffusion. Diffusion is also used for image inpainting, super-resolution, and image-to-image translation, since the same noise-removal machinery can be conditioned on a partially masked or low-resolution input. Beyond static images, diffusion techniques extend to video generation, 3D shape and scene synthesis (score distillation for neural fields), audio synthesis, and even protein structure generation, all reusing the same forward-noising/reverse-denoising skeleton. Because the core objective is a stable regression loss, diffusion models have proven far easier to scale reliably to billions of parameters and training images than adversarial alternatives, which is a major reason they now dominate commercial generative image products.

Ask the tutor: Can you walk me through how classifier-free guidance changes the sampling loop's math, step by step?