Denoising Diffusion Probabilistic Models

Denoising Diffusion Probabilistic Models (DDPMs) are a class of latent variable generative models that learn to reverse a fixed Markov chain which gradually corrupts data by adding Gaussian noise, training a neural network to approximate the reverse denoising process and thereby enabling exact log-likelihood lower bounds and high-fidelity sample generation. They matter because they achieve state-of-the-art sample quality across images, audio, and video while admitting a tractable variational objective derived from first principles, making them the dominant paradigm in modern deep generative modeling. A counterintuitive fact is that despite being trained with a simple denoising regression loss on individual timesteps, DDPMs implicitly learn the full data score function — connecting them deeply to score-based generative models through a unifying SDE framework that was only recognized post-hoc.

The core insight

The fundamental aha moment is that destroying information gradually and reversibly is far easier to learn to undo than attempting to generate data in a single forward pass. If you make the forward corruption process small enough at each step — infinitesimally adding Gaussian noise — the reverse step is also approximately Gaussian, meaning a neural network only ever needs to predict Gaussian parameters rather than arbitrary distributions. This Gaussian-reversal property converts an intractable generation problem (sampling from a complex data manifold) into a sequence of tractable denoising subproblems, each solvable by minimizing a simple mean-squared-error objective. The entire power of DDPMs rests on this observation: the posterior $q(\mathbf{x}_{t-1}|\mathbf{x}_t, \mathbf{x}_0)$ is tractable and Gaussian, so the model never has to learn to jump from pure noise to data in one shot.

Common trapCommon Misconception

The most pervasive mistake is conflating the objective the network is trained on with what the network actually predicts at inference. Practitioners often assume they must train a network to predict the denoised image $\mathbf{x}_0$ directly — and while this parameterization is valid, Ho et al. (2020) showed that the noise-prediction (ε-prediction) parameterization yields significantly better sample quality for reasons related to the implicit weighting of frequency components in the loss. The deeper error is assuming these parameterizations are equivalent in practice: $\mathbf{x}_0$ -prediction, $\epsilon$ -prediction, and $v$ -prediction (Salimans & Ho, 2022) produce identical training objectives up to a timestep-dependent reweighting, and that reweighting profoundly affects which signal frequencies the model prioritizes during learning. The correct mental model is that the choice of prediction target is a loss-weighting decision with strong inductive biases, not a mere reparameterization.

How It Works

The Forward Process

The forward (diffusion) process is a fixed Markov chain that adds Gaussian noise to data over $T$ timesteps. Given a data point $\mathbf{x}_0 \sim q(\mathbf{x})$ , the forward process is defined as:

\quad q(\mathbf{x}_{1:T} | \mathbf{x}_0) = \prod_{t=1}^{T} q(\mathbf{x}_t | \mathbf{x}_{t-1}), \quad q(\mathbf{x}_t | \mathbf{x}_{t-1}) = \mathcal{N}(\mathbf{x}_t;\, \sqrt{1-\beta_t}\,\mathbf{x}_{t-1},\, \beta_t \mathbf{I})

Explanation: The forward process is a product of Gaussian transitions, each contracting the signal by $\sqrt{1-\beta_t}$ and injecting noise with variance $\beta_t$ , so repeated application drives $\mathbf{x}_T$ toward a standard normal. Here $\beta_t \in (0,1)$ is a variance schedule (fixed hyperparameter, often linear or cosine). Crucially, the forward process admits a closed-form marginal at any timestep $t$ via the reparameterization trick. Define $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s$ . Then:

\quad q(\mathbf{x}_t | \mathbf{x}_0) = \mathcal{N}\!\left(\mathbf{x}_t;\, \sqrt{\bar{\alpha}_t}\,\mathbf{x}_0,\, (1-\bar{\alpha}_t)\mathbf{I}\right), \quad \text{equivalently } \mathbf{x}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon}, \quad \boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})

Explanation: The marginal at time $t$ is a linear interpolation in signal-to-noise ratio space between the clean data $\mathbf{x}_0$ and pure Gaussian noise $\boldsymbol{\epsilon}$ , with $\bar{\alpha}_t$ controlling the mix — this equation enables sampling $\mathbf{x}_t$ in a single step without simulating the full chain. The forward posterior conditioned on $\mathbf{x}_0$ is also Gaussian (derived via Bayes' rule):

\quad q(\mathbf{x}_{t-1} | \mathbf{x}_t, \mathbf{x}_0) = \mathcal{N}\!\left(\mathbf{x}_{t-1};\, \tilde{\boldsymbol{\mu}}_t(\mathbf{x}_t, \mathbf{x}_0),\, \tilde{\beta}_t \mathbf{I}\right)

\tilde{\boldsymbol{\mu}}_t = \frac{\sqrt{\bar{\alpha}_{t-1}}\beta_t}{1 - \bar{\alpha}_t}\mathbf{x}_0 + \frac{\sqrt{\alpha_t}(1-\bar{\alpha}_{t-1})}{1-\bar{\alpha}_t}\mathbf{x}_t, \qquad \tilde{\beta}_t = \frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_t}\beta_t

Explanation: This tractable Gaussian posterior — which exists only because both the forward step and the marginal are Gaussian — is the analytical target the learned reverse process must match, and its existence is why the ELBO decomposes into KL divergences between Gaussians with closed-form solutions.

The Reverse Process and Variational Objective

The reverse generative process is parameterized as a learned Markov chain with the same functional form:

\quad p_\theta(\mathbf{x}_{0:T}) = p(\mathbf{x}_T) \prod_{t=1}^{T} p_\theta(\mathbf{x}_{t-1} | \mathbf{x}_t), \quad p_\theta(\mathbf{x}_{t-1} | \mathbf{x}_t) = \mathcal{N}\!\left(\mathbf{x}_{t-1};\, \boldsymbol{\mu}_\theta(\mathbf{x}_t, t),\, \sigma_t^2 \mathbf{I}\right)

Explanation: The reverse process starts from the prior $p(\mathbf{x}_T) = \mathcal{N}(\mathbf{0}, \mathbf{I})$ and applies learned Gaussian denoising steps, with $\boldsymbol{\mu}_\theta$ implemented by a neural network conditioned on both the noisy state and the current timestep. The evidence lower bound (ELBO) decomposes as:

\quad \mathcal{L} = \mathbb{E}_q\!\left[-\log p_\theta(\mathbf{x}_0 | \mathbf{x}_1) + \sum_{t=2}^{T} D_{\mathrm{KL}}\!\left(q(\mathbf{x}_{t-1}|\mathbf{x}_t,\mathbf{x}_0) \;\big\|\; p_\theta(\mathbf{x}_{t-1}|\mathbf{x}_t)\right) + D_{\mathrm{KL}}\!\left(q(\mathbf{x}_T|\mathbf{x}_0) \;\big\|\; p(\mathbf{x}_T)\right)\right]

Explanation: The ELBO splits into a reconstruction term, $T-1$ denoising matching terms (each a KL between Gaussians with a closed-form solution), and a prior matching term that is typically treated as constant since $q(\mathbf{x}_T|\mathbf{x}_0) \approx \mathcal{N}(\mathbf{0},\mathbf{I})$ for large $T$ .

The Simplified Training Objective

Ho et al. (2020) showed that using the $\epsilon$ -parameterization $\boldsymbol{\mu}_\theta(\mathbf{x}_t, t) = \frac{1}{\sqrt{\alpha_t}}\!\left(\mathbf{x}_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)\right)$ and dropping timestep-dependent loss weighting yields the simplified objective:

\quad \mathcal{L}_{\mathrm{simple}} = \mathbb{E}_{t \sim \mathcal{U}[1,T],\, \mathbf{x}_0 \sim q,\, \boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0},\mathbf{I})}\!\left[\left\|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta\!\left(\sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\boldsymbol{\epsilon},\, t\right)\right\|^2\right]

Explanation: The entire training procedure reduces to sampling a random timestep, corrupting $\mathbf{x}_0$ with the corresponding noise level, and training the network $\boldsymbol{\epsilon}_\theta$ to predict the noise that was added — a remarkably simple denoising regression loss that implicitly minimizes the ELBO.

researcher · Jonathan Ho | b. 1993 | ε-parameterization and simplified DDPM objective

Jonathan Ho completed his PhD at UC Berkeley under Pieter Abbeel and later joined Google Brain, where he led the team that produced the 2020 NeurIPS paper "Denoising Diffusion Probabilistic Models" — the work that transformed diffusion models from a theoretical curiosity into a practical generative modeling paradigm. His key contribution was the observation that the ε-prediction parameterization with a simplified, reweighted ELBO dramatically outperforms the full variational lower bound in sample quality, despite (or because of) discarding the theoretically motivated loss weighting. A surprising personal note: the DDPM paper was initially rejected from an earlier workshop on the grounds that the samples were not competitive with GANs, yet within two years the framework it introduced had displaced GANs as the dominant generative paradigm.

researcher · Jascha Sohl-Dickstein | b. 1980 | Originating diffusion generative models

Jascha Sohl-Dickstein, a researcher at Google Brain with a background in computational neuroscience and statistical physics, originated the use of non-equilibrium thermodynamics as the theoretical basis for generative modeling in his 2015 ICML paper "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" — the paper that introduced the forward-reverse diffusion framework five years before DDPMs made it practical. His foundational insight was that a system driven to thermal equilibrium (Gaussian noise) by a diffusion process can be reversed to generate samples if the reverse process is learned, drawing a formal analogy between generative modeling and the second law of thermodynamics. Sohl-Dickstein studied under Bruno Olshausen and has broad cross-disciplinary interests, including work on meta-learning and neural network loss landscapes that is largely unrelated to diffusion.

researcher · Yang Song | b. 1994 | Score-based generative models and SDE unification

Yang Song completed his PhD at Stanford under Stefano Ermon and is now a researcher at OpenAI, best known for developing score-based generative models through Langevin dynamics and for the landmark 2021 ICLR paper "Score-Based Generative Modeling through Stochastic Differential Equations" which proved that DDPMs and NCSN score models are special cases of a continuous-time SDE framework. His specific contribution to this topic is the proof that the reverse-time SDE of any forward diffusion process can be written in terms of the score $\nabla_{\mathbf{x}} \log p_t(\mathbf{x})$ , unifying discrete-time DDPMs, continuous normalizing flows, and score matching under one mathematical roof. A counterintuitive aspect of his work is that deterministic probability flow ODEs derived from the same score function generate samples of identical distributional quality to stochastic SDEs, enabling exact likelihood computation without sacrificing generation fidelity.

Figure

DDPM Forward and Reverse Process

Figure

DDPM Training vs Sampling Algorithm