Score Matching and Score-Based Generative Models

Score matching is a statistical estimation technique introduced by Aapo Hyvärinen (2005) that enables learning of unnormalized probability models by matching the gradient of the log-density (the score function) rather than the density itself, entirely bypassing intractable partition function computation. Score-based generative models, culminating in the stochastic differential equation (SDE) framework of Song et al. (2020), leverage this principle to define a continuous-time diffusion process that transforms data into noise and learns its time-reversed counterpart for generation. Remarkably, these models are mathematically equivalent to denoising diffusion probabilistic models (DDPMs) yet derive from a completely different theoretical lineage — estimation theory rather than variational inference.

The core insight

The key insight is that you can train a generative model without ever evaluating a normalized probability: instead of learning $p(\mathbf{x})$ , learn $\nabla_{\mathbf{x}} \log p(\mathbf{x})$ — the direction of steepest ascent in log-probability space. This score function is defined even when the normalizing constant is intractable, and once you have it, you can generate samples by simulating Langevin dynamics, which only requires scores. The "aha moment" is that generation is not about memorizing probability mass — it is about learning a vector field that points toward high-density regions, and you can learn this field purely from data using a denoising objective that has no free parameters requiring normalization.

Common trapCommon Misconception

Practitioners frequently conflate the denoising score matching (DSM) objective with the original score matching (SM) objective and assume they are merely implementation variants of the same thing — they are not. Original score matching (Hyvärinen 2005) requires computing the trace of the Jacobian $\text{tr}(\nabla_{\mathbf{x}}^2 \log p_\theta)$ , which is $O(d^2)$ and completely infeasible in high dimensions. Denoising score matching (Vincent 2011) replaces this with a noise-conditional regression objective that is $O(d)$ and scales to images, but it implicitly learns scores of noise-perturbed distributions, not the original data distribution — a distinction that has deep implications for the noise schedule design in diffusion models. The correct mental model: DSM is not an approximation to SM; it is a fundamentally different (and in practice superior) objective that targets a family of smoothed densities indexed by noise level.

How It Works

Score Functions and Fisher Divergence

Let $p_{\text{data}}(\mathbf{x})$ be the unknown data distribution and $p_\theta(\mathbf{x})$ a model defined up to a normalizing constant: $p_\theta(\mathbf{x}) = \tilde{p}_\theta(\mathbf{x}) / Z(\theta)$ . The score function of any distribution $q$ is defined as:

\quad \mathbf{s}_\theta(\mathbf{x}) \;=\; \nabla_{\mathbf{x}} \log p_\theta(\mathbf{x}) \;=\; \nabla_{\mathbf{x}} \log \tilde{p}_\theta(\mathbf{x})

Explanation: Because $Z(\theta)$ does not depend on $\mathbf{x}$ , its gradient vanishes, so the score of the unnormalized model equals the score of the normalized model — the partition function drops out entirely. Here $\mathbf{s}_\theta : \mathbb{R}^d \to \mathbb{R}^d$ is a vector field. The natural objective is the Fisher divergence:

\quad J_{\text{SM}}(\theta) \;=\; \frac{1}{2}\,\mathbb{E}_{p_{\text{data}}}\!\left[\,\bigl\|\nabla_{\mathbf{x}} \log p_\theta(\mathbf{x}) - \nabla_{\mathbf{x}} \log p_{\text{data}}(\mathbf{x})\bigr\|^2\right]

Explanation: This measures the average squared distance between the model's score field and the data's score field, analogous to a squared loss in function space over the data manifold.

Hyvärinen's Integration-by-Parts Trick

Since $\nabla_{\mathbf{x}} \log p_{\text{data}}$ is unknown, Hyvärinen (2005) showed via integration by parts (assuming boundary terms vanish) that $J_{\text{SM}}$ equals the explicit score matching objective:

\quad J_{\text{ESM}}(\theta) \;=\; \mathbb{E}_{p_{\text{data}}}\!\left[\,\text{tr}\!\left(\nabla_{\mathbf{x}} \mathbf{s}_\theta(\mathbf{x})\right) + \frac{1}{2}\,\bigl\|\mathbf{s}_\theta(\mathbf{x})\bigr\|^2\right] + \text{const}

Explanation: The trace term $\text{tr}(\nabla_{\mathbf{x}} \mathbf{s}_\theta)$ is the sum of all second-order partial derivatives $\sum_i \partial s_{\theta,i}/\partial x_i$ , which penalizes score fields whose divergence differs from the data's, and equals the Fisher divergence up to a constant independent of $\theta$ . This is a complete equality, not an approximation. However, computing $\text{tr}(\nabla_{\mathbf{x}} \mathbf{s}_\theta(\mathbf{x}))$ requires $d$ backpropagation passes through the network, making it $O(d)$ in time but with a large constant — and $O(d^2)$ memory for the full Jacobian.

Denoising Score Matching

Vincent (2011) showed a cleaner equivalence. Let $q_\sigma(\tilde{\mathbf{x}} \mid \mathbf{x}) = \mathcal{N}(\tilde{\mathbf{x}}; \mathbf{x}, \sigma^2 I)$ be a Gaussian corruption kernel, and $q_\sigma(\tilde{\mathbf{x}}) = \int q_\sigma(\tilde{\mathbf{x}}\mid\mathbf{x})\,p_{\text{data}}(\mathbf{x})\,d\mathbf{x}$ the corresponding smoothed distribution. The denoising score matching objective is:

\quad J_{\text{DSM}}(\theta;\sigma) \;=\; \frac{1}{2}\,\mathbb{E}_{p_{\text{data}}(\mathbf{x})}\,\mathbb{E}_{q_\sigma(\tilde{\mathbf{x}}\mid\mathbf{x})}\!\left[\,\left\|\mathbf{s}_\theta(\tilde{\mathbf{x}}, \sigma) - \nabla_{\tilde{\mathbf{x}}} \log q_\sigma(\tilde{\mathbf{x}} \mid \mathbf{x})\right\|^2\right]

Explanation: The target score $\nabla_{\tilde{\mathbf{x}}} \log q_\sigma(\tilde{\mathbf{x}}\mid\mathbf{x}) = -(\tilde{\mathbf{x}} - \mathbf{x})/\sigma^2$ is the negative scaled noise added during corruption, so the network learns to predict the noise direction — this is why DDPM's $\epsilon$ -prediction and score matching are equivalent. This objective is minimized over the same minimizer as $J_{\text{SM}}$ with respect to $q_\sigma$ , but requires only a single forward pass per sample.

Noise-Conditional Score Network and NCSN

Song & Ermon (2019) proposed the Noise-Conditional Score Network (NCSN) by training a single network $\mathbf{s}_\theta(\mathbf{x}, \sigma)$ across a geometric sequence of noise levels $\{\sigma_i\}_{i=1}^L$ , minimizing the weighted sum:

\quad \mathcal{L}_{\text{NCSN}}(\theta) \;=\; \frac{1}{L}\sum_{i=1}^{L} \lambda(\sigma_i)\,J_{\text{DSM}}(\theta; \sigma_i), \qquad \lambda(\sigma_i) = \sigma_i^2

Explanation: Weighting by $\sigma_i^2$ normalizes the magnitude of each score target $-(\tilde{\mathbf{x}} - \mathbf{x})/\sigma_i^2$ so that all noise levels contribute comparably to training regardless of scale, and $\lambda(\sigma) = \sigma^2$ corresponds to the optimal weighting under the Fisher information of the Gaussian kernel. Generation proceeds by annealed Langevin dynamics: starting from noise, run $M$ steps of $\mathbf{x}_{t+1} = \mathbf{x}_t + \epsilon_i \mathbf{s}_\theta(\mathbf{x}_t, \sigma_i) + \sqrt{2\epsilon_i}\,\mathbf{z}_t$ for each $\sigma_i$ from large to small.

SDE Framework (Song et al. 2020)

The SDE generalization unifies all diffusion models. The forward process is an Itô SDE:

\quad d\mathbf{x} \;=\; \mathbf{f}(\mathbf{x}, t)\,dt \;+\; g(t)\,d\mathbf{w}

Explanation: Here $\mathbf{f}(\mathbf{x},t)$ is the drift coefficient (e.g., $-\frac{1}{2}\beta(t)\mathbf{x}$ for the VP-SDE), $g(t)$ is the scalar diffusion coefficient, and $\mathbf{w}$ is standard Brownian motion, so this SDE continuously transforms data $p_0$ into a tractable prior $p_T \approx \mathcal{N}(0,I)$ . By Anderson (1982), the time-reversal of this SDE is:

\quad d\mathbf{x} \;=\; \left[\mathbf{f}(\mathbf{x}, t) - g(t)^2\,\nabla_{\mathbf{x}} \log p_t(\mathbf{x})\right]dt \;+\; g(t)\,d\bar{\mathbf{w}}

Explanation: The reverse SDE has an additional score-dependent drift $-g^2 \nabla_\mathbf{x} \log p_t(\mathbf{x})$ that "undoes" the forward diffusion, and since we approximate this score with $\mathbf{s}_\theta(\mathbf{x},t)$ , a trained score network is all that is needed to run generation. The score network is trained via the continuous-time analog of DSM:

\quad \mathcal{L}_{\text{SDE}}(\theta) \;=\; \mathbb{E}_{t \sim \mathcal{U}[0,T]}\,\mathbb{E}_{p_0(\mathbf{x}_0)}\,\mathbb{E}_{p_{t|0}(\mathbf{x}_t|\mathbf{x}_0)}\!\left[\lambda(t)\,\bigl\|\mathbf{s}_\theta(\mathbf{x}_t, t) - \nabla_{\mathbf{x}_t} \log p_{t|0}(\mathbf{x}_t \mid \mathbf{x}_0)\bigr\|^2\right]

Explanation: Since $p_{t|0}(\mathbf{x}_t \mid \mathbf{x}_0)$ is a Gaussian with known mean and variance given by the SDE's transition kernel, the conditional score target is computable analytically, making this loss tractable for any noise schedule $(f, g)$ . The SDE framework encompasses VP-SDE (equivalent to DDPM), VE-SDE (equivalent to NCSN), and sub-VP-SDE as special cases with different drift/diffusion choices.

Probability Flow ODE

Every SDE in the framework has a corresponding probability flow ODE with identical marginals:

\quad d\mathbf{x} \;=\; \left[\mathbf{f}(\mathbf{x}, t) - \frac{1}{2}\,g(t)^2\,\nabla_{\mathbf{x}} \log p_t(\mathbf{x})\right]dt

Explanation: This deterministic ODE preserves the marginal distributions $p_t$ of the SDE at every $t$ while being invertible, enabling exact likelihood computation via the instantaneous change-of-variables formula (the continuous normalizing flow log-likelihood) and accelerated sampling via adaptive ODE solvers.

researcher · Aapo Hyvärinen | b. 1970 | Inventor of Score Matching

Hyvärinen is a Finnish statistician and computational neuroscientist at the Gatsby Computational Neuroscience Unit and University of Helsinki, best known for his work on Independent Component Analysis (ICA) and unsupervised learning. His 2005 Journal of Machine Learning Research paper "Estimation of Non-Normalized Statistical Models by Score Matching" introduced the integration-by-parts trick that eliminates the partition function from estimation, directly enabling the score-based generative model revolution fifteen years later. Remarkably, Hyvärinen developed score matching in the context of fitting energy-based models to neural data, with no anticipation of its eventual use in image synthesis — it was a statistical estimation paper repurposed into the foundation of state-of-the-art generative modeling.

researcher · Yang Song | b. 1994 | Score-Based Generative Models via SDEs

Yang Song is a research scientist at OpenAI whose PhD work at Stanford (advised by Stefano Ermon) produced the NCSN framework (NeurIPS 2019) and the landmark SDE paper "Score-Based Generative Modeling through Stochastic Differential Equations" (ICLR 2021 Outstanding Paper), which unified all diffusion models under a single continuous-time framework. His specific contribution was recognizing that Anderson's 1982 reverse-time SDE theorem, combined with the DSM objective trained on the transition kernels of a forward SDE, provides a complete recipe for learning to reverse arbitrary noise processes. Song was the first to demonstrate likelihood computation in diffusion models via the probability flow ODE, achieving competitive bits-per-dimension on CIFAR-10 against flow models.

researcher · Pascal Vincent | b. 1970 | Denoising Score Matching

Pascal Vincent is a professor at Université de Montréal and a member of MILA, closely associated with Yoshua Bengio's group, where he developed denoising autoencoders and the denoising score matching equivalence. His 2011 Neural Computation paper "A Connection Between Score Matching and Denoising Autoencoders" proved that training a denoising autoencoder to remove Gaussian noise is equivalent to score matching on the noise-corrupted distribution — a result that reduced the computational cost of score matching from $O(d)$ Jacobian evaluations to a single forward pass. This paper, originally framed in terms of representation learning, turned out to be the algorithmic engine behind DDPM's $\epsilon$ -prediction objective.

Figure

Score-Based Generative Model: Forward and Reverse Process

Figure

Score Function as a Vector Field

Intuition

The Mental Model

Imagine you are lost in a dense fog on a mountainous landscape, where altitude corresponds to log-probability. You cannot see the mountain (you cannot evaluate or normalize $p(\mathbf{x})$ ), but you can feel the slope under your feet at any location — that slope is the score function $\nabla_{\mathbf{x}} \log p(\mathbf{x})$ . Score matching trains a neural network to reproduce that slope everywhere in the landscape, purely from the constraint that the network's "felt slope" should match the actual terrain's slope on average over where we stand (data samples). Once the network knows the slope everywhere, we can generate new samples by starting from a random position (pure noise) and following the slope uphill — this is Langevin dynamics. In the SDE framework, the landscape itself changes over time (it gets flatter as we add noise), and the score network learns the slope of this evolving landscape at every moment. The probability flow ODE is then equivalent to a GPS route that navigates deterministically through this time-indexed family of landscapes, arriving at high-altitude regions (likely data points) without any randomness.

Historical Development

Origins · 1 minhow this was discovered

History

1985

Stuart Geman and Donald Geman introduce Gibbs sampling, demonstrating that one can sample from unnormalized distributions using local conditionals — an early hint that normalization constants are not needed for sampling.

1994

Radford Neal's thesis on Bayesian learning for neural networks introduces Hamiltonian Monte Carlo and Langevin dynamics as practical sampling algorithms requiring only score evaluations, not partition functions.

2002

Yee Whye Teh, Max Welling, and collaborators study energy-based models and the difficulty of training them via maximum likelihood due to intractable normalizing constants, motivating alternative objectives.

2005

Aapo Hyvärinen publishes "Estimation of Non-Normalized Statistical Models by Score Matching" in JMLR, proving the integration-by-parts identity that eliminates the partition function from model estimation entirely.

2010–2011

Pascal Vincent publishes "A Connection Between Score Matching and Denoising Autoencoders" in Neural Computation, establishing that DSM with Gaussian noise equals a denoising autoencoder objective — reducing score matching to a simple regression problem.

2019

Yang Song and Stefano Ermon introduce NCSN (Noise Conditional Score Networks) at NeurIPS 2019, the first generative model using annealed Langevin dynamics guided by learned noise-conditional scores, achieving competitive sample quality on CIFAR-10 and CelebA.

2020

Jonathan Ho, Ajay Jain, and Pieter Abbeel introduce DDPM (Denoising Diffusion Probabilistic Models) at NeurIPS 2020, showing