Denoising and Score-Based Models

Let $p(x)$ be an unknown data distribution over $\mathbb{R}^d$ (e.g. natural images flattened into vectors). Its score function is defined as

This is a vector field: at every point $x$ in space it tells you the direction of steepest ascent of the log-density. Crucially, the score does not depend on the normalizing constant $Z = \int \tilde p(x)\, dx$ of an unnormalized density $\tilde p(x) = Z \cdot p(x)$, since $\nabla_x \log \tilde p(x) = \nabla_x \log p(x) + \nabla_x \log Z = \nabla_x \log p(x)$ (the constant's gradient is zero). This is what makes the score attractive: many models can output an unnormalized energy or likelihood easily, but normalizing it (computing $Z$) is intractable in high dimensions. Estimating the score avoids that computation entirely.

Score matching is the general name for training a parametric vector field $s_\theta(x)$ to approximate $s(x) = \nabla_x \log p(x)$, typically by minimizing the expected squared error

The apparent problem is that the right-hand side needs the true score, which we don't have. Hyvärinen's original score matching (2005) showed this objective is equal, up to a constant, to a quantity involving only $s_\theta$ and its Jacobian trace — computable from samples alone, no density evaluation needed. However, that formulation requires computing $\mathrm{tr}(\nabla_x s_\theta(x))$, which is expensive for large neural networks (it costs $O(d)$ backward passes in the naive case).

Denoising score matching (DSM), introduced by Vincent (2011), gives a far cheaper alternative. Instead of matching the score of the clean data distribution directly, DSM perturbs each data point $x$ with Gaussian noise, $\tilde x = x + \sigma \epsilon$, $\epsilon \sim \mathcal N(0, I)$, and matches the score of the noised distribution $q_\sigma(\tilde x) = \int p(x)\, \mathcal N(\tilde x; x, \sigma^2 I)\, dx$. The remarkable fact — proved below — is that this noised score has a closed form conditional on $x$, turning score estimation into a simple regression: predict the (scaled) noise from the corrupted sample.

Step 1 — The denoising score matching identity. For Gaussian corruption $\tilde x = x + \sigma\epsilon$, the conditional density is $q_\sigma(\tilde x \mid x) = \mathcal N(\tilde x; x, \sigma^2 I)$, so

Vincent's theorem states that minimizing

has the exact same minimizer $s_\theta$ as matching the true marginal score $\nabla_{\tilde x} \log q_\sigma(\tilde x)$ of the noised data distribution. In other words, training a network to predict the (negative, scaled) noise from a noisy sample is identical, in expectation over data, to training it to output $\nabla_{\tilde x}\log q_\sigma(\tilde x)$. This is the bridge between "denoising autoencoders" and "score estimators."

Step 2 — Reparametrize as noise prediction. Networks are usually trained to predict $\epsilon$ directly via $\epsilon_\theta(\tilde x, \sigma)$, related to the score by

The training loss becomes the familiar denoising objective

which is exactly the loss used to train DDPM-style diffusion models — confirming that "predict the noise" training is score matching in disguise.

Step 3 — Multiple noise scales. A single $\sigma$ is not enough: at low noise, the score is only accurate near the (thin) data manifold, so a sampler starting from random noise (a low-density region) gets an unreliable gradient signal. The fix, due to Song & Ermon (2019), is to train one network conditioned on a range of noise levels $\sigma_1 > \sigma_2 > \dots > \sigma_L$ (large to small), giving a noise-conditional score network (NCSN) $s_\theta(x, \sigma_i)$. The training objective sums weighted DSM losses across levels:

(the $\sigma_i^2$ weighting balances the loss scale across noise levels, since raw DSM loss magnitudes scale like $1/\sigma_i^2$).

Step 4 — Langevin dynamics for sampling. Given a good score estimate, sampling from $p(x)$ uses Langevin dynamics: starting from $x_0$ (e.g. uniform or Gaussian noise), iterate

for small step size $\eta > 0$. As $\eta \to 0$ and $t \to \infty$, the distribution of $x_t$ converges to $p(x)$ regardless of the starting point — this is a classical result from stochastic differential equations (Langevin dynamics is the Euler–Maruyama discretization of a diffusion process whose stationary distribution is exactly $p(x)$).

Step 5 — Annealed Langevin dynamics. To handle the multiple noise scales, sampling starts with a large $\sigma_1$ (score is smooth and reliable everywhere, since noise has spread density over the whole space) and gradually anneals down to $\sigma_L \approx 0$ (score is sharp, matching the true data score), running a few Langevin steps at each level and using the previous level's output as the next level's initialization. ```text ALGORITHM: Train Noise-Conditional Score Network Input: dataset D, noise scales sigma_1 > sigma_2 > ... > sigma_L, network s_theta Output: trained parameters theta repeat until convergence: sample minibatch {x_1, ..., x_B} from D for each x_b in minibatch: sample i ~ Uniform{1, ..., L} # random noise level sample epsilon ~ Normal(0, I) x_tilde = x_b + sigma_i epsilon # corrupt target = -epsilon / sigma_i # true score of noised conditional pred = s_theta(x_tilde, sigma_i) loss_b = sigma_i^2 || pred - target ||^2 # weighted DSM loss theta = theta - lr * gradient( mean(loss_b), theta ) return theta

ALGORITHM: Annealed Langevin Dynamics (Sampling) Input: trained s_theta, noise scales sigma_1 > ... > sigma_L, step size epsilon_step, steps-per-level T Output: sample x approximately from data distribution p(x) x = sample from simple prior (e.g. uniform noise) for i = 1 to L: alpha_i = epsilon_step (sigma_i / sigma_L)^2 # step size scaled per level for t = 1 to T: z = sample Normal(0, I) x = x + (alpha_i / 2) s_theta(x, sigma_i) + sqrt(alpha_i) * z return x ``` The figure below shows the overall pipeline: a clean sample is corrupted at several noise scales, a shared network learns to output the corresponding score at each scale, and annealed Langevin sampling reverses the process from noise back to data.

Training complexity. Each training step requires one forward and backward pass of the score network per minibatch element, exactly like standard supervised regression. If the network has $P$ parameters and processes a batch of size $B$, the cost per step is $O(B \cdot C_\text{net})$ where $C_\text{net}$ is the cost of one forward/backward pass (dominated by the network architecture, e.g. a U-Net's convolution cost). Crucially, DSM avoids the $O(d)$-times-more-expensive Jacobian-trace term required by vanilla score matching — this is the entire practical reason DSM (and its noise-prediction reparametrization) became the standard training recipe rather than Hyvärinen's original objective.

Sampling complexity. Annealed Langevin dynamics runs $L$ noise levels with $T$ steps each, so generating one sample costs $O(L \cdot T)$ network evaluations, each $O(C_\text{net})$. This is markedly more expensive than a single forward pass of a GAN or VAE decoder — the classic accuracy/speed tradeoff of iterative generative samplers, shared with diffusion models generally. Typical values are $L \sim 10$–$1000$ noise levels and $T \sim 1$–$100$ steps, so total sampling steps range from the low hundreds to tens of thousands depending on the sampler and schedule.

Correctness of denoising score matching (proof sketch). We show $J_\text{DSM}$ and the marginal score-matching objective share the same minimizer. Expand the marginal objective for the noised distribution $q_\sigma(\tilde x) = \int p(x) q_\sigma(\tilde x|x)\,dx$:

Expand the square; the cross term is $-\mathbb E_{q_\sigma(\tilde x)}\big[s_\theta(\tilde x)^\top \nabla_{\tilde x}\log q_\sigma(\tilde x)\big] = -\int s_\theta(\tilde x)^\top \nabla_{\tilde x} q_\sigma(\tilde x)\, d\tilde x$ (using $\nabla \log q = \nabla q / q$). Writing $q_\sigma(\tilde x) = \int p(x) q_\sigma(\tilde x|x)\,dx$ and swapping the order of integration and differentiation (justified for smooth Gaussian kernels),

Substituting back shows the cross term equals $-\mathbb E_{p(x), q_\sigma(\tilde x|x)}\big[s_\theta(\tilde x)^\top \nabla_{\tilde x}\log q_\sigma(\tilde x|x)\big]$, i.e. the marginal score can be replaced by the conditional score $\nabla_{\tilde x}\log q_\sigma(\tilde x|x) = -\epsilon/\sigma$ inside the expectation, without changing the cross term. The $\|s_\theta(\tilde x)\|^2$ term is unaffected by this substitution (it doesn't involve the true score at all), and the $\|\nabla \log q_\sigma\|^2$ term is a constant independent of $\theta$. Hence

so the two objectives have identical gradients with respect to $\theta$ and identical minimizers. This is the precise statement of Vincent's theorem: denoising is provably equivalent to score matching, not merely a heuristic approximation.

Correctness of Langevin sampling. The stochastic process $dx_t = \tfrac12 \nabla_x \log p(x_t)\, dt + dW_t$ (a Langevin diffusion, $W_t$ a Wiener process) has $p(x)$ as its unique stationary distribution — this follows from the Fokker–Planck equation: the density evolution $\partial_t \rho_t = -\nabla\cdot(\rho_t \nabla \log p) + \tfrac12 \Delta \rho_t$ vanishes exactly when $\rho_t = p$ (direct substitution, using $\nabla \log p = \nabla p / p$). The discrete-time update in the algorithm is the Euler–Maruyama discretization of this SDE; as step size $\eta \to 0$ and iteration count $\to \infty$, the discretized chain's distribution converges to $p(x)$ under mild regularity (smoothness, tail conditions on $p$). In practice $\eta$ is finite, introducing $O(\eta)$ discretization bias, and mixing time can be slow when $p$ has well-separated modes — precisely the failure mode that motivates using many noise scales: at large $\sigma$, the noised distribution $q_\sigma$ has merged, well-connected modes (since convolving with a wide Gaussian smooths out the density), making Langevin mixing fast, and progressively smaller $\sigma$ refines the sample without needing to cross low-density valleys from scratch.

Honest tradeoffs. Score-based/diffusion samplers trade slow, iterative sampling for training stability (no adversarial min-max as in GANs) and for tractable exact-ish likelihoods via the SDE/ODE formulation. They also need careful noise-schedule design ($\sigma_1$ large enough to smooth all modes together, $\sigma_L$ small enough to recover fine detail) — a schedule that is wrong in either direction visibly degrades sample quality.

Consider a toy 1-D dataset with two clusters: $p(x)$ is a mixture $\tfrac12 \mathcal N(-3, 0.2^2) + \tfrac12 \mathcal N(3, 0.2^2)$. Suppose we have (for illustration) the exact score $s(x) = \nabla_x \log p(x)$ and want to see how one Langevin step behaves at two different noise scales. At $\sigma = 0$ (no smoothing, i.e. the raw data score) and starting at $x_0 = 0$ — exactly between the two modes — the true density is nearly flat and symmetric there, so $s(0) \approx 0$: the sampler receives almost no directional signal and must rely entirely on injected noise $z_t$ to randomly drift toward one mode. This is slow and can get stuck bouncing near $x=0$ for a long time if $\eta$ is small. Now consider training at a large noise scale, $\sigma_1 = 3$. The noised distribution $q_{\sigma_1}(\tilde x)$ convolves the two narrow Gaussians with a wide $\mathcal N(0, 9)$ kernel, producing a single broad, unimodal bump centered near $0$. Its score at $\tilde x = 0$ is still $0$ by symmetry, but nearby, e.g. at $\tilde x = 1$, the score is a clear, strong restoring pull because the density is now smoothly varying (no flat plateau), giving Langevin dynamics a usable gradient everywhere. Table below traces one representative sampling run (illustrative numbers, $\eta = 0.2$, single Langevin step per level shown):

The trace illustrates the essential mechanism: at large $\sigma$ the score field is smooth and provides a weak but non-degenerate signal that breaks the symmetric tie, nudging the sample toward one basin; as $\sigma$ anneals down, the score sharpens and increasingly concentrates the sample exactly onto a mode ($x \approx 3$), reproducing samples indistinguishable from the true cluster at $3$. Had we started directly at $\sigma \approx 0$, the near-zero score at $x=0$ would have left the sampler with no directional cue, illustrating why annealing is not an optimization trick but a necessity for score-based sampling in multimodal or manifold-supported data (real images live on a low-dimensional manifold, an extreme version of this same low-density problem).

Score-based generative modeling sits at the confluence of statistical estimation theory, stochastic differential equations, and deep generative modeling. Aapo Hyvärinen formalized score matching as a general estimation principle for unnormalized models, sidestepping the need to compute partition functions. Pascal Vincent later proved that a simple denoising task is mathematically equivalent to matching the score of a Gaussian-smoothed data distribution, linking score matching to the older tradition of denoising autoencoders. Yang Song and Stefano Ermon then showed that combining multiple noise scales with annealed Langevin dynamics makes score-based sampling practical on high-dimensional image data, directly foreshadowing modern diffusion samplers and their SDE-based generalization.

Score-based and denoising principles are the mathematical backbone of modern diffusion generative models used in text-to-image systems (Stable Diffusion, Imagen, DALL·E 2's diffusion decoder), where the "noise prediction" U-Net is precisely a score estimator at many noise scales, as derived above. The same machinery underlies diffusion-based audio synthesis (WaveGrid, DiffWave), molecule and protein conformation generation, and video generation models. Beyond generation, score estimates are used for out-of-distribution detection (regions of low score-consistency indicate off-manifold inputs), inverse problems such as image inpainting and super-resolution (via guided Langevin sampling with a data-consistency term), and as a theoretical tool connecting energy-based models to tractable training via denoising objectives, avoiding expensive MCMC-based maximum-likelihood training. Any system that trains a neural network to "predict the noise" is, under the hood, relying on the score-matching equivalence explained here.

Ask the tutor: How does the score-SDE framework unify discrete noise-conditional score networks and DDPM's discrete-time diffusion process into one continuous-time model?