Mode Collapse in GANs

The GAN Objective and Its Pathologies

The standard GAN minimax objective is:

Explanation: The discriminator D tries to maximise this value by correctly labelling real samples x and fake samples G(z), while the generator G tries to minimise it by fooling D into labelling fakes as real.

Symbol guide:

• $G$: generator network, maps latent code $z$ to data space
• $D$: discriminator network, outputs probability of a sample being real
• $p_{\text{data}}$: true data distribution
• $p_z$: prior over latent codes (usually $\mathcal{N}(0,I)$)
• $\mathbb{E}$: expectation operator

At the theoretical optimum, the generator matches the data distribution and $D^*(x) = \frac{1}{2}$ everywhere. But in practice, gradient descent on this objective is non-convex in the joint $(G, D)$ parameter space, and the generator can get trapped in degenerate local equilibria.

Why the Generator Prefers Collapsed Solutions

Consider the generator's gradient when the discriminator is near-optimal. The generator loss simplifies to minimising the Jensen–Shannon divergence:

Explanation: The generator is implicitly minimising the Jensen–Shannon divergence between the real and generated distributions, which is always between 0 and log 2 but has zero gradient when the supports of the two distributions do not overlap. When $p_G$ has non-overlapping support with $p_{\text{data}}$ (the typical early-training situation), JSD is constant at $\log 2$ and the generator receives zero informative gradient — it is equally wrong everywhere, so it makes no progress toward covering new modes. It instead follows any residual gradient signal toward the modes it already partially covers, reinforcing collapse.

The Wasserstein Perspective

The Wasserstein-1 (Earth Mover's) distance provides a more informative signal even for non-overlapping distributions:

Explanation: The Wasserstein distance measures the minimum "cost" of transporting mass from p_G to p_data, and unlike JSD it is continuous and differentiable almost everywhere, giving the generator a gradient even when the two distributions are completely disjoint.

Symbol guide:

• $W$: Wasserstein-1 distance
• $\sup$: supremum (maximum) over all 1-Lipschitz functions $f$
• $\|f\|_L \leq 1$: Lipschitz constraint ensuring the critic doesn't grow unbounded

This is the theoretical motivation for WGAN: by replacing the discriminator with a Lipschitz-constrained critic, the generator always has a meaningful gradient signal that pushes it toward uncovered modes rather than into collapse.

Measuring Mode Coverage: The Precision–Recall Decomposition

A rigorous diagnosis of mode collapse uses the precision–recall framework for generative models. For a threshold $k$-nearest-neighbour manifold estimate:

Explanation: Precision measures what fraction of generated samples fall inside the real data manifold (quality), while Recall measures what fraction of real data modes are covered by generated samples (diversity) — mode collapse shows up as near-zero Recall with high Precision.

Symbol guide:

• $X$: set of real data samples
• $Y$: set of generated samples
• $\text{manifold}(X)$: estimated support of distribution $X$ using $k$-NN balls
• $\mathbf{1}[\cdot]$: indicator function

A fully collapsed generator will score Precision ≈ 1 (everything it generates looks real) and Recall ≈ 0 (it covers almost none of the real modes) — the smoking gun signature of mode collapse that raw FID can miss.

Gradient Penalty for Enforcing the Lipschitz Constraint

WGAN-GP enforces the critic's Lipschitz constraint via a soft penalty:

Explanation: This penalty pushes the gradient norm of the critic to be exactly 1 at interpolated points between real and fake samples, enforcing the 1-Lipschitz condition that makes the Wasserstein critic mathematically valid.

Symbol guide:

• $\lambda$: penalty coefficient (typically 10)
• $\hat{x}$: samples drawn by linearly interpolating between real and generated points: $\hat{x} = \epsilon x + (1-\epsilon)G(z)$, $\epsilon \sim U[0,1]$
• $\nabla_{\hat{x}} D(\hat{x})$: gradient of the critic w.r.t. the interpolated point

```python

# PyTorch: WGAN-GP gradient penalty def gradient_penalty(critic, real, fake, device, lambda_gp=10.0): B = real.size(0) eps = torch.rand(B, 1, 1, 1, device=device).expand_as(real) interpolated = (eps real + (1 - eps) fake).requires_grad_(True) d_interp = critic(interpolated) grads = torch.autograd.grad( outputs=d_interp, inputs=interpolated, grad_outputs=torch.ones_like(d_interp), create_graph=True, retain_graph=True, )[0] grads = grads.view(B, -1) # flatten spatial dims gp = ((grads.norm(2, dim=1) - 1) ** 2).mean() # (||grad|| - 1)^2 return lambda_gp * gp ```