GAN Training Dynamics and Failure Modes

A GAN consists of a generator $G_\theta$ mapping noise $z \sim p_z$ to samples $G_\theta(z)$, and a discriminator $D_\phi$ mapping a sample $x$ to a probability that $x$ is real. Training is posed as the minimax game

For a fixed generator, the optimal discriminator has a closed form, $D^*(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)}$, where $p_g$ is the distribution induced by $G$. Substituting this back shows that, when the discriminator is optimal, the generator's objective reduces to (twice) the Jensen–Shannon divergence between $p_{\text{data}}$ and $p_g$, shifted by a constant. The theoretical global optimum is $p_g = p_{\text{data}}$, at which point $D^*(x) = \tfrac12$ everywhere: the discriminator cannot do better than chance. This is a Nash equilibrium: a pair $(\phi^*, \theta^*)$ such that neither player can unilaterally improve its objective by changing its own parameters while the other's stay fixed. Nash equilibria are the correct solution concept for two-player games, but they are fundamentally different from the minimizers that single-objective deep learning normally targets. A minimum of a single loss is a point where the gradient vanishes and the Hessian is positive semi-definite. A Nash equilibrium of a differentiable game is a point where the joint gradient vanishes, but the relevant second-order object is not a Hessian — it is the Jacobian of the joint gradient vector field, which need not be symmetric and can have complex eigenvalues. This asymmetry is the mathematical root of GAN training instability, and it is why naively applying gradient descent simultaneously to both players can produce cycling behavior that never converges, even in simple bilinear toy games. Two related pathologies deserve separate names because they have distinct causes. Vanishing gradients occur when the discriminator becomes too confident too early: if $D$ perfectly separates real from fake, then $\log(1 - D(G(z)))$ saturates and its gradient with respect to $G$'s parameters vanishes, starving the generator of learning signal. Mode collapse occurs when the generator converges to producing a narrow subset of the data distribution's modes — sometimes even a single point — because that subset currently fools the discriminator, and the generator has no explicit incentive to cover the full support of $p_{\text{data}}$.

The standard training loop alternates stochastic gradient updates on the discriminator and generator. The following pseudocode reflects the practically used non-saturating generator loss, introduced in the original GAN paper as a fix for early vanishing gradients, alongside the option to insert a stabilization technique. ``` Inputs: real data distribution p_data noise distribution p_z generator G_theta, discriminator D_phi k: number of discriminator updates per generator update m: minibatch size eta_D, eta_G: learning rates repeat until convergence or fixed number of iterations:

# ---- Discriminator phase (k steps) ---- for step = 1 to k: sample real batch {x_1, ..., x_m} ~ p_data sample noise batch {z_1, ..., z_m} ~ p_z fake batch {x_hat_1, ..., x_hat_m} = { G_theta(z_i) }

# discriminator wants D(real) -> 1, D(fake) -> 0 L_D = -(1/m) * sum_i [ log D_phi(x_i) + log(1 - D_phi(x_hat_i)) ]

# optional stabilizer added here, e.g. gradient penalty:

# L_D += lambda gradient_penalty(D_phi, x_i, x_hat_i) phi <- phi - eta_D grad_phi(L_D)

# ---- Generator phase (1 step) ---- sample noise batch {z_1, ..., z_m} ~ p_z fake batch {x_hat_i} = { G_theta(z_i) }

# non-saturating loss: generator wants D(fake) -> 1, not (1 - D) -> 0 L_G = -(1/m) sum_i [ log D_phi(x_hat_i) ] theta <- theta - eta_G grad_theta(L_G)

# ---- Monitoring (every N iterations) ---- log( L_D, L_G, D(real) average, D(fake) average, sample diversity metric ) ``` Two design choices in this loop matter enormously and are worth calling out explicitly.

Why non-saturating loss instead of the theoretical minimax loss. The original generator objective is $\min_\theta \mathbb{E}_z[\log(1 - D(G(z)))]$. Early in training, $D$ easily rejects fake samples, so $D(G(z)) \approx 0$, and $\log(1-D(G(z))) \approx \log(1)=0$ with a near-zero gradient in the regime where $D(G(z))$ is small — the function $\log(1-y)$ has small slope near $y=0$. The non-saturating alternative $-\log D(G(z))$ has slope $-1/D(G(z))$, which is large precisely when $D(G(z))$ is small, so the generator gets strong gradient signal exactly when it needs it most. Both losses share the same fixed point (they push $D(G(z))$ toward 1), but the non-saturating version has much better gradient behavior away from equilibrium; it sacrifices the clean Jensen–Shannon-divergence interpretation of the original objective in exchange for usable gradients.

Why alternate updates rather than solve jointly. The theoretical objective calls for a full inner maximization over $\phi$ before every step of $\theta$ (i.e., $D$ should always be near-optimal for the current $G$). In practice this is approximated with $k$ discriminator steps ($k=1$ is common) per generator step. Using a fully-optimized $D$ at every step is expensive and, as discussed below, an over-strong discriminator causes vanishing gradients — so $k$ is a knob that trades off approximation quality against gradient health.

Per-iteration time and space complexity. Each discriminator step costs one forward and backward pass through $D$ on a batch of size $m$, plus a forward pass through $G$ to produce fakes; each generator step costs a forward pass through $G$ and a forward-backward pass through $D$ (since gradients must flow back through $D$ into $G$). If $D$ and $G$ have $P_D$ and $P_G$ parameters respectively, and forward/backward cost is linear in parameters and batch size (as with standard feedforward or convolutional layers), one full outer iteration with $k$ discriminator steps costs

Space complexity is $O(m \cdot (P_D + P_G))$ dominated by storing activations for backpropagation through both networks on a batch, the same order as training a single network of comparable size — GANs do not asymptotically cost more per step than supervised training, they simply need more steps and more careful tuning to reach a usable equilibrium.

Correctness in the idealized (non-parametric) case. The classic argument for the original loss proceeds in two stages, and it is worth walking through why it is only a pointwise, infinite-capacity argument. Stage 1 — optimal discriminator. For a fixed $G$, and hence fixed $p_g$, the discriminator objective at each $x$ is

maximized over $y \in (0,1)$ at $y^* = \frac{a}{a+b}$, giving $D^*(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x)+p_g(x)}$. This is a pointwise concave maximization, so the maximum is unique and correct wherever $a + b > 0$. Stage 2 — optimal generator given optimal discriminator. Substituting $D^*$ into $V$ gives

Since Jensen–Shannon divergence is non-negative and zero only when the two distributions coincide, $C(G)$ is globally minimized exactly at $p_g = p_{\text{data}}$, with value $-\log 4$. This shows the global solution of the idealized game is correct: matching the data distribution is the unique optimum.

Why this proof does not certify that training converges there. The proof assumes (a) the discriminator is optimized to convergence at every step, and (b) $G$ and $D$ have infinite capacity to represent the corresponding functions/distributions exactly, and (c) updates are performed in the space of distributions/functions, not in the space of a finite set of neural network parameters. In practice none of these hold: $D$ is updated with a handful of gradient steps, not to convergence; $G$ and $D$ are parametric with finite capacity; and gradient descent moves parameters $\theta,\phi$, not distributions $p_g$ directly. The correct object to analyze is the joint vector field

and simultaneous or alternating gradient descent is really discretized simulation of the dynamical system $\dot\theta = -\nabla_\theta L_G$, $\dot\phi = -\nabla_\phi L_D$. Near a stationary point, local behavior is governed by the Jacobian of $F$. Unlike a single-objective minimization, this Jacobian is generally not symmetric, so its eigenvalues can be complex. If the eigenvalues have zero real part and nonzero imaginary part — which happens exactly at the classic "bilinear game" $V=\theta\phi$ used as a minimal GAN toy model — gradient descent produces undamped oscillation/cycling around the equilibrium rather than convergence to it, even though the equilibrium exists and is unique. This is the rigorous reason GAN training can fail to converge even when everything is theoretically well-posed: the fixed point is a saddle of a game, not a minimum of a function, and generic gradient dynamics around saddles of asymmetric games need not be attracting.

Trade-offs. Techniques that stabilize training generally do so by modifying the loss landscape so the Jacobian's eigenvalues stay well-behaved (real part negative, small imaginary part) near equilibrium, at some cost: gradient penalties add compute for double backpropagation through $D$; spectral normalization constrains $D$'s expressiveness (Lipschitz constant $\leq 1$ per layer) which can slow learning; Wasserstein-style critics require multiple critic updates per generator update, increasing wall-clock cost per generator step even though the equilibrium reasoning improves.

Consider a generator trying to model a simple 2-D two-mode data distribution: half the real data is clustered near $(-2,0)$ and half near $(2,0)$. Track a plausible failure trajectory across training epochs.

Now trace what a gradient penalty style fix (from Wasserstein-GAN with gradient penalty) changes in this trajectory. Instead of $D$ outputting a saturating probability, $D$ becomes a critic $f_\phi$ with output in $\mathbb{R}$, trained to maximize $\mathbb{E}[f(x)] - \mathbb{E}[f(G(z))]$ subject to being 1-Lipschitz, enforced by penalizing $(\|\nabla_x f(\hat x)\|_2 - 1)^2$ at interpolated points $\hat x = \epsilon x + (1-\epsilon) G(z)$. Because the Lipschitz constraint prevents $f$ from becoming arbitrarily steep around any single point, the critic cannot become "too confident too fast" the way the sigmoid discriminator did at epoch 25 above. Concretely, at epoch 25 in the fixed pipeline, $\|\nabla_x f\|$ near the collapsed cluster is penalized down toward 1 rather than exploding, so the gradient $\nabla_\theta L_G$ flowing back into $G$ stays informative — pulling probability mass toward $(-2,0)$ while it is still under-represented, rather than only after D has already saturated. The mechanism is not magic: it directly targets the term that caused the epoch-26 vanishing-gradient row in the table.

!A conceptual illustration of adversarial training as a two-player game

GAN training dynamics research directly shaped the tools used in production generative-image systems throughout the 2016–2021 era: StyleGAN and its successors (used in face synthesis, art tools, and synthetic data pipelines) rely heavily on progressive growing, path-length regularization, and careful learning-rate balancing, all of which are direct engineering responses to the instabilities described here. Wasserstein-style losses and spectral normalization are standard components in image-to-image translation systems (CycleGAN-family architectures), super-resolution networks, and voice/audio synthesis GANs, because these domains cannot tolerate mode collapse without visibly repetitive or artifact-laden outputs. Beyond generative modeling itself, the game-theoretic lens on adversarial optimization — a min-max objective whose stationary points are saddles rather than minima — reappears directly in adversarial training for robust classifiers, where a model is trained against an adversary crafting worst-case perturbations, which is the natural bridge into Adversarial Examples and Robustness. Understanding why GAN saddle points are hard to reach with naive gradient descent is the same mathematical machinery needed to understand why adversarially robust training is also notoriously unstable and sensitive to hyperparameters, and it connects back to core ideas in gradient descent and general optimization theory for non-convex, non-concave objectives.

Ask the tutor: How exactly does the Wasserstein distance stay differentiable when the generator and data distributions have disjoint support, and why does Jensen-Shannon divergence fail there?