Generative Adversarial Networks for Images

A Generative Adversarial Network consists of two differentiable functions, both usually convolutional neural networks. The generator $G_\theta: \mathbb{R}^d \to \mathbb{R}^{H\times W \times C}$ maps a latent noise vector $z \sim p_z$ (typically Gaussian or uniform) to a synthetic image $G(z)$. The discriminator $D_\phi: \mathbb{R}^{H\times W\times C} \to [0,1]$ takes an image and outputs the probability that the image came from the real data distribution $p_{\text{data}}$ rather than from $G$. The two networks are trained jointly but with opposing objectives, which is what makes this "adversarial." Formally, the training objective is the minimax game

The discriminator wants to push $D(x)\to 1$ for real images and $D(G(z))\to 0$ for fakes, maximizing $V$. The generator wants the opposite: it wants $D(G(z))\to 1$, i.e., it wants to fool the discriminator, which corresponds to minimizing $V$. Crucially there is no single loss function being jointly minimized by both networks — this is a game with a Nash equilibrium, not an optimization problem with a global minimum, and that distinction explains much of the instability discussed later. The key theoretical result, due to the original 2014 formulation, is that if $D$ is trained to optimality for a fixed $G$, the optimal discriminator has a closed form,

where $p_g$ is the distribution induced by $G$. Substituting this back into $V(D^*, G)$ reduces the generator's objective to (up to a constant) the Jensen–Shannon divergence between $p_{\text{data}}$ and $p_g$. Since the JS divergence is minimized (equal to zero) exactly when $p_g = p_{\text{data}}$, the global optimum of the game is a generator that perfectly reproduces the true data distribution, with the discriminator reduced to outputting $\tfrac12$ everywhere, unable to do better than a coin flip.

Training proceeds by alternating gradient updates: several (often one) step(s) updating $D$ with $G$ fixed, followed by one step updating $G$ with $D$ fixed, repeated until convergence or a fixed number of iterations. Each network only ever sees gradients through its own parameters, backpropagated through the other network's forward pass when needed (the generator's gradient must flow through the frozen discriminator). A critical practical modification is the non-saturating generator loss. Early in training, $G$ is poor and $D$ correctly rejects its samples with high confidence, so $D(G(z)) \approx 0$. The original generator loss $\log(1-D(G(z)))$ has near-zero gradient in this regime (it saturates), starving the generator of learning signal exactly when it needs it most. The fix is to instead train $G$ to maximize $\log D(G(z))$ directly. This has the same fixed point (both are minimized/maximized when $D(G(z))=1$) but provides much stronger gradients when $G$ is weak, because $\log D(G(z))$ is steep near $D=0$ while $\log(1-D(G(z)))$ is nearly flat there. ```

# GAN training loop (non-saturating generator loss, minibatch SGD) initialize generator parameters theta initialize discriminator parameters phi for iteration = 1 to N_iterations:

# ---- Discriminator step(s) ---- for k = 1 to k_disc_steps: sample minibatch of m real images {x_1, ..., x_m} from data sample minibatch of m noise vectors {z_1, ..., z_m} from p_z fake_images <- G(z_i; theta) for i = 1..m # no gradient needed yet

# discriminator loss: maximize log D(real) + log(1 - D(fake)) L_D <- -(1/m) sum_i [ log D(x_i; phi) + log(1 - D(fake_images_i; phi)) ] phi <- phi - lr_D gradient(L_D, phi) # gradient ASCENT on V == descent on -V

# ---- Generator step ---- sample minibatch of m noise vectors {z_1, ..., z_m} from p_z fake_images <- G(z_i; theta) for i = 1..m # gradient must flow through here

# non-saturating loss: minimize -log D(fake) instead of maximize log(1 - D(fake)) L_G <- -(1/m) sum_i [ log D(fake_images_i; phi) ] theta <- theta - lr_G gradient(L_G, theta) return G, D ``` Two implementation details matter for correctness. First, when computing `L_D`, the fake images are treated as constants with respect to $\phi$ (gradients do not need to flow into $G$), which is why many implementations detach $G$'s output from the computation graph during the discriminator step — this saves memory and avoids accidentally updating $\theta$ with the wrong loss. Second, when computing `L_G`, the discriminator's parameters $\phi$ must be held fixed (no update to $\phi$ from this loss); only gradients with respect to $\theta$ are taken, using $\phi$ purely as a frozen scoring function.

Time complexity per iteration. A single training iteration costs one forward and backward pass through $G$ (cost $O(P_G)$ where $P_G$ is the number of FLOPs in $G$'s architecture, dominated by convolution operations scaling with image resolution and channel width) plus $k$ forward/backward passes through $D$ for the discriminator sub-steps (cost $O(k \cdot P_D)$). For a convolutional architecture operating on an $H\times W$ image with $F$ feature channels and $L$ layers, per-layer cost is roughly $O(H W F^2 k^2)$ for a $k\times k$ kernel, so total cost per iteration is linear in the number of pixels processed and quadratic in channel width — identical asymptotic scaling to a standard CNN classifier or Image Generation Fundamentals autoencoder, since a GAN is architecturally just two CNNs. There is no additional algorithmic complexity beyond ordinary Backpropagation through each network; the added cost relative to a single supervised network is roughly a constant factor of $1+k$.

Space complexity is $O(P_G + P_D)$ for parameters plus the usual activation memory for backpropagation through both networks, proportional to batch size times the sum of feature-map sizes at every layer. Because gradients for $D$'s step do not need to be stored through $G$ (fake images can be detached), peak memory during the discriminator step only needs $D$'s activation graph, not $G$'s.

Correctness argument (why equilibrium implies matching distributions). The proof sketch has two parts, matching the derivation in "The idea" section. 1. Optimal discriminator for fixed $G$. For a fixed generator distribution $p_g$, $V(D,G)$ is, pointwise in $x$, of the form $a\log y + b \log(1-y)$ with $a = p_{\text{data}}(x)$, $b=p_g(x)$, $y = D(x)$. Differentiating with respect to $y$ and setting to zero gives $y^* = \tfrac{a}{a+b}$, recovering $D^*_G(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x)+p_g(x)}$. This is a maximum because the second derivative of $a\log y + b\log(1-y)$ is negative for $a,b\ge 0, y\in(0,1)$. 2. Global minimum over $G$. Substituting $D^*_G$ back in,

using the definition of Jensen–Shannon divergence in terms of KL divergences to the mixture $\tfrac12(p_{\text{data}}+p_g)$. Since $\mathrm{JSD}\ge 0$ with equality iff the two distributions are identical, $V(D^*_G,G)$ is minimized exactly at $p_g = p_{\text{data}}$, achieving the global minimum value $-\log 4$.

Honest caveats. This proof assumes $D$ and $G$ have unbounded capacity (they range over all possible functions, not just those representable by a fixed neural architecture), that $D$ is trained to full convergence at every step (not just a few gradient steps, as done in practice), and that the alternating gradient procedure actually converges to this equilibrium. None of these hold exactly in practice: real networks have finite capacity, $D$ is never trained to optimality, and simultaneous gradient descent on a non-convex-non-concave minimax game has no general convergence guarantee — the dynamics can cycle or diverge even when a Nash equilibrium exists. This gap between the elegant theory and the messy practice is exactly why GAN training is known to be finicky, and why the specific failure mode below is so important to understand.

Consider training a GAN on a toy 2D dataset that is not images but is easy to trace by hand: $p_{\text{data}}$ is a mixture of four 2D Gaussians arranged at the corners of a square (a common diagnostic used because it is easy to visualize whether $G$ covers all four modes or only some). Assume $z \in \mathbb{R}^2$ Gaussian noise, $G$ and $D$ are small MLPs, and we track qualitative behavior over training epochs.

This trace illustrates the central practical failure: between epochs 200 and 210, $G$ never learns to cover the whole distribution — it learns to win locally against whatever $D$ currently penalizes least, then $D$ catches up, and $G$ relocates its entire probability mass to a different single mode rather than spreading across all four. This oscillating, mode-hopping behavior is mode collapse, and it is a direct consequence of $G$'s objective only requiring it to fool $D$ on some subset of the space, not to match the full density $p_{\text{data}}$ at every training step (recall the JS-divergence equivalence only holds when $D$ is trained to full optimality, which never happens with a handful of gradient steps). Fixes demonstrated in row 5 of the table are real techniques used in practice: minibatch discrimination lets $D$ compare examples within a batch, so it can detect when a whole batch of "fakes" is suspiciously identical (real batches from a mixture of four Gaussians should show diversity; a mode-collapsed fake batch will not), giving $G$ a gradient signal to diversify. Unrolled GANs let $G$'s update account for several future steps of $D$'s optimization, discouraging $G$ from exploiting a discriminator that is about to be corrected.

The GAN framework arrived at a moment when explicit generative models — density estimators, variational autoencoders' precursors, and Boltzmann machines — struggled to produce sharp, high-resolution samples because they optimized likelihood-based objectives that reward covering probability mass everywhere, including blurry averages between modes. Casting generation as a game between a forger and a detective, rather than as direct density estimation, allowed sharpness to emerge naturally: the discriminator specifically punishes blur once it learns that blurry images are a giveaway of forgery. This reframing, simple in hindsight, unlocked a decade of rapid progress in photorealistic image synthesis.

GANs underpin a wide range of production and research systems in computer vision. Photo-realistic face synthesis systems (used for avatar generation, synthetic training data, and — controversially — deepfakes) are built directly on the generator-discriminator architecture described here. Super-resolution systems that upscale low-resolution images use adversarial losses layered on top of pixel-wise losses specifically because a purely pixel-wise objective produces blurry upscaling, while the discriminator's texture-sensitivity forces sharp, plausible high-frequency detail. Data augmentation pipelines in domains with scarce labeled data (medical imaging, rare-defect industrial inspection) use trained generators to synthesize additional realistic training examples for downstream classifiers. The conditional variant of the adversarial game, where both $G$ and $D$ are given an additional input (a class label, a sketch, a semantic map, or a source-domain image), is the direct foundation of Image-to-Image Translation systems that convert sketches to photos or day scenes to night scenes. The same discriminator-driven notion of a learned perceptual loss — punishing outputs that "look wrong" according to a trained critic rather than according to a fixed pixel distance — reappears as a key ingredient in Neural Style Transfer pipelines and in perceptual loss functions used throughout modern Image Generation Fundamentals research, including diffusion-model guidance and autoencoder-based compression systems.

Ask the tutor: How does the Wasserstein GAN loss change the training dynamics and specifically help mitigate mode collapse compared to the original minimax formulation?