Generative Adversarial Networks

The Minimax Objective

Let $G_\theta$ be the generator parameterized by $\theta$, mapping a latent vector $z \sim p_z(z)$ (typically $\mathcal{N}(0, I)$) to data space, and let $D_\phi$ be the discriminator parameterized by $\phi$, outputting a scalar probability that its input is real. The original GAN objective is:

Explanation: The discriminator maximizes its ability to assign high probability to real samples and low probability to fake ones, while the generator minimizes the same quantity — pulling in opposite directions on a shared objective. At the global optimum, the optimal discriminator for a fixed generator is:

Explanation: The ideal critic assigns each point a probability proportional to how much more likely it is to come from real data than from the generator, interpolating smoothly between 0 and 1. Substituting $D^*$ back into the objective yields a reformulation in terms of Jensen-Shannon divergence:

Explanation: At optimality, the generator is being trained to minimize the Jensen-Shannon divergence between the generated distribution and the real data distribution, which equals zero only when the two distributions are identical.

The Non-Saturating Generator Loss

In practice, $\log(1 - D_\phi(G_\theta(z)))$ saturates early in training when the discriminator easily rejects fakes. The standard fix is to flip the generator's loss:

Explanation: Instead of minimizing the discriminator's confidence in rejecting fakes, the generator now maximizes the discriminator's confidence in accepting them, providing much stronger gradients when outputs are obviously fake.

Wasserstein GAN Distance

The Wasserstein GAN replaces the Jensen-Shannon divergence with the Earth Mover's distance, enforced via a Lipschitz constraint (weight clipping or gradient penalty):

Explanation: The critic $f_\phi$ (no longer a probability) is constrained to be 1-Lipschitz, and its expected output gap between real and fake data is a meaningful, continuous distance metric that provides stable gradients even when distributions do not overlap.

Gradient Penalty (WGAN-GP)

Rather than weight clipping, WGAN-GP enforces the Lipschitz constraint by penalizing the gradient norm at interpolated points $\hat{x} = \epsilon x + (1-\epsilon)G(z)$:

Explanation: By sampling points between real and generated distributions and penalizing deviations of the critic's gradient norm from 1, we softly enforce the Lipschitz condition without clipping and without introducing optimization artifacts. ---

---

---

PyTorch Implementation

```python import torch import torch.nn as nn import torch.optim as optim

# ── Minimal DCGAN-style components ────────────────────────────────────────── class Generator(nn.Module): def __init__(self, z_dim=100, img_channels=1, features=64): super().__init__() self.net = nn.Sequential(

# z_dim → 7×7×(features2) nn.ConvTranspose2d(z_dim, features 2, 7, 1, 0, bias=False), nn.BatchNorm2d(features * 2), nn.ReLU(True),

# 7×7 → 14×14 nn.ConvTranspose2d(features * 2, features, 4, 2, 1, bias=False), nn.BatchNorm2d(features), nn.ReLU(True),

# 14×14 → 28×28 nn.ConvTranspose2d(features, img_channels, 4, 2, 1, bias=False), nn.Tanh(), # output ∈ [-1, 1] ) def forward(self, z): return self.net(z.view(-1, z.shape[1], 1, 1))

class Discriminator(nn.Module): def __init__(self, img_channels=1, features=64): super().__init__() self.net = nn.Sequential(

# 28×28 → 14×14 nn.Conv2d(img_channels, features, 4, 2, 1, bias=False), nn.LeakyReLU(0.2, inplace=True),

# 14×14 → 7×7 nn.Conv2d(features, features 2, 4, 2, 1, bias=False), nn.BatchNorm2d(features 2), nn.LeakyReLU(0.2, inplace=True), nn.Conv2d(features * 2, 1, 7, 1, 0, bias=False), nn.Sigmoid(), ) def forward(self, x): return self.net(x).view(-1)

# ── Training loop ──────────────────────────────────────────────────────────── def train_gan(G, D, loader, z_dim=100, n_epochs=50, device="cuda"): G, D = G.to(device), D.to(device) opt_G = optim.Adam(G.parameters(), lr=2e-4, betas=(0.5, 0.999)) opt_D = optim.Adam(D.parameters(), lr=2e-4, betas=(0.5, 0.999)) bce = nn.BCELoss() for epoch in range(n_epochs): for real, _ in loader: real = real.to(device) B = real.size(0)

# ── Step 1: Update Discriminator ────────────────────────────── z = torch.randn(B, z_dim, device=device) fake = G(z).detach() # stop gradient through G real_loss = bce(D(real), torch.ones(B, device=device)) fake_loss = bce(D(fake), torch.zeros(B, device=device)) loss_D = (real_loss + fake_loss) / 2 opt_D.zero_grad(); loss_D.backward(); opt_D.step()

# ── Step 2: Update Generator (non-saturating loss) ──────────── z = torch.randn(B, z_dim, device=device) fake = G(z) # fresh forward pass loss_G = bce(D(fake), torch.ones(B, device=device)) # fool D opt_G.zero_grad(); loss_G.backward(); opt_G.step() print(f"Epoch {epoch+1:3d} | D: {loss_D:.4f} | G: {loss_G:.4f}")

# ── WGAN-GP gradient penalty ───────────────────────────────────────────────── def gradient_penalty(D, real, fake, device, lambda_gp=10.0): B = real.size(0) eps = torch.rand(B, 1, 1, 1, device=device) # random interpolation x_hat = (eps real + (1 - eps) fake).requires_grad_(True) d_hat = D(x_hat) grad = torch.autograd.grad( outputs=d_hat, inputs=x_hat, grad_outputs=torch.ones_like(d_hat), create_graph=True, retain_graph=True )[0]

# ||∇D(x_hat)||₂ should equal 1 gp = lambda_gp * ((grad.norm(2, dim=[1,2,3]) - 1) ** 2).mean() return gp ``` ---