Weight Decay and L2 Regularization

The basic idea is to add an extra term to the loss function that grows larger whenever the model's weights grow larger. This creates a tug-of-war: the gradient of the data loss pulls weights toward values that fit the training data, while the regularization term pulls all weights back toward zero. The modified loss function looks like this:

Explanation: The total loss is the original data loss plus a penalty equal to half the sum of all squared weights, scaled by a small constant lambda.

Symbols:

• $L_{\text{reg}}$ — the new regularized loss your optimizer actually minimizes
• $L_{\text{data}}$ — your original loss (cross-entropy, MSE, etc.)
• $\lambda$ (lambda) — the regularization strength, a hyperparameter you choose (e.g. 0.01)
• $w_i$ — each individual weight in the model
• $\sum_i w_i^2$ — the sum of all squared weights (also called the "L2 norm squared")

When you take the gradient of this modified loss and update weights, the update rule becomes:

Explanation: Each weight moves down the data gradient as usual, but also gets subtracted a small fraction of itself — this is the "decay" part.

Symbols:

• $\eta$ (eta) — the learning rate
• $\frac{\partial L_{\text{data}}}{\partial w_i}$ — the gradient of the data loss with respect to weight $w_i$
• $\eta \lambda w_i$ — the weight decay term; proportional to the current weight value

The last term $-\eta \lambda w_i$ is exactly "weight decay": the weight shrinks by a small percentage of itself on every single update. If lambda is 0.01 and the learning rate is 0.001, each weight loses 0.001% of its value every step, regardless of what the gradients are doing. Here is a minimal working example in PyTorch: ```python import torch import torch.nn as nn

# Simple model model = nn.Linear(10, 1)

# Option 1: L2 regularization via weight_decay in optimizer

# This is TRUE weight decay in SGD (equivalent to L2 reg for SGD) optimizer = torch.optim.SGD( model.parameters(), lr=0.01, weight_decay=1e-4 # this is lambda )

# Option 2: AdamW — decoupled weight decay (recommended for Adam) optimizer_adamw = torch.optim.AdamW( model.parameters(), lr=1e-3, weight_decay=1e-2 )

# Training step looks identical — the magic happens inside the optimizer loss_fn = nn.MSELoss() X = torch.randn(32, 10) y = torch.randn(32, 1) optimizer.zero_grad() loss = loss_fn(model(X), y) loss.backward() optimizer.step() # weight decay applied automatically here ``` You can also visualize the effect of different lambda values: ```python import numpy as np import matplotlib.pyplot as plt

# Show how weight decay shrinks a weight over 100 steps with no gradient signal lambdas = [0.0, 0.01, 0.05, 0.1] steps = 100 w0 = 1.0 # starting weight value lr = 0.1 fig, ax = plt.subplots() for lam in lambdas: weights = [w0] w = w0 for _ in range(steps): w = w - lr lam w # pure decay, no data gradient weights.append(w) ax.plot(weights, label=f"λ={lam}") ax.set_xlabel("Step") ax.set_ylabel("Weight value") ax.set_title("Weight decay toward zero (no gradient signal)") ax.legend() plt.tight_layout() plt.show() ``` Running this will show that with $\lambda = 0$, the weight stays at 1.0 forever. With larger lambda values, it decays exponentially toward zero — the model "forgets" large weights unless the data insists on keeping them.

Dropout Regularization · Batch Normalization · AdamW Optimizer · Learning Rate Scheduling · Bias-Variance Tradeoff