Weight Decay and L2 Regularization
Imagine training a deep network that memorizes every quirk of its training set, including noise. It scores near-perfect training accuracy but fails badly on new data. A simple, decades-old fix exists. Discourage the optimizer from letting weights grow large without good reason. Large weights let a network fit sharp, idiosyncratic patterns. This article covers two closely related but subtly different ways to enforce that discouragement. One is adding an L2 penalty to the loss function. The other is directly shrinking weights at every update step. This second method is known as decoupled weight decay. For plain stochastic gradient descent these two techniques are mathematically identical. That is why generations of practitioners used the terms "L2 regularization" and "weight decay" interchangeably. But once adaptive optimizers like Adam and Its Variants enter the picture, the equivalence quietly breaks. The L2 penalty gets divided by the same adaptive per-parameter learning rate that scales the gradient. This distorts the intended shrinkage. Decoupled weight decay was introduced explicitly through the AdamW optimizer. It restores the clean shrinkage behavior by applying decay separately from the gradient-based update. This distinction matters enormously in practice. Papers and practitioners reported meaningfully better generalization simply by switching approaches. They moved from "Adam with L2 penalty" to "AdamW with decoupled decay." This was done using otherwise identical architectures and data. It is one of the clearest examples in deep learning of this kind. A seemingly cosmetic implementation detail can have real, measurable consequences for model quality. This is a reminder about regularization and optimizer internals. Their interaction cannot be reasoned about by formula alone. This article builds the concept from first principles. It covers the L2-penalized loss, its gradient, and the classical SGD equivalence with weight decay. It explains why Adam breaks that equivalence, and details the AdamW fix. We give exact pseudocode for both variants. We also provide a worked numerical trace showing the divergence in behavior. Common pitfalls that trip up practitioners moving between frameworks are also covered. The discussion connects forward to Overfitting and Regularization Strategies and Early Stopping. These are two other tools in the same generalization toolbox.
- L2 regularization adds `(λ/2)‖w‖²` to the loss; its gradient contributes `λw`, pulling weights toward zero.
- Weight decay is the update rule `w ← (1-ηλ)w - η·(gradient of unregularized loss)`, applied directly to parameters.
- For plain SGD, L2 regularization and weight decay are mathematically the same algorithm.
- For adaptive optimizers like Adam, L2 regularization gets rescaled by the adaptive learning rate; weight decay does not — they diverge.
- AdamW decouples the decay step from the gradient-based moment update, restoring the intended uniform shrinkage.
- Both act as an implicit prior (Gaussian prior on weights in a MAP view) that trades training fit for smaller weight norms and better generalization.
Weight decay shrinks every parameter toward zero by a fixed multiplicative factor each step, and this coincides with an L2 penalty only when the optimizer's per-parameter step size is not itself a function of the gradient's history.
The idea
L2 regularization modifies the training objective by adding a penalty proportional to the squared magnitude of the weights. Given an unregularized loss `L(w)` (e.g., cross-entropy averaged over a batch), the regularized objective is ``` J(w) = L(w) + (λ/2) · ‖w‖² ``` where `λ ≥ 0` is the regularization coefficient and `‖w‖² = Σᵢ wᵢ²` sums over all regularized parameters (typically weights, not biases). Taking the gradient with respect to `w` gives ``` ∇J(w) = ∇L(w) + λw ``` so every gradient step is nudged by an extra term `λw` that points opposite to `w`, i.e., toward the origin. This is why L2 regularization is sometimes described as inducing a Gaussian prior on the weights in a Bayesian maximum-a-posteriori interpretation: assuming `w ~ N(0, σ²I)` and taking the negative log-posterior recovers exactly this quadratic penalty, with `λ` tied to `1/σ²`.
Weight decay, in its classical sense, is not a loss modification but an update rule. At each step it directly shrinks parameters by a multiplicative factor before or alongside the gradient step: ``` w ← (1 - ηλ) · w - η · ∇L(w) ``` Here `η` is the learning rate. This is the rule implemented in the original weight-decay literature and in plain SGD-based frameworks. The crucial conceptual move is to treat "shrink the weight" and "follow the loss gradient" as two separate, additive updates rather than a single gradient computed from a combined objective. For vanilla SGD, substituting `∇J(w) = ∇L(w) + λw` into the standard SGD update `w ← w - η∇J(w)` gives ``` w ← w - η(∇L(w) + λw) = (1 - ηλ)w - η∇L(w) ``` which is identical to the weight-decay rule above. This is the origin of the interchangeable usage of the two terms: under SGD, L2 regularization is weight decay, term for term. The divergence appears only when the optimizer applies a per-parameter, gradient-history-dependent scaling — as Adam and Its Variants does — between the gradient and the parameter update.
How it works
Step 1 — SGD with L2 (equivalently, weight decay). Compute the gradient of the task loss, add `λw`, and take a scaled step. Because the `λw` term is scaled by the same learning rate as the gradient term, this exactly reproduces multiplicative shrinkage.
Step 2 — Adam with L2 penalty ("the naive port"). Many implementations simply add `λw` to the gradient before it enters Adam's moment estimates: ``` g ← ∇L(w) + λw m ← β₁ m + (1-β₁) g v ← β₂ v + (1-β₂) g² w ← w - η · m̂ / (√v̂ + ε) ``` The problem: the penalty term `λw` gets folded into `v`, Adam's second-moment (adaptive learning rate) estimator. Parameters with large historical gradients get a smaller effective decay, and parameters with small historical gradients get a larger effective decay — the opposite of a clean, uniform shrinkage. The regularization strength becomes entangled with the optimizer's curvature adaptation in a way that is hard to reason about or tune.
Step 3 — AdamW: decoupled weight decay. The fix is to remove `λw` from the gradient fed into the moment estimators, and instead apply the multiplicative shrinkage directly to the parameter, using the global learning rate rather than the adaptive per-parameter one: ``` g ← ∇L(w) # gradient of task loss only, no penalty mixed in m ← β₁ m + (1-β₁) g v ← β₂ v + (1-β₂) g² w ← w - η · m̂ / (√v̂ + ε) - η · λ · w # decay applied separately, using raw w ``` Now the decay term is a plain multiplicative shrink `w ← (1 - ηλ)w` layered on top of the Adam step, exactly mirroring the SGD case, and independent of each coordinate's gradient history.
Pseudocode
``` function TRAIN_STEP_SGD_WEIGHT_DECAY(w, grad_fn, eta, lambda): g = grad_fn(w) # gradient of task loss L only w = (1 - eta lambda) w - eta g return w function TRAIN_STEP_ADAM_L2(w, m, v, t, grad_fn, eta, lambda, beta1, beta2, eps): g = grad_fn(w) + lambda w # penalty folded into gradient (NAIVE / coupled) m = beta1 m + (1 - beta1) g v = beta2 v + (1 - beta2) (g g) t = t + 1 m_hat = m / (1 - beta1 <strong> t) v_hat = v / (1 - beta2 </strong> t) w = w - eta m_hat / (sqrt(v_hat) + eps) return w, m, v, t function TRAIN_STEP_ADAMW(w, m, v, t, grad_fn, eta, lambda, beta1, beta2, eps): g = grad_fn(w) # gradient of task loss ONLY m = beta1 m + (1 - beta1) g v = beta2 v + (1 - beta2) (g g) t = t + 1 m_hat = m / (1 - beta1 <strong> t) v_hat = v / (1 - beta2 </strong> t) w = w - eta m_hat / (sqrt(v_hat) + eps) # adaptive gradient step w = w - eta lambda w # decoupled multiplicative decay return w, m, v, t ``` Edge cases handled: at `t = 1`, bias correction terms `(1 - β₁ᵗ)` and `(1 - β₂ᵗ)` are non-zero (since `β₁, β₂ < 1`), so no division-by-zero occurs. If `λ = 0`, all three routines reduce to their undecayed counterparts, confirming decay is a strict additive control knob. Biases and normalization parameters (e.g., BatchNorm scale/shift) are conventionally excluded from `w` in the decay term, since shrinking them toward zero has no principled generalization justification and can hurt training.
Complexity & correctness
Time complexity. Both the coupled-L2 and decoupled-decay variants add strictly `O(d)` work per step, where `d` is the number of parameters: one extra elementwise multiply-and-add (`λw` or `ηλw`) alongside the existing `O(d)` gradient step, moment updates, and parameter update. This is asymptotically free — regularization does not change the per-step complexity class of training, only its constant factor. Over `T` steps, total added cost is `O(Td)`, the same order as the unregularized optimizer itself.
Space complexity. No new state is required. Weight decay reuses the parameter tensor `w`; L2 regularization computes `λw` on the fly from the existing tensor. Compare this to Adam's own memory overhead of two extra `O(d)` buffers (`m` and `v`) — weight decay adds zero additional buffers on top of that.
Correctness of the SGD equivalence. The claim "L2 penalty = weight decay under SGD" follows from direct algebraic substitution, not empirical coincidence: ``` J(w) = L(w) + (λ/2)‖w‖² ∇J(w) = ∇L(w) + λw SGD: w ← w - η∇J(w) = w - η∇L(w) - ηλw = (1-ηλ)w - η∇L(w) ``` This is an exact identity for every step and every parameter, with no approximation — hence "interchangeable" is not folklore but a theorem for this specific optimizer.
Why the equivalence breaks under Adam. Adam's update divides the (bias-corrected) gradient moment by `√v̂ + ε`, where `v` is an exponential moving average of squared gradients. When the penalty `λw` is folded into the gradient before this division, the effective per-coordinate shrinkage becomes ``` effective decay on coordinate i ∝ λ·wᵢ / (√v̂ᵢ + ε) ``` which depends on `v̂ᵢ`, i.e., on that coordinate's recent gradient magnitude history. Coordinates with large, consistently active gradients (steep loss directions) get a much smaller proportional decay than coordinates with small or sparse gradients. This is precisely backwards from the goal of uniform, magnitude-based shrinkage: parameters that already move a lot get under-regularized, while quiet parameters are over-regularized. AdamW's fix — applying `w ← w - ηλw` outside the gradient-moment pipeline — restores a decay term with no dependence on `v̂`, so the shrinkage factor `(1 - ηλ)` is identical across all coordinates, exactly reproducing the SGD-equivalence structure inside an adaptive optimizer.
Tradeoffs. Decoupled decay requires an extra design decision: with which `λ` and `η` schedule? Because decay is now scaled by `η` directly (not folded into the loss), when practitioners use learning-rate warmup or cosine decay schedules, the effective regularization strength drifts with the learning rate unless `λ` is explicitly re-tuned or scheduled independently. Some implementations therefore apply decay with a schedule proportional to `η` (as in the original AdamW paper) while others decouple it from `η` entirely — a live area of tuning subtlety, not a solved constant.
Worked example
Consider one parameter `w = 2.0` with a hypothetical gradient sequence over three steps: `g₁ = 0.01`, `g₂ = 0.01`, `g₃ = 4.0` (simulating a mostly-flat loss landscape that suddenly spikes). Let `η = 0.1`, `λ = 0.1`, `β₁ = 0.9`, `β₂ = 0.999`, `ε = 1e-8`. We compare Adam+L2 (coupled) vs. AdamW (decoupled), tracking only the decay contribution's effective strength.
Coupled (Adam+L2): the gradient fed to the moments is `g' = g + λw`.
| Step | g | g' = g+λw | v (EMA of g'²) | v̂ | decay-driven Δw component (∝ λw/√v̂) |
|---|---|---|---|---|---|
| 1 | 0.01 | 0.01+0.2=0.21 | 0.000044 | 0.044 | large: λw/√v̂ ≈ 0.2/0.21 ≈ 0.95 |
| 2 | 0.01 | 0.21 | ~0.000088 | 0.061 | still large: ≈0.2/0.25 ≈ 0.80 |
| 3 | 4.0 | 4.2 | jumps to ~17.6 | ~4.2 | shrinks sharply: ≈0.2/2.05 ≈ 0.10 |
Because `v̂` explodes after the large gradient at step 3, the effective decay contribution on `w` collapses by roughly 9× relative to step 1 — even though `λ` and `w` never changed. The optimizer is silently weakening regularization exactly when a large gradient arrives, which is the opposite of a stable, predictable prior.
Decoupled (AdamW): the moments only ever see `g` (no `λw` mixed in), and the decay step is applied separately: ``` w ← w - η·(adam step from g alone) - η·λ·w ``` Here the decay term is `η·λ·w = 0.1 · 0.1 · 2.0 = 0.02` at every step, regardless of whether `g` was `0.01` or `4.0`. The shrinkage is stable, predictable, and set entirely by the hyperparameters `η, λ`, exactly matching the SGD-equivalence intuition practitioners rely on when choosing `λ`. This trace is the concrete mechanism behind the widely reported empirical finding: models trained with AdamW often generalize better than identically-configured Adam+L2 models, because the decoupled version delivers the regularization strength the practitioner actually intended, uncontaminated by gradient-magnitude fluctuations across training.
Common pitfalls & traps
| Pitfall | Why it happens | Fix |
|---|---|---|
| Using Adam's built-in `weight_decay` argument, assuming it behaves like SGD's | Some libraries historically implemented `weight_decay` as coupled L2 inside Adam, not true decoupled decay | Check documentation; use the explicit AdamW optimizer class when decoupling is desired |
| Applying decay to biases and BatchNorm/LayerNorm affine parameters | Naive "apply to all parameters" loops sweep up 1-D scale/shift terms | Filter parameter groups; exclude 1-D tensors (biases, norm gains) from decay |
| Re-using the same `λ` when switching from SGD to AdamW | The interaction between `η` and `λ` differs since AdamW multiplies decay by `η` directly | Re-tune `λ` (or use the schedule-decoupled variant) after switching optimizers |
| Forgetting decay interacts with LR schedules | If `η` warms up or decays, the effective shrink factor `ηλ` moves with it even if `λ` is fixed | Explicitly decide whether decay should track the LR schedule or be scheduled independently |
| Treating L2-regularization and weight-decay as always equivalent | True only for plain (non-adaptive) SGD; false for Adam, RMSProp, etc. | Reason from the update rule, not the loss function, once adaptive optimizers are involved |
Pioneers & history
Weight decay traces back to classical regularization theory in statistics (ridge regression, Tikhonov regularization) long before deep learning, and was popularized in neural network training in the late 1980s as an explicit tool against overfitting. The precise dissection of L2-vs-decoupled-decay under adaptive optimizers is comparatively recent, driven by the widespread adoption of Adam.
Origins · 1 minIlya Loshchilov and how this was discovered
Where this matters
Weight decay (in its decoupled AdamW form) is the default optimizer regularizer in most modern transformer training pipelines, including large language models and vision transformers, where it is standard alongside learning-rate warmup and cosine decay schedules. Frameworks such as PyTorch and JAX/Optax expose both coupled (`Adam` with `weight_decay`, historically) and decoupled (`AdamW`) variants, and the choice measurably affects downstream benchmark accuracy in published ablations. Production training recipes for models like BERT, GPT-style architectures, and ResNet variants under adaptive optimizers explicitly recommend AdamW with carefully tuned decay coefficients, often applied selectively (excluding biases and normalization parameters) via parameter groups. Understanding this distinction is also foundational before studying broader Overfitting and Regularization Strategies, where weight decay sits alongside dropout, data augmentation, and Early Stopping as complementary tools that each attack overfitting from a different angle — decay through parameter-magnitude control, early stopping through training-time control.
Ask the tutor: How would you empirically detect, from training curves alone, whether a codebase's "Adam with weight_decay" is coupled L2 or true decoupled decay?
Is weight decay the same as L2 regularization?
Only under specific conditions. For plain stochastic gradient descent, adding an L2 penalty to the loss and directly shrinking weights each step (weight decay) produce algebraically identical updates — this is provable by substitution, not just empirically observed. Once you move to adaptive optimizers such as Adam, the two diverge: L2 regularization added to the loss gets folded into the gradient, which then gets divided by an adaptive per-parameter scale, distorting the intended uniform shrinkage. Weight decay applied directly to the parameter (as in AdamW) avoids this distortion. So the terms are historically interchangeable but technically distinct once you specify the optimizer.
Why does AdamW often generalize better than Adam with L2 penalty?
Because AdamW delivers the regularization strength the practitioner actually configured, while Adam+L2 delivers a distorted version of it. In Adam+L2, parameters with large recent gradients get an artificially weakened decay (since the penalty is divided by a large adaptive scale), while quiet parameters get over-decayed. This means the "effective" regularization strength varies unpredictably across parameters and across training time. AdamW's decoupled decay applies the same multiplicative shrink to every parameter regardless of gradient history, matching the clean Gaussian-prior interpretation of L2 regularization. Empirically, this more faithful regularization tends to produce models with better validation and test performance at matched training loss.
Should I apply weight decay to biases and normalization parameters?
Generally no. The theoretical justification for weight decay is a Gaussian prior favoring small weight magnitudes as a proxy for simpler, less overfit functions. Biases shift activations and don't contribute to model complexity in the same way; shrinking them toward zero has little regularization benefit and can hurt fitting flexibility. Normalization layer scale and shift parameters (like BatchNorm's gamma and beta) similarly aren't the source of overfitting-prone sharp decision boundaries. Most production training recipes explicitly partition parameters into "decay" and "no-decay" groups, applying weight decay only to weight matrices and convolutional filters, and excluding 1-D bias and normalization parameters.
Does increasing weight decay always improve generalization?
No — like any regularization strength, weight decay has a sweet spot. Too small a value under-regularizes, allowing overfitting as before. Too large a value over-constrains the model, pulling weights toward zero so aggressively that the network cannot fit even the true signal in the data, causing underfitting and increased training and validation loss. The optimal value is typically found via hyperparameter search (grid or random search over `λ`, often on a log scale), and it interacts with learning rate, batch size, and training duration — moving to a new dataset or architecture usually requires re-tuning it rather than reusing a previously "good" value blindly.
How does weight decay interact with learning rate schedules like warmup and cosine decay?
In decoupled AdamW, the parameter shrinkage per step is `ηλ`, meaning the effective decay strength scales directly with the current learning rate `η`. If `η` is annealed via a cosine schedule or warmed up from zero, the effective regularization strength moves with it: early in warmup, decay is weak; near the end of cosine annealing, decay is also weak, even if `λ` itself is held constant. Some training recipes explicitly account for this by scheduling `λ` in the opposite direction, or by treating `ηλ` as the tuned quantity directly. This coupling is a genuine subtlety and a common source of discrepancy between papers that report different optimal `λ` values for what looks like "the same" optimizer.
Can weight decay replace other regularization techniques like dropout?
Not as a full replacement — they attack overfitting through different mechanisms and are typically used together. Weight decay constrains parameter magnitude directly via the optimizer's update rule, favoring smoother, lower-norm solutions. Dropout stochastically removes units during training, acting more like an ensemble/noise-injection regularizer. Data augmentation expands the effective training distribution, and early stopping limits the number of updates a model receives. These techniques are complementary; state-of-the-art training recipes for vision and language models routinely combine several of them (e.g., AdamW plus dropout plus data augmentation) rather than relying on any single one, as covered further in Overfitting and Regularization Strategies.
What value of the weight decay coefficient λ is typical in practice?
There is no universal constant — typical ranges span roughly `1e-5` to `1e-1` depending on model size, dataset scale, and optimizer. Large-scale transformer training commonly uses values around `0.01` to `0.1` with AdamW, while smaller convolutional networks trained with SGD often use `1e-4` to `5e-4`. The right value depends on the interaction with learning rate (since AdamW's effective shrink is `ηλ`), batch size, and the amount of other regularization already present (dropout rate, augmentation strength). Practitioners typically tune `λ` via a hyperparameter sweep specific to their architecture and dataset scale rather than porting a published value directly.