AdamW and Decoupled Weight Decay

AdamW is an optimization algorithm for training deep neural networks that decouples L2 weight decay from the adaptive gradient update, applying the decay directly to the parameters rather than through the gradient. Key result in words: for adaptive methods like Adam, the common practice of adding an L2 penalty to the loss is NOT equivalent to true weight decay, because the adaptive denominator rescales the penalty; decoupling restores the intended uniform shrinkage and dramatically improves generalization. Where it applies: training transformers, CNNs, and virtually all large-scale modern deep learning. Who/when: Ilya Loshchilov and Frank Hutter, 2017 ("Decoupled Weight Decay Regularization," ICLR 2019).

AdamW modifies the Adam optimizer by removing the weight-decay term from the gradient (which Adam would then adaptively rescale) and instead subtracting a fixed fraction $λ$ of each weight directly during the parameter update. The distinction matters because in standard Adam, an L2 regularizer $\frac{λ}{2} ∥ θ ∥^{2}$ contributes a gradient $λ θ$ that gets divided by the per-coordinate adaptive learning rate $\overset{v}{^}_{t} + ϵ$ — meaning weights with large historical gradients are decayed less, exactly opposite to the regularization one intends. The surprising fact: for plain SGD, L2 regularization and weight decay are mathematically identical, so the entire deep learning community implicitly assumed this equivalence held for Adam too — it does not, and the error silently degraded generalization for years. Decoupled weight decay is now the default in essentially every serious training pipeline.

The Core Insight

L2 regularization and weight decay are the same thing only for optimizers with a coordinate-uniform effective learning rate (like vanilla SGD). For adaptive optimizers, the adaptive preconditioner couples them in a way that corrupts the regularization. The fix is to decouple: never let weight decay enter the moving averages or the adaptive rescaling — apply it as an independent, multiplicative shrinkage of the weights.

Formal Statement

L2 Regularization vs. Weight Decay

Consider minimizing a loss $f (θ)$ . Two distinct mechanisms are often conflated:

L2 regularization augments the objective:

f^{reg} (θ) = f (θ) + \frac{λ ^{'}}{2} ∥ θ ∥_{2}^{2}, \nabla f^{reg} (θ) = \nabla f (θ) + λ^{'} θ .

⟳differentiate the quadratic penalty;

\frac{d}{d θ} \frac{1}{2} ∥ θ ∥^{2} = θ

Weight decay modifies the update rule directly, shrinking parameters by a multiplicative factor each step:

θ_{t} = (1 - η λ) θ_{t - 1} - η g_{t},

⟳take the SGD step

θ_{t} = θ_{t - 1} - η g_{t}

and prepend an explicit exponential shrinkage

(1 - η λ)

on the previous parameters.

where $η$ is the learning rate, $λ$ the weight-decay coefficient, $λ^{'}$ the L2 coefficient, and $g_{t} = \nabla f (θ_{t - 1})$ the loss gradient.

Equivalence under SGD

For plain stochastic gradient descent, L2 regularization with coefficient $λ^{'}$ is exactly equivalent to weight decay with coefficient $λ = λ^{'}$ . That is, the iterates produced by [1] under SGD coincide with those of [2].

Step 1. Apply SGD to the L2-regularized objective using gradient from [1]:

θ_{t} = θ_{t - 1} - η (g_{t} + λ^{'} θ_{t - 1}) .

⟳substitute

\nabla f^{reg} = g_{t} + λ^{'} θ_{t - 1}

into the SGD step.

Step 2. Distribute the

η

and collect the

θ_{t - 1}

terms:

θ_{t} = θ_{t - 1} - η g_{t} - η λ^{'} θ_{t - 1} = (1 - η λ^{'}) θ_{t - 1} - η g_{t} .

⟳factor out

θ_{t - 1}

from

θ_{t - 1} - η λ^{'} θ_{t - 1}

Step 3. Equation [4] is identical to the weight-decay rule [2] with

λ = λ^{'}

. Hence the two are interchangeable for SGD.

■

The equivalence breaks once the gradient is preconditioned by a non-scalar matrix $M_{t}$ (e.g. Adam's diagonal adaptive scaling), because the L2 term $λ^{'} θ$ is then multiplied by $M_{t}$ while the decay term in [2] is not. We prove the failure below.

Adam

Adam maintains exponential moving averages of the gradient (first moment $m_{t}$ ) and squared gradient (second moment $v_{t}$ ), with decay rates $β_{1}, β_{2} \in [0, 1)$ :

m_{t} = β_{1} m_{t - 1} + (1 - β_{1}) g_{t}, v_{t} = β_{2} v_{t - 1} + (1 - β_{2}) g_{t}^{2},

⟳these are standard EMA recursions;

g_{t}^{2}

is element-wise squaring.

with bias-corrected estimates and the update

\overset{m}{^}_{t} = \frac{m _{t}}{1 - β _{1}^{t}}, \overset{v}{^}_{t} = \frac{v _{t}}{1 - β _{2}^{t}}, θ_{t} = θ_{t - 1} - η \frac{m ^ _{t}}{v ^ _{t} + ϵ} .

⟳at step

t

an EMA initialized at zero is biased toward zero by factor

(1 - β^{t})

; dividing removes the bias. The final term preconditions the gradient by the inverse RMS of recent gradients.

Here $ϵ$ is a small constant (e.g. $1 0^{- 8}$ ) for numerical stability; all operations are element-wise.

AdamW — Decoupled Weight Decay

AdamW applies the adaptive update to the pure loss gradient (no L2 term inside $m_{t}, v_{t}$ ) and adds decoupled decay:

θ_{t} = θ_{t - 1} - η (\frac{m ^ _{t}}{v ^ _{t} + ϵ} + λ θ_{t - 1}),

⟳take the Adam update [6], then subtract an additional

η λ θ_{t - 1}

that is NOT passed through the

\overset{v}{^}_{t}

preconditioner — this is the decoupling.

equivalently, with a schedule multiplier $η_{t} = η s_{t}$ (where $s_{t}$ is the learning-rate schedule):

θ_{t} = (1 - η_{t} λ) θ_{t - 1} - η_{t} \frac{m ^ _{t}}{v ^ _{t} + ϵ} .

⟳rewrite [7] grouping the

θ_{t - 1}

terms; in the original paper the decay is scaled by the same schedule

s_{t}

as the learning rate but is independent of

η

's base magnitude.

Here $m_{t}, v_{t}$ are computed from $g_{t} = \nabla f (θ_{t - 1})$ without the $λ θ$ term.

Why It Holds / Derivation

Why L2 ≠ Weight Decay under Adam

Step 1. "Adam with L2 regularization" feeds the regularized gradient

g_{t}^{reg} = g_{t} + λ^{'} θ_{t - 1}

into the moments. The update direction becomes

θ_{t} = θ_{t - 1} - η \frac{m _{t}^{reg}}{v _{t}^{reg} + ϵ},

⟳substitute

g_{t}^{reg}

for

g_{t}

throughout [5]–[6]; both moments now contain the

λ^{'} θ

contamination.

where both $m_{t}^{reg}$ and $v_{t}^{reg}$ depend on $λ^{'} θ_{t - 1}$ .

Step 2. Approximate near convergence where the contribution of the penalty to the second moment is dominated by the genuine gradient magnitude. The effective decay applied to coordinate

i

scales as

effective decay_{i} \approx \frac{η λ ^{'} θ _{t - 1, i}}{v ^ _{t, i} + ϵ} .

⟳isolate the part of [9] proportional to

θ_{t - 1, i}

, namely

λ^{'} θ_{t - 1, i}

in the numerator divided by the per-coordinate preconditioner

\overset{v}{^}_{t, i}

Step 3. Coordinates with large historical gradients (large

\overset{v}{^}_{t, i}

) receive less decay; coordinates with small gradients receive more. This per-coordinate distortion means the regularization no longer shrinks all weights uniformly — the intended global

ℓ_{2}

ball penalty is replaced by an irregular, gradient-dependent one. Step 4. In AdamW [7], the decay term

λ θ_{t - 1}

is added outside the preconditioner, so every coordinate is shrunk by exactly

η λ

regardless of

\overset{v}{^}_{t, i}

. This restores uniform decay and recovers the true regularizing geometry.

■

Ilya Loshchilov & Frank Hutter | 2017–2019

Working at the University of Freiburg's machine learning lab, Loshchilov and Hutter were studying learning-rate schedules (their earlier SGDR / warm-restart work) when they noticed that Adam consistently generalized worse than SGD with momentum on image classification — a folklore observation across the field. Rather than accept it, they diagnosed the cause: practitioners were copying L2 regularization code from SGD into Adam, unknowingly entangling it with the adaptive denominator. Their fix was almost embarrassingly simple — move two lines of code — yet it closed much of the Adam-vs-SGD generalization gap. The paper's clarity (it includes side-by-side pseudocode highlighting the decoupling line) made AdamW an instant default. Hutter is also a central figure in AutoML and hyperparameter optimization; Loshchilov later contributed to cosine annealing schedules now used universally.

Building Intuition

Think of weight decay as a spring pulling every parameter toward zero with force proportional to its displacement. In SGD, every coordinate feels the same spring stiffness, so the metaphor is faithful. Adam, however, gives each coordinate its own adaptive "gear ratio" $1/ \overset{v}{^}_{i}$ — coordinates seeing big gradients move in low gear (small steps), small-gradient coordinates in high gear. When you bolt the L2 spring inside the gearbox, its restoring force gets divided by the same gear ratio: the parameters that already move slowly get pulled hard toward zero, and the fast-moving ones barely feel the spring at all. That's backwards — those big-gradient weights are often the important, high-magnitude ones you'd most want to regularize. AdamW bolts the spring outside the gearbox, so it pulls uniformly on everyone, as a regularizer should.

L2-regularization inside Adam's preconditioner distorts decay per-coordinate; AdamW applies it uniformly outside

Worked Examples

Single-coordinate decay distortion

Given: Learning rate

η = 1 0^{- 3}

, intended decay

λ = 0.1

, weight

θ = 0.5

. Two coordinates with bias-corrected second moments

\overset{v}{^}_{A} = 1 0^{- 4}

(quiet coordinate) and

\overset{v}{^}_{B} = 1.0

(active coordinate),

ϵ \approx 0

. Adam + L2 (coupled), effective decay from [10],

decay_{i} = \frac{η λ θ}{v ^ _{i}}

Coordinate A: $\overset{v}{^}_{A} = 1 0^{- 2}$ , decay $= \frac{1 0 ^{- 3} \cdot 0.1 \cdot 0.5}{1 0 ^{- 2}} = \frac{5 \times 1 0 ^{- 5}}{1 0 ^{- 2}} = 5 \times 1 0^{- 3}$ .
Coordinate B: $\overset{v}{^}_{B} = 1$ , decay $= \frac{1 0 ^{- 3} \cdot 0.1 \cdot 0.5}{1} = 5 \times 1 0^{- 5}$ .

Ratio A:B $= 100 : 1$ — the quiet coordinate is decayed 100× more than the active one.

AdamW (decoupled), effective decay

= η λ θ = 1 0^{- 3} \cdot 0.1 \cdot 0.5 = 5 \times 1 0^{- 5}

for both coordinates.

Adam+L2: 5 \times 1 0^{- 3} vs 5 \times 1 0^{- 5}; AdamW: 5 \times 1 0^{- 5} uniformly (dimensionless weight units)

Multiplicative shrinkage over an epoch

Given: AdamW with constant schedule

s_{t} = 1

η = 3 \times 1 0^{- 4}

λ = 0.05

, and a parameter whose gradient signal is negligible (so the adaptive term ≈ 0 for that coordinate). How much does it shrink over

N = 10, 000

steps? Step 1. Per-step shrink factor from [8]:

1 - η λ = 1 - 3 \times 1 0^{- 4} \cdot 0.05 = 1 - 1.5 \times 1 0^{- 5}

. Step 2. Over

N

steps:

θ_{N} / θ_{0} = (1 - 1.5 \times 1 0^{- 5})^{10000} \approx e^{- 1.5 \times 1 0^{- 5} \cdot 1 0^{4}} = e^{- 0.15}

. Step 3.

e^{- 0.15} \approx 0.861

θ_{N} \approx 0.861 θ_{0} \Rightarrow \approx 13.9% shrinkage (dimensionless)

Note this shrinkage is independent of the adaptive denominator — exactly the clean, predictable behavior decoupling guarantees.

Common Misconceptions

Decay is NOT scaled by the adaptive denominator

The single most common bug is writing grad = grad + wd * param before the moment updates in an Adam implementation and calling it "weight decay." That is L2 regularization, and under Adam it is not weight decay. The PyTorch torch.optim.Adam(weight_decay=...) historically did exactly this (coupled L2); only torch.optim.AdamW implements true decoupling. Mixing them up silently changes your effective regularization by orders of magnitude across coordinates.

Optimal λ differs between Adam-L2 and AdamW

Because the coupled and decoupled versions apply decay differently, the optimal weight-decay coefficient is different — you cannot transfer a tuned $λ$ from one to the other. Loshchilov & Hutter also observed that in AdamW, the optimal $λ$ and optimal $η$ are far more decoupled (nearly orthogonal in hyperparameter space), making grid search dramatically easier — a practical bonus of the method.

Where L2 = weight decay still holds

If you replace Adam's adaptive denominator with a scalar (set $\overset{v}{^}_{t} \equiv c$ , a constant shared across all coordinates), then the preconditioner is $1/ c$ everywhere and divides every term — including the L2 contribution — by the same factor. In that degenerate case the distinction collapses and coupled L2 again equals weight decay (with $λ = λ^{'} / c$ ). The pathology is specifically the per-coordinate, non-uniform nature of $\overset{v}{^}_{t}$ . This confirms the proof's claim: it is the coordinate-dependence of the preconditioner, not adaptivity per se, that breaks the equivalence.

Connections

AdamW sits at the intersection of optimization and regularization theory. It refines Adam by correcting how regularization interacts with its preconditioner, and generalizes the SGD identity that links L2 Regularization with Weight Decay. The decoupling principle echoes Proximal Gradient Methods: weight decay is precisely the proximal operator of the $ℓ_{2}$ penalty, $prox_{η λ ∥ \cdot ∥^{2} /2} (θ) = θ / (1 + η λ) \approx (1 - η λ) θ$ , applied after the gradient step rather than folded into it — making AdamW a quasi-proximal/forward-backward scheme. The same coordinate-distortion analysis applies to any Adaptive Gradient Methods (RMSProp, Adagrad, Adadelta), each of which benefits from decoupling. The choice of learning-rate multiplier $s_{t}$ for the decay term connects to Learning Rate Scheduling and the authors' own Cosine Annealing / warm-restart work.

Real-World Applications

Transformer language models — GPT-2/3, BERT, T5, and LLaMA families all train with AdamW; it is the de facto standard optimizer for pretraining large language models, where weight decay (~0.1) materially affects final perplexity.
Vision Transformers (ViT) and ConvNeXt — AdamW is essential for stable ViT training; SGD struggles, and coupled-L2 Adam underperforms by several accuracy points on ImageNet.
Hugging Face Transformers — the Trainer API defaults to AdamW, exposing weight_decay as the decoupled coefficient and applying it to all parameters except biases and LayerNorm weights (a common heuristic).
Stable Diffusion / latent diffusion models — text-to-image generators are trained with AdamW; decoupled decay stabilizes the very large U-Net + text-encoder parameter sets.
PyTorch & TensorFlow core libraries — torch.optim.AdamW, tf.keras.optimizers.AdamW, and JAX/Optax's adamw are first-class built-ins.
Fine-tuning pipelines (LoRA, full fine-tune) — AdamW with small $λ$ is the default for adapting foundation models to downstream tasks.
Reinforcement learning from human feedback (RLHF) — policy-optimization stages in alignment pipelines commonly use AdamW for the reward and policy models.

Historical Development

1988–1991 — Weight decay popularized as a neural-net regularizer (Krogh & Hertz, A Simple Weight Decay Can Improve Generalization, NeurIPS 1991), establishing the SGD identity.
2011 — Adagrad (Duchi, Hazan, Singer) introduces per-coordinate adaptive learning rates, planting the seed for the decoupling problem.
2012 — RMSProp (Tieleman & Hinton, Coursera lecture) introduces the EMA of squared gradients used by Adam.
2014 — Adam introduced by Kingma & Ba (Adam: A Method for Stochastic Optimization); its paper adds L2 via the gradient, cementing the coupled convention.
2016 — Loshchilov & Hutter publish SGDR (warm restarts / cosine annealing), the schedule research that led them toward Adam's deficiencies.
2017 (Nov) — Loshchilov & Hutter post Fixing Weight Decay Regularization in Adam (arXiv:1711.05101), introducing AdamW and AdamWR.
2018 — Wilson et al. (The Marginal Value of Adaptive Gradient Methods) document Adam's generalization gap vs. SGD — context AdamW directly addresses.
2019 — The revised paper, Decoupled Weight Decay Regularization, is published at ICLR 2019; AdamW becomes the recommended default.
2019–2020 — AdamW adopted by major frameworks and by transformer training recipes (BERT, GPT-2), becoming ubiquitous.

Subtleties & Limits

Several questions remain open or actively debated. (1) Optimal decay placement under schedules: whether the decay multiplier should track the learning-rate schedule $s_{t}$ (original paper) or remain fixed is empirically inconsistent; "fully decoupled" variants and AdamW with constant $λ$ both appear in practice. (2) Which parameters to decay: the heuristic of excluding biases, LayerNorm/BatchNorm scale-and-shift parameters from decay is folklore with limited theory; its optimality is unproven. (3) Interaction with normalization: for scale-invariant weights (those followed by a normalization layer), weight decay's true effect is an effective learning-rate modulation rather than capacity control (Zhang et al., van Laarhoven), complicating the regularization interpretation entirely. (4) Convergence guarantees: like Adam, AdamW lacks clean convergence proofs on non-convex problems matching SGD's, and Adam's original convergence proof was later found flawed (Reddi et al., AMSGrad). (5) Generalization theory: why decoupled decay generalizes better is still not rigorously characterized beyond the per-coordinate uniformity argument; the precise relationship to flat minima and implicit bias is open.