Adam Optimizer

Adam (Adaptive Moment Estimation) is a first-order stochastic optimization algorithm that maintains exponentially decaying running averages of both the gradient (first moment) and its element-wise square (second moment), using these to compute per-parameter adaptive learning rates. Key result: combines the benefits of momentum and RMSProp with bias-corrected moment estimates, achieving robust convergence with minimal hyperparameter tuning. Where it applies: training deep neural networks, large-scale machine learning, non-convex stochastic optimization. Who/when: Diederik P. Kingma and Jimmy Ba, 2014 (ICLR 2015).

The Adam optimizer is an algorithm for first-order gradient-based optimization of stochastic objective functions, designed to be computationally efficient, memory-frugal, and invariant to diagonal rescaling of gradients. It updates each parameter using estimates of the first moment (the mean) and the second raw moment (the uncentered variance) of the gradients, normalizing the step so that effective updates have magnitude approximately bounded by the learning rate. Adam matters because it has become the de facto default optimizer in deep learning, dramatically reducing the need for learning-rate scheduling on a wide class of problems. A surprising fact: despite its empirical dominance, the original convergence proof for Adam in the convex online setting was later shown to be flawed, and counterexamples exist where Adam fails to converge — motivating variants like AMSGrad. The method elegantly unifies Momentum and RMSProp into a single adaptive scheme with principled bias correction.

The Core Insight

Different parameters in a high-dimensional model often have gradients of vastly different scales and statistics. Adam estimates, per coordinate, both the typical direction of descent (via an exponential moving average of gradients — the first moment) and the typical magnitude of fluctuations (via an exponential moving average of squared gradients — the second moment). Dividing the smoothed gradient by the square root of the smoothed squared gradient yields a signal-to-noise normalized step: coordinates with consistent gradients move confidently, while noisy coordinates are damped automatically. Bias correction repairs the systematic underestimation that arises from initializing the moving averages at zero.

Formal Statement

Adam Update Rule

Let $f (θ)$ be a stochastic objective with parameters $θ \in R^{d}$ , and let $g_{t} = \nabla_{θ} f_{t} (θ_{t - 1})$ be the stochastic gradient at step $t$ . Adam maintains two exponential moving averages, $m_{t}$ (first moment) and $v_{t}$ (second raw moment), and updates parameters as follows.

The first moment estimate:

m_{t} = β_{1} m_{t - 1} + (1 - β_{1}) g_{t}

⟳this is an exponential moving average; expand recursively to see

m_{t} = (1 - β_{1}) \sum_{i = 1}^{t} β_{1}^{t - i} g_{i}

The second raw moment estimate (element-wise square $g_{t}^{2} = g_{t} ⊙ g_{t}$ ):

v_{t} = β_{2} v_{t - 1} + (1 - β_{2}) g_{t}^{2}

⟳identical EMA structure applied to squared gradients;

v_{t} = (1 - β_{2}) \sum_{i = 1}^{t} β_{2}^{t - i} g_{i}^{2}

Bias-corrected estimates:

\overset{m}{^}_{t} = \frac{m _{t}}{1 - β _{1}^{t}}, \overset{v}{^}_{t} = \frac{v _{t}}{1 - β _{2}^{t}}

⟳take expectation of

m_{t}

assuming stationary gradients;

E [m_{t}] = (1 - β_{1}^{t}) E [g_{t}]

, so divide by

(1 - β_{1}^{t})

to remove bias.

The parameter update:

θ_{t} = θ_{t - 1} - α \frac{m ^ _{t}}{v ^ _{t} + ϵ}

⟳divide the bias-corrected mean step by the bias-corrected RMS magnitude;

ϵ

prevents division by zero.

Symbols:

α

= step size (learning rate, default

1 0^{- 3}

);

β_{1} \in [0, 1)

= first-moment decay (default

0.9

);

β_{2} \in [0, 1)

= second-moment decay (default

0.999

);

ϵ

= numerical stabilizer (default

1 0^{- 8}

);

⊙

= Hadamard (element-wise) product; all divisions and square roots are element-wise. Initialization:

m_{0} = 0

v_{0} = 0

Effective Step Bound

Assuming $ϵ = 0$ , the magnitude of the effective step taken by Adam in each coordinate satisfies a bound:

∣ Δ θ_{t} ∣ = α \frac{m ^ _{t}}{v ^ _{t}} \leq α \cdot max (1, \frac{1 - β _{1}}{1 - β _{2}})

In the common case $1 - β_{1} \geq 1 - β_{2}$ (e.g. defaults: $0.1 \geq 0.0316$ ), the bound is $∣Δ θ_{t} ∣ \leq α$ , establishing a trust region independent of gradient scale.

⟳bound

∣ \overset{m}{^}_{t} ∣ \leq \frac{( 1 - β _{1} ) \sum β _{1}^{t - i} ∣ g _{i} ∣}{1 - β _{1}^{t}}

and

\overset{v}{^}_{t} \geq

a corresponding term; in the sparse-gradient limit the ratio

\frac{1 - β _{1}}{1 - β _{2}}

emerges from a single nonzero gradient.

Why It Holds / Derivation

Bias Correction Derivation

We show that the raw moving average $v_{t}$ underestimates the true second moment, and that the correction in equation [3] removes this bias exactly under a stationarity assumption.

Step 1. Unroll the recursion in equation [2] from

v_{0} = 0

v_{t} = (1 - β_{2}) i = 1 \sum t β_{2}^{t - i} g_{i}^{2}

⟳substitute

v_{t - 1}

repeatedly; the geometric weights

β_{2}^{t - i}

and the leading

(1 - β_{2})

factor follow by induction.

Step 2. Take the expectation, and assume the second moment is approximately stationary,

E [g_{i}^{2}] \approx E [g_{t}^{2}]

E [v_{t}] = (1 - β_{2}) E [g_{t}^{2}] i = 1 \sum t β_{2}^{t - i} + ζ = E [g_{t}^{2}] (1 - β_{2}^{t}) + ζ

where

ζ

is small if the stationarity assumption holds (it absorbs the drift of

E [g_{i}^{2}]

across time).

⟳the finite geometric series

\sum_{i = 1}^{t} β_{2}^{t - i} = \frac{1 - β _{2}^{t}}{1 - β _{2}}

cancels the leading

(1 - β_{2})

Step 3. The factor

(1 - β_{2}^{t}) < 1

is the source of bias — early in training (

t

small) it can be far below

1

, severely shrinking

v_{t}

. Dividing by it yields an unbiased estimate:

E [\overset{v}{^}_{t}] = E [\frac{v _{t}}{1 - β _{2}^{t}}] \approx E [g_{t}^{2}]

⟳divide equation [7] by

(1 - β_{2}^{t})

; the leading term becomes

E [g_{t}^{2}]

exactly. The identical argument applies to

\overset{m}{^}_{t}

■

Diederik P. Kingma & Jimmy Lei Ba | 2014

Diederik Kingma (then a PhD student at the University of Amsterdam under Max Welling) and Jimmy Ba (then at the University of Toronto under Geoffrey Hinton) introduced Adam in a paper submitted to ICLR 2015. Kingma is also a principal architect of the variational autoencoder (VAE) and normalizing flows; Ba contributed foundational work on layer normalization and attention. The name "Adam" derives from "Adaptive Moment estimation" and is not an acronym in the conventional sense, nor a person's name. Their paper provided both the practical algorithm and an attempted regret-based convergence analysis in the online convex optimization framework of Zinkevich.

Building Intuition

Picture rolling a ball down a noisy, anisotropic landscape. Momentum ( $m_{t}$ ) gives the ball inertia: it accumulates velocity in consistent directions, smoothing over the jitter of mini-batch noise. The second moment ( $v_{t}$ ) acts like a per-direction shock absorber: in directions where the gradient is large and erratic, $v_{t}$ is large, so the update is shrunk; in flat, quiet directions, $v_{t}$ is small, so the step grows.

The ratio $\overset{m}{^}_{t} / \overset{v}{^}_{t}$ is essentially a signal-to-noise ratio. When gradients agree across mini-batches, $\overset{m}{^}_{t} \approx \overset{v}{^}_{t}$ and the ratio is near $\pm 1$ — full-step confidence. When gradients are pure noise with zero mean, $\overset{m}{^}_{t} \to 0$ while $\overset{v}{^}_{t}$ stays positive, so the ratio shrinks and Adam automatically slows down. This is why Adam needs so little manual tuning: it self-normalizes its step to the local geometry. Bias correction matters most in the first few dozen steps, where without it the zero-initialized averages would make the algorithm take dangerously large early steps.

Adam's adaptive step vs. SGD on an ill-conditioned valley

Worked Examples

Single-Step Update with Numbers

Given: a single parameter

θ_{0} = 1.000

, first gradient

g_{1} = 0.5

, defaults

α = 1 0^{- 3}

β_{1} = 0.9

β_{2} = 0.999

ϵ = 1 0^{- 8}

, with

m_{0} = v_{0} = 0

. Step 1 — first moment:

m_{1} = 0.9 (0) + 0.1 (0.5) = 0.05

. Step 2 — second moment:

v_{1} = 0.999 (0) + 0.001 (0.25) = 0.00025

. Step 3 — bias correction (

t = 1

1 - β_{1}^{1} = 0.1

1 - β_{2}^{1} = 0.001

\overset{m}{^}_{1} = \frac{0.05}{0.1} = 0.5, \overset{v}{^}_{1} = \frac{0.00025}{0.001} = 0.25

Note that after correction,

\overset{m}{^}_{1} = g_{1}

and

\overset{v}{^}_{1} = g_{1}^{2}

exactly — the first step recovers the raw gradient and its square. Step 4 — update:

θ_{1} = 1.000 - 1 0^{- 3} \cdot \frac{0.5}{0.25 + 1 0 ^{- 8}} = 1.000 - 1 0^{- 3} \cdot \frac{0.5}{0.5} = 1.000 - 0.001

θ_{1} = 0.999

The effective step is exactly $α = 1 0^{- 3}$ — the trust-region bound of equation [5] is saturated on the very first step regardless of the gradient's actual size.

Noise Suppression Over Many Steps

Given: suppose the gradient at a coordinate is zero-mean noise: it equals

+ 1

on odd steps and

- 1

on even steps (so the true descent signal is zero, but the magnitude is constant). Compute the steady-state behavior with defaults. Second moment (steady state): since

g_{t}^{2} = 1

always,

v_{t} \to 1

, hence

\overset{v}{^}_{t} \to 1

. First moment (steady state):

m_{t}

oscillates around

0

. After many steps, the EMA of an alternating

\pm 1

with

β_{1} = 0.9

has amplitude

∣ m_{\infty} ∣ = \frac{( 1 - β _{1} )}{1 + β _{1}} = \frac{0.1}{1.9} \approx 0.0526

(geometric series of alternating signs). After bias correction (factor

\approx 1

at large

t

∣ \overset{m}{^}_{t} ∣ \approx 0.0526

. Effective step:

∣Δ θ ∣ = α \frac{∣ m ^ _{t} ∣}{v ^ _{t}} \approx 1 0^{- 3} \cdot \frac{0.0526}{1.0} \approx 5.3 \times 1 0^{- 5}

∣Δ θ ∣ \approx 5.3 \times 1 0^{- 5} ≪ α

Compare: a consistent gradient of $+ 1$ would give $∣Δ θ ∣ \approx α = 1 0^{- 3}$ , roughly 19× larger. Adam automatically suppresses the step in the noisy direction — its signal-to-noise adaptivity in action.

Common Misconceptions

Adam Is Not Guaranteed to Converge

A widespread belief is that Adam's published convergence proof guarantees convergence on all convex problems. This is false. Reddi, Kale & Kumar (2018, "On the Convergence of Adam and Beyond") identified a flaw in the original regret analysis: the proof incorrectly assumed a positive-semidefinite quantity that can in fact be negative. The effective learning-rate matrix $Γ_{t} = \frac{v ^ _{t}}{α _{t}} - \frac{v ^ _{t - 1}}{α _{t - 1}}$ need not be PSD because $\overset{v}{^}_{t}$ can decrease. They constructed simple convex problems where Adam converges to the worst possible solution.

A Convex Problem Where Adam Fails

Consider online linear optimization with the periodic loss sequence (period 3), $x \in [- 1, 1]$ :

f_{t} (x) = {C x - x if t mod 3 = 1 otherwise

for large constant

C > 2

. The optimal fixed decision is

x = - 1

(the rare large positive gradient

C

should dominate the two small negative gradients

- 1

). However, because Adam's

v_{t}

heavily damps the rare large gradient (its squared value

C^{2}

is averaged with weight

1 - β_{2}

and then divided by

\overset{v}{^}_{t}

, which is dominated by the frequent

- 1

contributions), the effective influence of the informative gradient is reduced, and Adam drifts toward

x = + 1

— the opposite of optimal. This non-vanishing regret is impossible for plain SGD with decaying steps. The fix, AMSGrad, enforces

\overset{v}{^}_{t} = max (\overset{v}{^}_{t - 1}, v_{t})

so the denominator is non-decreasing, restoring the PSD property.

Adam ≠ Best Generalization

Empirically, on some tasks (notably image classification with CNNs), well-tuned SGD-with-momentum generalizes better than Adam despite Adam achieving lower training loss faster. Wilson et al. (2017) argue adaptive methods can converge to flatter-but-worse minima or solutions with larger weight norm. This motivated decoupled weight decay (AdamW), where weight decay is applied directly to parameters rather than folded into the gradient, since standard L2 regularization interacts poorly with adaptive per-coordinate scaling.

Connections

Adam sits at the confluence of several optimization lineages. Setting $β_{2} \to 1$ and removing the second-moment normalization recovers classical Momentum (specifically, heavy-ball with EMA). Setting $β_{1} = 0$ and using only the second moment gives essentially RMSProp, Hinton's unpublished method. The element-wise division by $\overset{v}{^}_{t}$ approximates a diagonal preconditioner, linking Adam to Newton's Method and quasi-Newton ideas, though Adam uses only first-order information and a crude diagonal curvature surrogate. Adam is a member of the broader family of Stochastic Gradient Descent methods and is analyzed within Online Convex Optimization via regret bounds. Its bias-correction step has a clean interpretation in terms of Exponential Moving Average estimators. The relationship to Natural Gradient Descent is partial: $\overset{v}{^}_{t}$ resembles the diagonal of the empirical Fisher information, though this correspondence is only approximate and the subject of ongoing debate.

Real-World Applications

Transformer language models (GPT, BERT, T5): Adam and its variant AdamW are the standard optimizers for pretraining large transformers; the original BERT and GPT papers use Adam with warmup schedules. AdamW is the canonical choice for nearly all modern LLM training.
Generative Adversarial Networks (GANs): the DCGAN and StyleGAN families use Adam with reduced $β_{1}$ (often $0.5$ ) to stabilize the adversarial min-max dynamics.
Variational Autoencoders (VAEs): Kingma's own VAE work pairs naturally with Adam for optimizing the evidence lower bound (ELBO).
Reinforcement learning (PPO, DQN, A3C): Adam is the default optimizer in most deep-RL implementations due to its robustness to non-stationary objectives and noisy policy gradients.
Neural machine translation: the original "Attention Is All You Need" transformer uses Adam with a custom inverse-square-root learning-rate schedule and warmup.
Diffusion models (Stable Diffusion, DALL·E): trained with AdamW over very long horizons; Adam's adaptivity handles the heterogeneous gradient scales across U-Net depths.
Recommendation systems and sparse embeddings: Adam's per-coordinate scaling handles the extreme gradient sparsity of large embedding tables, where most parameters receive gradients only rarely.

Historical Development

1951 — Robbins & Monro introduce the stochastic approximation framework, the theoretical bedrock of all SGD methods.
1964 — Polyak introduces the heavy ball momentum method, the ancestor of Adam's first-moment term.
2011 — Duchi, Hazan & Singer publish AdaGrad, the first widely used per-coordinate adaptive method, accumulating the full sum of squared gradients.
2012 — Tieleman & Hinton present RMSProp (in a Coursera lecture, never formally published), replacing AdaGrad's accumulation with an exponential moving average — directly the second-moment idea Adam reuses.
2012 — Zeiler publishes AdaDelta, another EMA-based adaptive method addressing AdaGrad's vanishing learning rate.
December 2014 — Kingma & Ba post the Adam paper to arXiv; it is accepted to ICLR 2015, and rapidly becomes the most-cited optimization method in deep learning.
2016 — Dozat introduces Nadam, incorporating Nesterov-accelerated momentum into Adam.
2018 — Reddi, Kale & Kumar (ICLR) expose the convergence flaw and propose AMSGrad; Loshchilov & Hutter propose AdamW with decoupled weight decay.
2019–2021 — A wave of variants: RAdam (rectified variance), LAMB (layer-wise scaling for huge batches), AdaBelief, Lion; debate continues over Adam's generalization properties versus SGD.

Subtleties & Limits

Several deep questions about Adam remain only partially resolved. (1) Generalization gap: there is still no complete theory explaining when and why SGD generalizes better than Adam, nor a sharp characterization of the minima each method selects in non-convex landscapes. (2) Non-convex convergence guarantees: rigorous convergence rates for Adam (and variants) on smooth non-convex objectives require additional assumptions (e.g. bounded gradients, specific $β$ relationships) and the constants are often vacuous; the precise role of $ϵ$ in convergence is subtle. (3) The Fisher-information interpretation: whether $\overset{v}{^}_{t}$ should be viewed as a principled approximation to natural-gradient preconditioning, or merely an effective heuristic, is contested. (4) Hyperparameter coupling: the interaction between $α$ , $β_{2}$ , weight decay, and learning-rate warmup is poorly understood theoretically, despite enormous practical importance — RAdam's analysis of early-training variance is one partial answer. (5) Optimal $β$ scaling with batch size and sequence length for billion-parameter models remains largely empirical.