The Advantage Function

In a Markov Decision Process with policy $\pi$, the state-value function $V^\pi(s) = \mathbb{E}_\pi[\,G_t \mid S_t = s\,]$ gives the expected return from state $s$ under $\pi$, while the action-value function $Q^\pi(s,a) = \mathbb{E}_\pi[\,G_t \mid S_t=s, A_t=a\,]$ gives the expected return from taking action $a$ in $s$ and following $\pi$ thereafter. The advantage function is defined as their difference:

Because $V^\pi(s) = \sum_a \pi(a\mid s)\, Q^\pi(s,a)$, the advantage is exactly the deviation of $Q^\pi(s,a)$ from its $\pi$-weighted average over actions at $s$. Consequently, $\mathbb{E}_{a\sim\pi(\cdot\mid s)}[A^\pi(s,a)] = 0$ for every state $s$: on average, an action is neither better nor worse than the policy itself, by construction. This zero-centering is the key structural property that makes $A$ so useful — it isolates the relative merit of each action, stripped of the state's overall level of return. This idea connects directly to the concept of a Baselines and Variance Reduction in the policy-gradient theorem. Recall that the vanilla policy gradient is

and that subtracting any state-only baseline $b(s_t)$ from $G_t$ leaves the gradient unbiased while typically reducing its variance, since $\mathbb{E}_{a_t}[\nabla_\theta \log\pi_\theta(a_t\mid s_t)\, b(s_t)] = 0$. Choosing $b(s) = V^\pi(s)$ turns the term $G_t - b(s_t)$ into (in expectation) exactly the advantage $A^\pi(s_t,a_t)$. This is not an arbitrary choice of baseline: $V^\pi(s)$ is the action-independent part of $Q^\pi(s,a)$, so subtracting it is the natural way to remove state-level variance while retaining all information about which action is better than another. This connects the advantage function tightly to Actor-Critic Methods, where a learned critic estimates $V(s)$ (or, in some variants, $Q(s,a)$ directly) precisely to compute this quantity online.

The practical challenge is that $Q^\pi$ and $V^\pi$ are unknown and must be estimated from experience. The standard actor-critic recipe avoids learning two separate networks by exploiting the Bellman equation for $Q^\pi$:

Substituting a single sampled transition $(s,a,r,s')$ gives a one-sample, unbiased estimate of $Q^\pi(s,a)$ as $r + \gamma V^\pi(s')$. Plugging this into the advantage definition:

which is exactly the TD error $\delta$. This is the crucial simplification: a single learned value function $V_\phi(s)$ suffices to estimate the advantage, with no need to separately learn $Q$. The TD error is an unbiased estimator of $A^\pi(s,a)$ given a perfect critic $V_\phi = V^\pi$; with an imperfect (learned, bootstrapped) critic it is a biased but typically much lower-variance estimator, since it uses only a one-step lookahead instead of the full Monte Carlo return. The resulting actor-critic training loop alternates policy improvement (actor update) using $\hat A$ and value-function fitting (critic update) using TD targets. ``` Algorithm: One-Step Advantage Actor-Critic (A2C-style) Input: policy network pi_theta, value network V_phi, learning rates alpha_theta, alpha_phi, discount gamma Initialize theta, phi arbitrarily for each episode: s <- initial state while s is not terminal: a ~ pi_theta( . | s) # sample action s', r, done <- environment.step(a)

# --- compute advantage estimate via TD error --- if done: target <- r else: target <- r + gamma * V_phi(s') delta <- target - V_phi(s) # delta ≈ A(s,a)

# --- critic update: regress V_phi(s) toward target --- phi <- phi + alpha_phi delta grad_phi V_phi(s)

# --- actor update: policy gradient scaled by advantage --- theta <- theta + alpha_theta delta grad_theta log pi_theta(a | s) s <- s' ``` Two implementation details matter for correctness. First, the TD target `target` must be computed using a stopped-gradient copy of `V_phi(s')` — the critic parameters used inside the target should not receive gradient flow from the target term itself, otherwise the regression is not a valid semi-gradient TD update (this mirrors the target-network discipline used in value-based methods). Second, in batched/parallel implementations (e.g. A3C, A2C), the same `delta` is reused for both the actor and critic updates, which is efficient but means any critic bias directly contaminates the actor's gradient — motivating more careful multi-step estimators later.

Time and space complexity. The advantage estimate via TD error costs $O(1)$ per transition beyond the forward pass of the critic network, since it is a single subtraction and scalar multiply once $V_\phi(s)$ and $V_\phi(s')$ are computed. If the critic is a neural network with $P$ parameters, each update costs $O(P)$ for the forward and backward pass, identical order to a plain value-based or REINFORCE-style update — the advantage machinery adds no asymptotic overhead. Space is $O(P_\theta + P_\phi)$ for actor and critic parameters, plus $O(1)$ additional storage for the scalar advantage per transition (versus $O(T)$ if full Monte-Carlo returns for an episode of length $T$ were stored and processed for a REINFORCE-style baseline).

Why the baseline doesn't bias the gradient. This is the central correctness argument and deserves a careful derivation. Start from the policy-gradient theorem in its state-value form:

For any function $b(s)$ that does not depend on $a$,

since $\sum_a \pi_\theta(a\mid s) = 1$ for every $\theta$. Therefore

for any choice of $b(s)$, including $b(s) = V^\pi(s)$. This proves that replacing $Q^\pi(s,a)$ with $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$ inside the policy-gradient expectation changes nothing in expectation — the estimator remains exactly unbiased. This is why the advantage function is not a heuristic fix but a mathematically justified reparameterization of the same gradient.

Why it (typically) reduces variance. Writing the per-sample gradient estimator as $g(s,a) = \nabla_\theta\log\pi_\theta(a\mid s)\cdot X(s,a)$ for some scalar $X$, the variance of $g$ scales (holding the log-prob factor fixed in distribution) with $\mathrm{Var}[X(s,a)]$. Since $A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)$ subtracts off the state-dependent mean of $Q^\pi(s,\cdot)$ under $\pi$, it satisfies $\mathrm{Var}_{a}[A^\pi(s,a)] \le \mathrm{Var}_a[Q^\pi(s,a) - c]$ for a constant baseline $c$, and more importantly removes cross-state variance that raw returns $G_t$ carry (different states can have wildly different typical returns, all irrelevant to which action is better within a state). It can be shown that the variance-minimizing baseline within the class of state-dependent baselines is not exactly $V^\pi(s)$ but a $Q$-weighted quantity; however $V^\pi(s)$ is extremely close to optimal in practice and is vastly simpler to estimate, which is why it is the standard choice.

Tradeoff: bias from bootstrapping. The one-step TD error $\delta_t$ used in practice is an unbiased estimate of $A^\pi(s,a)$ only if $V_\phi = V^\pi$ exactly. Since $V_\phi$ is itself learned and imperfect, $\delta_t$ carries bias proportional to the critic's approximation error, in exchange for a large reduction in variance relative to full Monte-Carlo advantage estimates ($G_t - V_\phi(s_t)$, which have zero bootstrapping bias but much higher variance because they sum many stochastic future rewards). This precise bias–variance knob — one-step TD versus full Monte Carlo — is exactly what motivates blending multiple horizons, the idea developed fully in Generalized Advantage Estimation.

Consider a tiny two-state MDP: a robot at position $s_0$ (far from goal) or $s_1$ (adjacent to goal), with actions `advance` and `wait`, discount $\gamma = 0.9$, and reward $+10$ upon reaching the goal, $0$ otherwise. Suppose the critic has already learned (approximately correct) value estimates:

The agent is in $s_0$ and samples action `advance`, transitioning to $s_1$ with reward $r=0$.

The advantage is strongly positive: advancing from $s_0$ led to a state ($s_1$) worth much more than $s_0$ was estimated to be worth, so `advance` outperformed the policy's expectation at $s_0$. The actor update increases $\log \pi_\theta(\text{advance}\mid s_0)$ proportionally to $+4.1$. Now suppose instead the agent had sampled `wait` at $s_0$, staying at $s_0$ with reward $0$:

Here the advantage is mildly negative: waiting is slightly worse than average at $s_0$, so its log-probability is nudged down. Notice the contrast with what a raw-return-based (no baseline) REINFORCE update would do: both actions eventually lead to a terminal reward of $+10$ several steps later, so both trajectories would receive large positive returns $G_t > 0$, and both `advance` and `wait` would have their probabilities pushed up — failing to distinguish that `advance` is markedly better. The advantage function is precisely what recovers this distinction, since it references each action's outcome against the state's own expected value rather than against zero.

The advantage function traces back to foundational work connecting dynamic programming and reinforcement learning, formalized clearly once the actor-critic architecture and the policy-gradient theorem were established.

The advantage function underlies nearly every modern deep policy-gradient system. Trust-Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) — used to train large-scale robotic control policies and, notably, to fine-tune large language models via reinforcement learning from human feedback — both scale their surrogate objectives by advantage estimates rather than raw returns, because stable, low-variance credit assignment is essential when updates must remain conservative. Asynchronous Advantage Actor-Critic (A3C) and its synchronous counterpart A2C were among the first deep-RL algorithms to train competitively on Atari benchmarks using many parallel actors, each computing local advantage estimates. In robotics, advantage-based actor-critic methods are used to train locomotion and manipulation policies where reward signals are sparse and credit assignment across long horizons is otherwise intractable. Even value-based methods borrow the idea: the Dueling DQN architecture explicitly decomposes its Q-network output into separate value and advantage streams, directly exploiting the identity $Q(s,a) = V(s) + A(s,a)$ for more sample-efficient learning. Understanding the advantage function is therefore a prerequisite for understanding essentially all contemporary policy-optimization algorithms.

Ask the tutor: How does Generalized Advantage Estimation blend multiple n-step TD errors to further control the bias–variance tradeoff in advantage estimation?