Deterministic Policy Gradients

A stochastic policy $\pi_\theta(a\mid s)$ assigns a probability density to every action; the standard policy gradient theorem requires integrating over this density, $\int_\mathcal{A} \pi_\theta(a\mid s)\,\nabla_\theta \log \pi_\theta(a\mid s)\, Q^\pi(s,a)\, da$, which is estimated by sampling actions. A deterministic policy instead is a function $\mu_\theta: \mathcal{S} \to \mathcal{A}$ that outputs one action per state with no randomness. This looks restrictive — how do you explore, and how do you differentiate through a delta function? — but the DPG theorem (Silver et al., 2014) shows that the deterministic policy still admits a clean gradient formula, and that this formula is the natural limit of the stochastic policy gradient as the policy's variance shrinks to zero. The key quantity is the action-value gradient, $\nabla_a Q^\mu(s,a)$, evaluated at the action the current policy would take, $a = \mu_\theta(s)$. Because the policy is deterministic, the performance objective $J(\theta) = \mathbb{E}_{s\sim d^\mu}[Q^\mu(s,\mu_\theta(s))]$ depends on $\theta$ only through $\mu_\theta(s)$, so the chain rule gives

This expectation is only over the state-visitation distribution $d^\mu$, never over actions — the single structural fact that makes DPG lower-variance than stochastic policy gradient methods (compare to Actor-Critic Methods, where the analogous expression additionally integrates the log-probability score function over actions). In practice $Q^\mu$ is unknown, so it is replaced by a learned critic $Q_w(s,a)$, giving an off-policy deterministic actor-critic: the critic learns $Q_w \approx Q^\mu$ by temporal-difference learning from data generated by any behavior policy (typically $\mu_\theta$ plus exploration noise), while the actor is updated by ascending $\nabla_a Q_w$ through $\mu_\theta$. This is exactly the mechanism behind Deep Deterministic Policy Gradient (DDPG), which pairs this update rule with deep-network function approximators, experience replay, and target networks borrowed from Deep Q-Networks.

Setup. The actor is a neural network $\mu_\theta(s)$ mapping states directly to continuous actions (often passed through a $\tanh$ to bound the range). The critic is a separate network $Q_w(s,a)$ estimating the action-value of the actor's action. Both are trained jointly but with different loss signals: the critic minimizes a Bellman TD-error, the actor maximizes the critic's output.

Exploration. Since $\mu_\theta$ is deterministic, exploration requires artificially injecting noise into the behavior policy used to collect data — commonly Ornstein-Uhlenbeck noise (temporally correlated, good for physical control) or simple Gaussian noise added to the action before it is applied to the environment:

This exploration noise is not part of the policy being optimized; it only shapes the data-collection distribution, which is legitimate because DPG is an off-policy method — the critic's TD target uses the greedy action $\mu_\theta(s')$, not the noisy behavior action, so learning is decoupled from exploration exactly as in Q-Learning.

Critic update. The critic is trained by minimizing the mean-squared TD error against a bootstrapped target computed with target networks $\mu_{\theta'}, Q_{w'}$ (slowly-updated copies of the actor and critic) to stabilize the moving bootstrap target:

Actor update. The actor is updated by the deterministic policy gradient, using the current critic $Q_w$:

Target network update. Targets are updated by Polyak averaging (soft update) rather than periodic hard copies, which further smooths the bootstrap target:

``` Algorithm: Deep Deterministic Policy Gradient (DDPG) Input: environment, learning rates alpha_actor, alpha_critic, soft-update rate tau, discount gamma, replay capacity C, minibatch size N, exploration noise process N_t Initialize critic Q_w(s,a) and actor mu_theta(s) with random weights Initialize target networks: w' <- w, theta' <- theta Initialize replay buffer D with capacity C for episode = 1, 2, ... do initialize random process N_t for exploration observe initial state s_1 for t = 1, 2, ..., T do select action a_t = mu_theta(s_t) + N_t # exploration execute a_t, observe reward r_t and next state s_{t+1} store transition (s_t, a_t, r_t, s_{t+1}) in D sample random minibatch of N transitions (s_i, a_i, r_i, s_i') from D

# --- critic update --- a_i' <- mu_theta'(s_i') # target action y_i <- r_i + gamma Q_w'(s_i', a_i') # TD target L(w) <- (1/N) sum_i ( Q_w(s_i, a_i) - y_i )^2 w <- w - alpha_critic * grad_w L(w)

# --- actor update (deterministic policy gradient) --- grad_theta J <- (1/N) * sum_i grad_a Q_w(s_i, a) |_{a = mu_theta(s_i)}

• grad_theta mu_theta(s_i)

theta <- theta + alpha_actor * grad_theta J

# --- soft update of target networks --- w' <- tau w + (1 - tau) w' theta' <- tau theta + (1 - tau) theta' s_t <- s_{t+1} end for end for ``` Notice the two backward passes stitched together in the actor update: first differentiate $Q_w$ with respect to its action input (not its weights), producing a vector $\nabla_a Q_w$ of the same dimension as the action space, then multiply this by the Jacobian $\nabla_\theta \mu_\theta$ of the actor network. This is ordinary backpropagation through a composed function $Q_w(s, \mu_\theta(s))$, treating the critic as fixed during the actor's gradient computation.

Per-step time complexity. Each environment step triggers one forward pass through the actor $O(|\theta|)$, one minibatch of $N$ transitions through the critic for the TD target $O(N \cdot |w|)$, one critic gradient step $O(N \cdot |w|)$, and one actor gradient step requiring a forward-backward pass through both critic and actor, $O(N\cdot(|w|+|\theta|))$. With fixed network sizes this is $O(N)$ per environment step, matching other minibatch-based off-policy deep-RL methods such as Deep Q-Networks; the constant factor is larger than plain DQN because DDPG maintains four networks (actor, critic, and their two targets) instead of two.

Space complexity. $O(|\theta| + |w| + C\cdot d)$, where $C$ is replay buffer capacity and $d$ is the size of one transition tuple $(s,a,r,s')$; the buffer dominates for large $C$, exactly as in any experience-replay method.

Why the deterministic policy gradient formula is correct. The theorem's core claim is

and it can be justified in two complementary ways. Direct differentiation. Write the performance objective using the recursive Bellman-style decomposition of the discounted state distribution,

Differentiating with respect to $\theta$ requires care because $d^\mu$ itself depends on $\theta$ (the policy determines which states are visited). The DPG theorem's proof (structurally identical to the stochastic Policy Gradient Theorem's proof) shows that the terms coming from $\nabla_\theta d^\mu$ telescope through the Bellman recursion and vanish from the final gradient expression, leaving only the direct-effect term $\nabla_\theta \mu_\theta(s)\nabla_a Q^\mu(s,a)$ integrated against the (now fixed, for gradient purposes) state distribution. This mirrors exactly how the score-function term $\nabla_\theta \log \pi_\theta$ absorbs the effect of $d^\pi$'s dependence on $\theta$ in the stochastic case, so the two theorems share the same proof skeleton. Limiting argument. Consider a stochastic policy $\pi_{\theta,\sigma}(a\mid s) = \mu_\theta(s) + \mathcal{N}(0,\sigma^2 I)$, a Gaussian centered at the deterministic policy's output. The ordinary stochastic policy gradient theorem applies to $\pi_{\theta,\sigma}$ for any $\sigma > 0$:

Under smoothness assumptions on $Q$ and $\mu_\theta$, taking $\sigma \to 0$ makes this expression converge exactly to the deterministic gradient formula above; this is the formal statement (proved in the original DPG paper) that the deterministic policy gradient is the zero-noise limit of the stochastic one. This is also the intuitive reason DPG is lower-variance: the stochastic gradient estimator's variance grows as $\sigma$ shrinks relative to the number of samples needed, but the deterministic estimator removes the action-sampling term altogether rather than sending its variance to infinity, because it is derived directly rather than as a noisy limit at implementation time.

On-policy vs. off-policy correctness. DDPG's critic is trained off-policy — the TD target bootstraps with $\mu_{\theta'}(s')$, the current (target) policy's action, regardless of what noisy action was actually taken. This is legitimate because the Bellman equation for $Q^\mu$ only requires the next action to follow $\mu$, not the action actually stored in the replay buffer; the stored action $a_t$ only needs to be an admissible sample from the environment's dynamics, which off-policy replay satisfies. This mirrors why Q-Learning is off-policy: the target uses $\max_a Q(s',a)$ (or here, $Q(s', \mu(s'))$) rather than the behavior policy's action.

Tradeoffs. The gain in gradient variance comes at real costs: (1) $Q_w$ must be differentiable in $a$ and reasonably accurate over the entire action space, not just at visited actions, because the actor update queries $\nabla_a Q_w$ at points possibly far from data; a poorly-fit critic creates a false gradient signal the actor will happily climb. (2) Exploration is decoupled from the policy's own randomness, so it must be hand-engineered (noise process, scale, decay schedule) rather than emerging naturally as in stochastic actor-critic. (3) The deterministic policy has no natural mechanism to represent multimodal optimal behavior — if two very different actions are equally good, $\mu_\theta$ must arbitrarily commit to one region, unlike a Gaussian or richer distribution.

Consider a 1-D continuous-control toy: state $s\in[-1,1]$ is a position error, action $a\in[-1,1]$ is a force, reward $r(s,a) = -s^2 - 0.1a^2$, dynamics $s_{t+1} = \text{clip}(s_t + 0.1a_t,\,-1,1)$, $\gamma = 0.9$. Suppose the critic has (locally) learned $Q_w(s,a) \approx -s^2 - 0.1a^2 + 5as$ near the current operating point (a toy linearization capturing some coupling between $s$ and $a$), and the actor is currently a linear function $\mu_\theta(s) = \theta \cdot s$ with $\theta = 0.4$. Take a minibatch of one transition for clarity: $s = 0.5$.

Step 1 — actor forward pass.

Step 2 — critic action-gradient. Differentiate the local critic model with respect to $a$:

At $(s,a) = (0.5, 0.2)$:

Step 3 — actor Jacobian. Since $\mu_\theta(s) = \theta s$,

Step 4 — deterministic policy gradient (single sample estimate).

Step 5 — actor update. With learning rate $\alpha_\text{actor} = 0.01$:

Meanwhile the critic separately performs its own TD update: suppose the observed reward is $r = -0.5^2 - 0.1(0.2)^2 = -0.254$, next state $s' = 0.5 + 0.1(0.2) = 0.52$, and the target actor gives $a' = \mu_{\theta'}(s') = 0.39\times 0.52 = 0.2028$ (using slightly-lagged target weight $\theta'=0.39$). If the target critic evaluates $Q_{w'}(s', a') = -0.1$, then

and the critic weights are nudged to reduce $(Q_w(s,a) - y)^2$ by gradient descent — a completely separate loss from the actor's update, run on the same minibatch. This illustrates the two-headed nature of the algorithm: one loss shapes $Q_w$ towards Bellman-consistency, a different, independent gradient path pushes $\mu_\theta$ to climb whatever $Q_w$ currently believes.

DDPG and its descendants are the default family for continuous-control benchmarks such as MuJoCo locomotion tasks, robotic arm manipulation, and simulated driving, where action spaces are naturally continuous rather than discretized. Industrial applications include robotic grasping and manipulation pipelines that require smooth, precise torque or velocity commands rather than discretized bins, and process-control settings (HVAC, chemical plants, power-grid balancing) where continuous set-points must be optimized under delayed, noisy feedback. The core mechanism — a differentiable critic providing a direct gradient signal through the actor's action output — is reused, with substantial modifications, in TD3 (which patches DDPG's overestimation bias with twin critics and delayed updates) and informs the deterministic-policy variants used inside Soft Actor-Critic implementations. Understanding DPG also clarifies the broader relationship between Actor-Critic Methods, The Advantage Function as a variance-reduction device for stochastic gradients, and the special case where variance reduction is instead achieved structurally by removing action-sampling from the policy entirely.

:::faq[How does DDPG handle bounded action spaces, like torques limited to [-1,1]?] The actor network typically ends in a $\tanh$ nonlinearity (or scaled sigmoid) to squash its raw output into the valid range before it is sent to the environment. This has a side effect on the gradient: when the pre-activation output saturates near the boundary, the $\tanh$ derivative approaches zero, which can stall the deterministic policy gradient exactly in the regions where the optimal action is at the boundary. Careful initialization and action-space scaling are needed to avoid the actor getting stuck near saturation early in training.

Ask the tutor: How does clipped double-Q learning in TD3 concretely change the critic update compared to vanilla DDPG?