Soft Actor-Critic

In standard RL, the objective is to maximize expected discounted return, `E[Σ γ^t r_t]`. Maximum-entropy RL augments this with an entropy bonus at every timestep: ``` J(π) = E[ Σ_t γ^t ( r(s_t, a_t) + α H(π(·|s_t)) ) ] ``` where `H(π(·|s)) = -E_{a~π}[log π(a|s)]` is the Shannon entropy of the policy's action distribution at state `s`, and `α > 0` is a temperature parameter controlling how strongly exploration is rewarded relative to task reward. Setting `α = 0` recovers ordinary RL. As `α` grows, the policy is pushed toward higher-entropy (more random) action distributions even at some cost to expected reward. This objective has a clean interpretation: instead of finding the single optimal action at each state, the agent finds the least committed distribution over near-optimal actions. Ties between equally good actions are broken by spreading probability mass across them rather than arbitrarily picking one. This matters enormously in continuous control, where the "best" action estimate is often noisy and a policy that commits early to a specific action can get stuck exploiting a spurious peak in the learned value function. To optimize this objective off-policy, SAC introduces a soft value function and a soft Q-function that fold the entropy term directly into the Bellman recursion: ``` Q_soft(s, a) = r(s, a) + γ E_{s'}[ V_soft(s') ] V_soft(s) = E_{a~π}[ Q_soft(s, a) - α log π(a|s) ] ``` Iterating the soft Bellman backup `T^π` defined by these equations is a contraction in the sup-norm (the same style of argument used for the standard Bellman operator), so repeated application converges to a unique fixed point `Q_soft^π` — this is the theoretical anchor for soft policy evaluation. Given a soft Q-function, soft policy improvement updates the policy toward the Boltzmann distribution over Q-values, `π_new(·|s) ∝ exp(Q_soft^{π_old}(s,·)/α)`, projected onto a tractable policy class (e.g., Gaussian) via KL divergence minimization. Alternating these two steps is soft policy iteration, and it can be shown to monotonically improve the soft objective — the theoretical justification developed in the original SAC papers. SAC is the practical, function-approximation version of this idealized procedure: Q-networks trained by soft Bellman regression, and a policy network trained to minimize the KL projection, both optimized by stochastic gradient descent using minibatches sampled off-policy from a replay buffer.

SAC maintains four learned components: 1. Policy network `π_θ(a|s)`, a squashed (tanh) diagonal Gaussian outputting mean and log-std, enabling the reparameterization trick: `a = tanh(μ_θ(s) + σ_θ(s) ⊙ ε)`, `ε ~ N(0, I)`, so gradients flow through the sampled action back into `θ`. 2. Two Q-networks `Q_{φ1}, Q_{φ2}` (clipped double Q-learning, as in Double and Dueling DQN), each regressed toward a shared soft Bellman target using the minimum of the two target estimates to curb overestimation. 3. Two target Q-networks `Q_{φ1'}, Q_{φ2'}`, updated by Polyak (exponential moving average) averaging for target stability, exactly as in DDPG/TD3-style methods. 4. Temperature `α` (log-parameterized for positivity), optionally learned automatically to hit a target entropy `H̄` (typically `-dim(A)`, i.e., minus the action dimensionality). Training alternates environment interaction (collecting transitions with the current stochastic policy) with off-policy gradient updates sampled uniformly from a replay buffer `D`. ``` PSEUDOCODE: Soft Actor-Critic Initialize policy network π_θ Initialize Q-networks Q_φ1, Q_φ2 Initialize target networks φ1' ← φ1, φ2' ← φ2 Initialize replay buffer D ← ∅ Initialize log α (if auto-tuning entropy), target entropy H̄ ← -dim(A) for each environment step: a ~ π_θ(·|s) # sample action via reparameterization s', r, done ← env.step(a) D.store(s, a, r, s', done) s ← s' (reset if done) for each gradient step: (s, a, r, s', done) ← sample minibatch from D

# --- soft Bellman target --- a' ~ π_θ(·|s') logp' ← log π_θ(a'|s') q_target ← min(Q_φ1'(s', a'), Q_φ2'(s', a')) - α logp' y ← r + γ (1 - done) * q_target

# --- critic update: regress both Q-networks toward y --- L_Q1 ← ( Q_φ1(s, a) - y )^2 L_Q2 ← ( Q_φ2(s, a) - y )^2 φ1 ← φ1 - lr_Q ∇_φ1 L_Q1 φ2 ← φ2 - lr_Q ∇_φ2 L_Q2

# --- actor update: maximize soft Q minus entropy penalty --- ã ~ π_θ(·|s) # reparameterized sample, gradient flows to θ logp_ã ← log π_θ(ã|s) q_min ← min(Q_φ1(s, ã), Q_φ2(s, ã)) L_π ← α logp_ã - q_min # minimize this (= maximize entropy-regularized Q) θ ← θ - lr_π ∇_θ L_π

# --- temperature update (automatic entropy tuning) --- L_α ← -α ( logp_ã.detach() + H̄ ) log α ← log α - lr_α ∇_{log α} L_α

# --- target network Polyak update --- φ1' ← τ φ1 + (1 - τ) φ1' φ2' ← τ φ2 + (1 - τ) φ2' ``` Two subtleties make this pseudocode correct rather than merely plausible. First, the actor loss uses `ã` sampled fresh from the current policy at state `s` (not the stored replay action `a`), because the actor is being optimized, not merely evaluated — the replay action was chosen by an old policy and offers no usable gradient toward improving `θ`. Second, `logp_ã.detach()` in the temperature loss stops gradients from flowing into the policy through that term; `α`'s loss only adjusts `α` itself, pushing entropy up when it is below target and down when it is above.

Per-update time complexity. Each gradient step performs a constant number of forward/backward passes through networks of fixed size: one policy forward+backward, four Q-network forward passes (two online, two target), and (if enabled) one scalar update for `α`. If networks have `P` parameters total, one gradient step costs `O(P)` — the same asymptotic cost as any deep actor-critic method; SAC adds only a small constant factor (roughly 2×) over single-critic methods like vanilla DDPG, for a large gain in stability. Sampling a minibatch of size `B` from the replay buffer costs `O(B)` with an array-backed circular buffer. Overall per-step cost is `O(B·P)`, dominated by network computation, matching DQN-style methods rather than scaling with trajectory length as REINFORCE-style on-policy methods do.

Space complexity. The replay buffer stores up to `N` transitions of fixed size (state, action, reward, next state, done flag), costing `O(N · (dim(S) + dim(A)))`. Network parameter storage is `O(P)` for each of the policy, two critics, and two target critics. Because SAC is off-policy, `N` can be made very large (often `10^6`) without discarding data after each policy update, unlike on-policy methods that must throw away trajectories once the policy changes.

Why clipped double Q-learning is necessary. A single bootstrapped Q-function trained by TD regression tends to overestimate values because `max` (or here, the entropy-regularized analogue) of noisy estimates is itself a positively biased estimator of the max of the true values — the same overestimation problem motivating Double and Dueling DQN. SAC takes `min(Q_φ1, Q_φ2)` in the target, which biases estimates downward instead, a much safer failure mode: a critic that transiently underestimates values still allows policy improvement to proceed to correct it, whereas overestimation can create false optima the policy locks onto and cannot escape by gradient ascent alone.

Why the soft Bellman backup converges. Define the soft Bellman operator `(T^π Q)(s,a) = r(s,a) + γ E_{s'}[ E_{a'~π}[Q(s',a') - α log π(a'|s')] ]`. For fixed `π`, this is an affine map with contraction factor `γ < 1` in the sup-norm, exactly like the ordinary Bellman operator, because it is a convex combination (over `s', a'`) of Q-values plus a state-and-action-dependent constant, scaled by `γ`. The Banach fixed-point theorem guarantees a unique fixed point `Q_soft^π`, and repeated application of `T^π` converges to it from any starting `Q`. This licenses soft policy evaluation.

Why soft policy improvement helps. Given `Q_soft^{π_old}`, define `π_new = argmin_{π' ∈ Π} D_KL( π'(·|s) || exp(Q_soft^{π_old}(s,·)/α) / Z(s) )`. It can be shown that `Q_soft^{π_new}(s,a) ≥ Q_soft^{π_old}(s,a)` for all `(s,a)`, via the same style of telescoping/monotonicity argument used to prove policy iteration improves value in ordinary MDPs, adapted to the entropy-augmented reward `r(s,a) + α H`. Alternating evaluation and improvement (soft policy iteration) therefore produces a monotonically non-decreasing sequence of soft value functions, converging to a policy that is optimal for the maximum-entropy objective within the policy class — the theoretical guarantee that SAC's practical, function-approximated procedure is trying to approximate.

Honest tradeoffs. SAC's stability and sample efficiency come at real costs: it maintains five networks (policy, two critics, two targets) plus optionally a learned temperature, increasing memory and implementation complexity relative to on-policy PPO. The entropy bonus can make SAC slower to fully exploit a near-deterministic optimal policy at convergence unless temperature decays appropriately (auto-tuning usually handles this but is itself a moving target that can destabilize training if the target entropy is poorly chosen). Its off-policy replay buffer, while efficient, means SAC is sensitive to distributional shift if the buffer contains data far from the current policy's visitation — a concern that intensifies sharply in the fully offline setting of Offline Reinforcement Learning, where no fresh on-policy correction is possible at all.

Consider a 1-D continuous-control toy: state `s ∈ [-1, 1]` is a cart position, action `a ∈ [-1, 1]` is a force, reward `r(s,a) = -s^2 - 0.1 a^2` (stay near center, minimize effort), transition `s' = clip(s + 0.1a, -1, 1)`. Suppose at some point during training, for state `s = 0.5`:

Actor loss for this sample: ``` L_π = α · log π(a|s) − min(Q_φ1, Q_φ2) = 0.15 × (-0.62) − (-0.31) = -0.093 + 0.31 = 0.217 ``` Gradient descent on `L_π` with respect to `θ` pushes the policy to increase `Q(s,a)` (raise the second term's magnitude relative to loss) while also increasing entropy — a lower `|log π(a|s)|`, i.e., a less sharply peaked distribution, reduces the first term's contribution to loss. The reparameterized sampling path means `∂L_π/∂θ` can be computed by ordinary backpropagation through `μ_θ`, `σ_θ`, the fixed noise `ε`, and the `tanh` squashing function, exactly as in a supervised network — this is the crucial trick that lets SAC use low-variance gradients despite having a stochastic policy, avoiding the high-variance score-function (REINFORCE-style) estimator entirely for the entropy term.

Critic target for the same transition, with next state `s' = clip(0.5 + 0.1×(-0.2165), -1,1) ≈ 0.4783`, next sampled action `a' = -0.18` (drawn similarly), `log π(a'|s') = -0.55`, target critics giving `Q_φ1'(s',a') = -0.24`, `Q_φ2'(s',a') = -0.27`, reward `r = -(0.5)^2 - 0.1(-0.2165)^2 ≈ -0.2547`, discount `γ = 0.99`: ``` q_target = min(-0.24, -0.27) - α·log π(a'|s') = -0.27 - 0.15×(-0.55) = -0.27 + 0.0825 = -0.1875 y = r + γ · q_target = -0.2547 + 0.99×(-0.1875) = -0.2547 - 0.1856 = -0.4404 ``` Both `Q_φ1(s,a) = -0.31` and `Q_φ2(s,a) = -0.29` are regressed toward `y = -0.4404` via squared error, nudging both critics downward for this state-action pair — illustrating the soft Bellman target in action: the entropy term `-α log π(a'|s')` adds a small positive bonus (`+0.0825`) to the raw value estimate, rewarding the policy for keeping `a'` unlikely to have been near-deterministic.

Soft Actor-Critic was introduced by Tuomas Haarnoja and collaborators at UC Berkeley (advised by Sergey Levine and Pieter Abbeel's research groups), building on a line of work in maximum-entropy reinforcement learning that traces back to earlier energy-based and soft Q-learning formulations. The idea of treating control as probabilistic inference — where optimal trajectories correspond to a Boltzmann distribution over returns — had circulated in the RL and robotics literature for years before SAC packaged it into a stable, scalable deep RL algorithm with off-policy sample efficiency competitive with the best on-policy methods of the time.

SAC is a default choice for continuous-control robotics research because it learns competent policies with comparatively few real-world environment interactions, a scarce resource when trials involve physical hardware. It has been used to train real robotic arms for in-hand manipulation and locomotion policies for legged robots directly from onboard sensors, where sample efficiency and robustness to modeling error both matter. In simulation, SAC is a standard baseline across MuJoCo, DeepMind Control Suite, and PyBullet benchmarks, frequently used as the backbone actor-critic inside more complex systems: model-based RL pipelines that use SAC as the policy-improvement step on top of a learned dynamics model, multi-task and meta-RL frameworks that share a SAC-style critic across tasks, and autonomous driving simulators exploring continuous steering and throttle control. Its off-policy, replay-buffer-based design also makes it a natural starting point for adapting continuous-control ideas into the Offline Reinforcement Learning setting, where an agent must learn entirely from a fixed dataset without further environment access — a setting that inherits SAC's critic and entropy machinery but must additionally guard against the distributional shift that off-policy bootstrapping can introduce when no corrective on-policy data is ever collected.

Ask the tutor: How does automatic temperature tuning actually derive its gradient, and why does targeting `H̄ = -dim(A)` work well in practice?