Continuous Action Spaces and Gaussian Policies

A stochastic policy over a continuous action space $\mathcal{A} \subseteq \mathbb{R}^d$ cannot be represented as a table or a softmax over discrete outcomes, because there is no way to normalize a sum over infinitely many actions. Instead, the policy is represented as a probability density function $\pi_\theta(a \mid s)$, and the most common choice is the multivariate Normal (Gaussian) density:

where $\mu_\theta(s) \in \mathbb{R}^d$ is the mean action predicted by a neural network, and $\Sigma_\theta(s)$ is a covariance matrix controlling exploration spread. In the overwhelming majority of practical implementations, $\Sigma_\theta(s)$ is restricted to be diagonal, $\Sigma_\theta(s) = \mathrm{diag}(\sigma_1^2, \dots, \sigma_d^2)$, so each action dimension gets its own independent standard deviation. This turns the joint density into a product of univariate Gaussians, one per action dimension, which is far cheaper to evaluate, sample, and differentiate than a full covariance matrix. Two design decisions recur across implementations. First, whether $\sigma_\theta(s)$ is state-dependent (output by the network alongside $\mu_\theta(s)$) or a single global learned parameter shared across all states — the latter is simpler and often used in early training stages of algorithms like PPO and TRPO. Second, how to keep $\sigma$ positive: rather than letting the network output $\sigma$ directly (which could go negative), the network typically outputs $\log \sigma$ (unconstrained) and exponentiates it, or outputs a raw value passed through a softplus function. This is the same "parameterize in log-space" trick used throughout probabilistic deep learning to keep scale parameters positive without clipping. The other core idea is the reparameterization trick: instead of treating sampling as a black-box stochastic operation that blocks gradient flow, we write the sampled action as a deterministic, differentiable function of the network outputs plus independent noise,

This makes $\partial a / \partial \theta$ well-defined, which is essential both for the score-function estimator used in REINFORCE / actor-critic methods, and for the pathwise gradient estimator used in algorithms like Soft Actor-Critic that differentiate a critic's output directly through the sampled action.

Step 1 — network forward pass. A shared or separate neural network maps the state $s$ to raw outputs: a mean vector $\mu_\theta(s)$, and either a log-std vector $\log\sigma_\theta(s)$ or a single global log-std parameter vector $\log\sigma_\theta$.

Step 2 — construct the diagonal Gaussian. Exponentiate to get $\sigma_\theta(s) = \exp(\log\sigma_\theta(s))$, clipped to a sane range (e.g. $[\sigma_{\min}, \sigma_{\max}]$) to avoid numerical blow-up.

Step 3 — sample via reparameterization. Draw $\epsilon \sim \mathcal{N}(0, I)$ and set $a_{\text{raw}} = \mu_\theta(s) + \sigma_\theta(s)\odot\epsilon$.

Step 4 — squash if the action space is bounded. If the true action range is $[-1, 1]^d$ (or any bounded box), pass $a_{\text{raw}}$ through $\tanh$: $a = \tanh(a_{\text{raw}})$, then rescale to the environment's actual bounds. This is necessary because a Gaussian has unbounded support, but almost every physical actuator has hard limits.

Step 5 — compute the log-probability, with the change-of-variables correction if squashed. For the unsquashed Gaussian,

If a squashing function $g$ (e.g. $\tanh$) is applied, the change-of-variables formula for densities requires subtracting the log-determinant of the Jacobian of $g$:

Step 6 — plug into the policy gradient. For an actor-critic update, the log-probability's gradient with respect to $\theta$ is combined with an advantage estimate $A(s,a)$ (from Policy Parameterization and Objectives):

Because everything above is differentiable in closed form, this expectation can be estimated with ordinary automatic differentiation over sampled trajectories. ```text function GAUSSIAN_POLICY_STEP(state s, θ): // Step 1: forward pass through policy network μ, log_σ ← PolicyNetwork(s; θ) // both vectors in ℝ^d log_σ ← clip(log_σ, LOG_SIGMA_MIN, LOG_SIGMA_MAX) σ ← exp(log_σ) // Step 2 & 3: reparameterized sample ε ← sample_standard_normal(d) a_raw ← μ + σ ⊙ ε // Step 4: squash into bounded action range (optional, if actions bounded) a_squash ← tanh(a_raw) a_env ← rescale(a_squash, action_low, action_high) // Step 5: log-probability with Jacobian correction logp_gauss ← -sum( (a_raw - μ)^2 / (2 σ^2) + log_σ + 0.5 log(2π) ) jac_corr ← sum( log(1 - a_squash^2 + ε_stable) ) // ε_stable avoids log(0) logp ← logp_gauss - jac_corr return a_env, logp, μ, σ

function ACTOR_CRITIC_UPDATE(trajectory batch, θ, critic φ): for each (s, a, r, s') in batch: a_env, logp, μ, σ ← GAUSSIAN_POLICY_STEP(s, θ) // during rollout: sample and store logp // ... environment executes a_env, returns r, s' ... for each stored (s, a, logp, r, s') in batch: V_s ← critic(s; φ) V_s_next ← critic(s'; φ) δ ← r + γ V_s_next - V_s // TD error / advantage estimate actor_loss ← actor_loss - logp stop_gradient(δ) // policy-gradient term critic_loss← critic_loss + δ^2 // value regression θ ← θ - η_actor ∇θ(actor_loss / batch_size) φ ← φ - η_critic ∇φ(critic_loss / batch_size) return θ, φ ``` The pseudocode separates the two concerns cleanly: `GAUSSIAN_POLICY_STEP` is a pure, reusable subroutine for turning a state into a differentiable sample and its log-density, and `ACTOR_CRITIC_UPDATE` shows how that log-density is consumed exactly as in the discrete-action case — the only thing that changed relative to a softmax policy is how $\log\pi_\theta(a\mid s)$ is computed.

Time complexity. For a diagonal Gaussian over $d$ action dimensions, sampling requires drawing $d$ standard normal variates and $O(d)$ elementwise arithmetic, so sampling is $O(d)$. Evaluating the log-density is likewise $O(d)$: a sum over $d$ independent terms, each a constant number of arithmetic operations. This is asymptotically identical to evaluating a softmax over $|\mathcal{A}|$ discrete actions when $d \approx \log|\mathcal{A}|$, but critically it does not scale with the size of the action space — for a full covariance matrix instead of a diagonal one, sampling and density evaluation cost $O(d^3)$ due to the Cholesky factorization needed to sample correlated Gaussians and to compute the determinant in the density; this is the core reason diagonal covariance is preferred in practice for $d$ in the tens or hundreds. Backpropagating gradients through the reparameterized sample and log-density adds only a constant-factor overhead to the $O(d)$ forward cost, since automatic differentiation is linear in the size of the computational graph.

Space complexity. The policy network needs an extra $O(d)$ output head (or $O(d)$ extra parameters for a global log-std vector) beyond whatever discrete-action network it is compared against, and storing the sampled action, noise, and log-density per trajectory step costs $O(d)$ per timestep — no additional memory beyond ordinary trajectory storage.

Correctness argument. The correctness of using a Gaussian policy rests on two facts, both of which must be verified rather than assumed. Fact 1 — the policy gradient theorem does not care about the family of $\pi_\theta$. The policy gradient theorem states

and this identity holds for any differentiable density $\pi_\theta(\cdot\mid s)$, discrete or continuous, as long as $\pi_\theta$ is differentiable in $\theta$ and has support covering the actions actually taken. The Gaussian is simply one valid choice of $\pi_\theta$; the proof of the theorem (via the log-derivative trick $\nabla_\theta \pi_\theta = \pi_\theta \nabla_\theta \log \pi_\theta$) never assumes discreteness. So plugging in a Gaussian's closed-form log-density is a legitimate instantiation, not an approximation. Fact 2 — the reparameterization trick preserves the expectation exactly. We need $\mathbb{E}_{a\sim\pi_\theta(\cdot\mid s)}[f(a)] = \mathbb{E}_{\epsilon\sim\mathcal{N}(0,I)}[f(\mu_\theta(s)+\sigma_\theta(s)\odot\epsilon)]$ for any function $f$, so that differentiating the right-hand side with respect to $\theta$ (which is legal because the randomness $\epsilon$ no longer depends on $\theta$) correctly estimates the gradient of the left-hand side. This equality is precisely the definition of an affine transformation of a standard Normal: if $\epsilon \sim \mathcal{N}(0,I)$ then $\mu + \sigma\odot\epsilon \sim \mathcal{N}(\mu,\mathrm{diag}(\sigma^2))$, a standard fact from the closure of Gaussians under affine maps. So the reparameterized sample has exactly the target distribution, not merely approximately — no bias is introduced by this substitution. Fact 3 — the squashing correction is forced by the change-of-variables formula, not a heuristic. If $a = g(u)$ for an invertible, differentiable $g$ (e.g. $\tanh$), and $u$ has density $p_U$, then the density of $a$ is $p_A(a) = p_U(g^{-1}(a))\,\left|\det \frac{\partial g^{-1}}{\partial a}\right|$, equivalently $\log p_A(a) = \log p_U(u) - \log\left|\det \frac{\partial g}{\partial u}\right|$. For elementwise $\tanh$, $\frac{\partial g}{\partial u_i} = 1-\tanh^2(u_i)$, giving exactly the correction subtracted in Step 5 above. Omitting this term produces a log-probability that does not correspond to any valid density over the squashed action — the policy gradient computed from it is then simply wrong, not merely noisier, which is why this is flagged as the central pitfall below.

Tradeoffs. Diagonal-covariance Gaussians sacrifice the ability to represent correlated action dimensions (e.g. two joints that should move together), trading expressiveness for the $O(d)$ versus $O(d^3)$ speedup. State-independent (global) $\sigma$ is simpler and more stable early in training but cannot adapt exploration to how confident the agent is in a given state, a limitation that state-dependent $\sigma_\theta(s)$ or later methods with entropy-regularized objectives address at the cost of extra network capacity and additional variance in the gradient estimator.

Consider a 1-D torque-control task where the true action range is $[-2, 2]$ N·m. Suppose at state $s$ the policy network (after a forward pass) outputs raw mean and log-std:

We draw a single standard normal sample $\epsilon = 0.80$ (fixed here for a reproducible trace).

(Sanity check: since we scaled $\tanh$'s output by 2, a fully rigorous treatment would add one more constant Jacobian term $-\log 2$ per action dimension for the final rescale — a fixed constant that does not depend on $\theta$ and so does not affect the policy gradient, which only needs $\nabla_\theta \log\pi_\theta$; it is nonetheless required if you want an exact log-likelihood, e.g. for comparing likelihoods across models.) Now suppose the critic estimates a TD error $\delta = r + \gamma V(s') - V(s) = +1.20$ (the action turned out better than expected). The actor loss contribution for this single transition is $-\log\pi_\theta(a\mid s)\cdot\delta = -(0.1932)(1.20) = -0.2318$. Backpropagating this scalar through the computational graph built in Steps 1–5 pushes $\theta$ in the direction that increases $\log\pi_\theta(a\mid s)$ — i.e., increases the density the policy assigns to actions like $1.55$ N·m in states like $s$ — exactly the intended effect of a positive advantage. Notice what would happen without the Jacobian correction: the reported log-probability would be $-0.7333$ instead of $0.1932$, a difference of nearly $0.93$ nats for this single sample — large enough to substantially distort the gradient direction and magnitude, especially when $a_{\text{raw}}$ is large and $\tanh$ is deep in its saturated region (illustrated next).

The Gaussian policy grew out of the broader tradition of stochastic approximation and policy-gradient methods that treat action selection as sampling from a parameterized distribution, tracing back to Ronald Williams's REINFORCE algorithm, which established the score-function estimator that any differentiable stochastic policy — Gaussian included — relies on.

Gaussian policies are the default action representation in almost every modern continuous-control deep-RL algorithm: Proximal Policy Optimization (PPO), Trust Region Policy Optimization (TRPO), and Soft Actor-Critic (SAC) all parameterize their actor as a diagonal Gaussian (SAC additionally applying the tanh squash and Jacobian correction described here). These algorithms power robotic manipulation and locomotion research, simulated physics benchmarks (MuJoCo, PyBullet, Isaac Gym), and increasingly real hardware — quadruped and humanoid locomotion controllers, dexterous manipulation policies, and autonomous vehicle low-level control loops that must output continuous steering, throttle, and braking commands. In finance and operations research, Gaussian policies appear in continuous portfolio-allocation and inventory-control RL formulations where actions are fractional quantities. Understanding this representation is also the conceptual stepping stone to Deterministic Policy Gradients, where the stochastic Gaussian sampling is replaced by a deterministic mean output plus externally injected exploration noise, trading the closed-form score function for a lower-variance but off-policy-only gradient estimator — a tradeoff that only makes sense once the mechanics and costs of the Gaussian policy are fully understood.

Ask the tutor: How does the choice between state-dependent and global standard deviation change the bias-variance tradeoff of the policy gradient estimator in practice?