Offline Reinforcement Learning

In the standard Markov Decision Process formulation, an RL agent alternates between acting in an environment and updating its policy from the resulting transitions. Off-policy algorithms relax this slightly: they can reuse past transitions stored in a replay buffer, as long as the agent is still occasionally collecting fresh data with its current policy. Offline RL removes even that: the agent receives a static dataset

generated by one or more unknown behavior policies $\pi_\beta$, and must output a policy $\pi$ that performs well when actually deployed, without ever touching the environment during training. Only at test time, after training is frozen, does the policy get to act. The central technical difficulty is distributional shift. Any actor-critic method built on the Bellman backup

must evaluate $Q$ at actions $a' \sim \pi(\cdot \mid s')$ sampled from the current, evolving policy — not from $\pi_\beta$. Online, this is harmless because if $\pi$ drifts toward some novel action, the agent can execute it, observe the true reward, and correct the estimate. Offline, there is no such correction: the Q-function is a neural network trained only on $(s,a)$ pairs from $\mathcal{D}$, and when queried at an out-of-distribution (OOD) action $a'$, it extrapolates. Function approximators, especially trained with few samples near a query point, tend to overestimate rather than underestimate value in unseen regions, because the policy improvement step explicitly searches for actions that maximize the (possibly erroneous) $Q$. This creates a vicious cycle: the policy is pulled toward whatever action the critic mistakenly rates highly, the critic is never corrected because that action is never actually tried, and the error compounds across Bellman backups (which propagate errors from $s'$ back to $s$). The result, empirically and theoretically, is that naively offline-trained SAC/TD3-style agents often collapse to policies far worse than the behavior policy that produced the data — sometimes even failing catastrophically despite the dataset containing near-expert trajectories. The unifying idea behind essentially all successful offline RL algorithms is to introduce some form of pessimism or conservatism about actions and states not well-supported by the data. This can be done at the level of the policy (constrain $\pi$ to stay close to $\pi_\beta$, or restrict its support to dataset actions), at the level of the value function (penalize $Q$ for OOD actions so it never overestimates them), or at the level of a learned dynamics model (penalize transitions into states where the model is uncertain). All three families are attempts to answer the same question: how do we act well using only what the data can actually tell us, refusing to trust extrapolation we cannot verify?

We walk through the general offline actor-critic recipe, then specialize to two canonical fixes: a policy-constraint method (BCQ-style) and a conservative-value method (CQL-style), which illustrate the two dominant design philosophies.

Policy constraint approach. Restrict $\pi$ so that it puts negligible probability on actions unlikely under $\pi_\beta(a\mid s)$. Concretely, BCQ (Batch-Constrained Q-learning) trains a generative model $G_\omega(s)$ (e.g. a conditional VAE) to imitate $\pi_\beta$, samples a handful of candidate actions from it, perturbs them slightly with a learned residual, and picks the candidate with highest $Q$ — never allowing the policy to propose actions the generative model deems implausible.

Conservative value approach. Conservative Q-Learning (CQL) instead directly shapes the Bellman objective: it adds a regularizer that pushes down $Q(s,a)$ for actions sampled from the current policy (likely OOD) while pushing up $Q(s,a)$ for actions actually seen in the dataset. This yields a lower bound on the true value function for the policy being trained, so the policy optimizer can never be fooled into chasing a phantom high-value action. ```text ALGORITHM: Offline Actor-Critic with Conservative Regularization (CQL-style) INPUT: static dataset D = {(s, a, r, s')}, discount γ, penalty weight α INIT: critic networks Q_θ1, Q_θ2, target networks Q_θ1', Q_θ2' actor network π_φ for iteration = 1, 2, ... do sample minibatch B = {(s, a, r, s')} ~ D # never touches env

# ---- Critic update ---- for each (s, a, r, s') in B: a' ~ π_φ(· | s') # policy's own next action y = r + γ * min(Q_θ1'(s', a'), Q_θ2'(s', a')) # standard target (Clipped Double Q)

# standard Bellman regression loss L_bellman = mean over B of (Q_θi(s,a) - y)^2 for i = 1,2

# conservative penalty: push DOWN Q at policy-sampled actions,

# push UP Q at dataset actions for each s in B: a_pi ~ π_φ(· | s) # OOD-risk action a_data = a from dataset at this s L_cql_i = logsumexp_a Q_θi(s, a) - Q_θi(s, a_data) # soft-max over actions minus data action

# (in practice approximated via sampled actions,

# including a_pi and random actions) L_critic_i = L_bellman_i + α * mean over B of L_cql_i # i = 1, 2 update θ1, θ2 by gradient descent on L_critic_1, L_critic_2

# ---- Actor update ---- for each s in B: a_new ~ π_φ(· | s) # reparameterized sample L_actor = - mean over B of min(Q_θ1(s, a_new), Q_θ2(s, a_new)) (+ optional behavior-cloning term λ * ||a_new - a_data||^2 , as in TD3+BC) update φ by gradient descent on L_actor

# ---- Target update ---- θi' ← τ θi + (1-τ) θi' for i = 1, 2 # Polyak averaging, as in SAC/TD3 end for OUTPUT: policy π_φ (deployed only now — no further training from real interaction) ``` Two design choices deserve emphasis. First, the actor is never allowed to update using real environment rollouts — every gradient in the loop above touches only $\mathcal{D}$. Second, the conservative penalty term is what prevents the standard runaway overestimation: because $\log\sum_a \exp Q(s,a)$ (a soft maximum over actions) is penalized while $Q(s, a_{\text{data}})$ is rewarded, the critic is explicitly discouraged from assigning high value to any action other than the ones actually observed, unless the Bellman error strongly disagrees. Tuning $\alpha$ trades off conservatism against the tightness of the value estimate — too large and the policy becomes overly close to pure imitation; too small and overestimation returns. The simplest and often surprisingly strong baseline, TD3+BC, replaces the whole CQL-style penalty with a single behavior-cloning regularizer on the actor loss, $\lambda \|\pi_\phi(s) - a_{\text{data}}\|^2$, tuned so that the actor is nudged toward data-supported actions while still being allowed to exploit accurate $Q$ estimates near them.

Computational complexity. Per gradient step, offline actor-critic methods cost the same as their online counterparts: one forward/backward pass through two critic networks and one actor network per minibatch, i.e. $O(B \cdot P)$ time per update where $B$ is batch size and $P$ the network parameter count, with $O(P)$ space for weights plus $O(B)$ for the minibatch. Conservative penalties like CQL's logsumexp term require sampling $M$ extra actions per state (typically $M \in [10, 20]$, drawn from the policy and from a uniform proposal) to approximate the log-partition function, multiplying critic compute by roughly $M{+}1$ but leaving the asymptotic order unchanged, $O(B \cdot M \cdot P)$. Model-based approaches (MOPO, MOReL) add the one-time cost of training an ensemble of $K$ dynamics models, $O(K \cdot |\mathcal{D}| \cdot P_{\text{model}})$, but subsequent policy optimization runs on synthetic rollouts and never touches the real environment. In all cases, wall-clock cost is dominated by the number of gradient steps, exactly as in standard deep RL, and dataset size $|\mathcal{D}|$ enters only linearly (one pass over data per epoch), so the algorithms scale to million-transition offline datasets without asymptotic surprises.

Why naive off-policy RL fails offline — the argument. Consider the Bellman backup with function approximation. Define the Bellman error at $(s,a)$ as $\delta(s,a) = |Q_\theta(s,a) - \mathcal{T}^\pi Q_\theta(s,a)|$, where $\mathcal{T}^\pi$ is the Bellman operator for policy $\pi$. For $(s,a)$ pairs in $\mathcal{D}$, gradient descent drives $\delta$ toward zero because the loss directly supervises those points. For $(s',a')$ with $a' \sim \pi(\cdot\mid s')$ but not in $\mathcal{D}$, there is no supervision signal at all — the network's value there is whatever the inductive bias of the function class produces, and because the policy-improvement step is an explicit maximization ($\pi \leftarrow \arg\max_\pi \mathbb{E}[Q(s,\pi(s))]$), it is selected to find wherever the approximator errs high. This is not a rare accident but a systematic selection effect: over enough optimization steps, the policy is guaranteed to discover the approximator's blind spots, because it is searching precisely for actions where $Q$ looks best. Once such an action is baked into the policy, the target $y = r + \gamma Q(s', a')$ used for the next Bellman update is itself computed at this erroneous point, so the overestimation propagates backward through the Bellman recursion to earlier states — a compounding error whose magnitude, in the worst case, grows with the effective horizon $1/(1-\gamma)$.

Why the fixes are correct. Policy-constraint methods sidestep the problem by construction: if $\pi$'s support is provably a subset of $\{a : \pi_\beta(a\mid s) > \epsilon\}$, then every action ever queried during training has appeared (or is arbitrarily close to one that has appeared) in $\mathcal{D}$, so the Bellman target is always evaluated in a region with real supervision, and the standard convergence guarantees for fitted Q-iteration with bounded function-approximation error apply. Conservative-value methods take a different route: CQL's penalty is constructed so that, under mild assumptions on the critic's expressiveness, the learned $Q_\theta$ is a provable lower bound on the true $Q^\pi$ for the policy $\pi$ being optimized — formally, $\mathbb{E}_{a\sim\pi}[Q_\theta(s,a)] \le \mathbb{E}_{a\sim\pi}[Q^\pi(s,a)]$ for all states in the data distribution, in the tabular/linear-approximation limit. A pessimistic lower bound is safe to optimize against, because the worst that can happen is the policy underperforms its estimate — it can never be lured into an action whose true value is actually catastrophic while its estimated value looks artificially high, which is precisely the failure mode that plagues naive off-policy learning. Model-based pessimism achieves an analogous guarantee by penalizing reward with model disagreement/uncertainty, effectively constructing a pessimistic MDP whose optimal policy is guaranteed (under model-error bounds) to perform at least as well in the true MDP as in the pessimistic one.

Tradeoffs. All three families sacrifice something: policy constraints and behavior cloning terms bias the learned policy toward mimicking $\pi_\beta$, capping how far it can improve past the data-generating policy. Conservative value methods can be overly cautious, underestimating value even for good, well-covered actions, especially with small $\alpha$ mistuning. Model-based pessimism inherits the difficulty of learning accurate dynamics models and calibrated uncertainty in high-dimensional or stochastic environments. There is no free lunch: offline RL performance is fundamentally bounded by the coverage of $\mathcal{D}$ — no algorithm can reliably learn a good policy for state-action regions the data never visited, a limitation formalized in offline RL sample-complexity bounds that scale with the concentrability coefficient (a measure of how well $\mathcal{D}$'s state-action distribution covers the distribution induced by the optimal policy).

Consider a toy 1-D continuous-action bandit-like MDP with a single state $s_0$, action range $a \in [-1, 1]$, immediate reward $r(a) = -(a - 0.3)^2$, and the episode terminates after one step ($\gamma$ irrelevant here — treat it as $\gamma=0$ for clarity of exposition; true optimum is $a^\star = 0.3$). Suppose the behavior policy sampled actions only from $a \in [-0.5, -0.1] \cup [0.1, 0.5]$ (it never tried the region right at $0.3$ exactly, but did sample nearby, e.g. $a=0.28$). We fit a critic $Q_\theta(s_0, a)$ on 200 logged transitions.

The naive critic, trained with unconstrained fitted-Q, has never seen $a=0.9$ or $a=-0.95$ and its neural network extrapolates poorly, assigning them spuriously high value ($+0.55$, $+0.20$) simply due to how the network's activation pattern generalizes outside the training manifold. An unconstrained actor optimizing $\arg\max_a Q_\theta(a)$ will then confidently select $a \approx 0.9$ — a genuinely bad action with true reward $-0.36$, far worse than the near-optimal $0.28$ actually present in the data. This is the overestimation failure made concrete: no environment interaction ever corrects the belief, because the policy is only ever evaluated offline against this same faulty critic during training. The CQL-regularized critic, by contrast, has been explicitly penalized for assigning high value to OOD actions like $0.9$ and $-0.95$ (which are heavily sampled during the logsumexp penalty computation, since the actor initially proposes them too), pushing their estimated values down toward or below their true values, while dataset actions like $0.28$ retain accurate (if conservatively shaded) estimates. The resulting policy correctly selects an action near $0.28$, close to the true optimum $0.3$, matching almost exactly what a well-covered dataset supports — it cannot do better than $0.28$ because the data never demonstrated anything closer to $0.3$, illustrating the coverage bound: the algorithm is only as good as its data, but it is safe, never catastrophically fooled.

Offline RL synthesizes ideas from batch reinforcement learning, causal inference, and deep function approximation, crystallizing into its modern form only once deep RL's instability under distributional shift became impossible to ignore.

Offline RL is the practical bridge between reinforcement learning theory and domains where exploration is expensive or forbidden. In robotics, it enables policy improvement from large teleoperation or autonomous-collection logs without risking hardware on an untrained agent. In recommendation systems, it allows optimizing long-term engagement metrics from historical click/watch logs, rather than only the myopic supervised objective of predicting the next click. In healthcare, offline RL research explores treatment-policy learning from electronic health records, where live experimentation on patients is ethically off-limits, though deployment there remains cautious and largely research-stage. Autonomous driving stacks use offline RL and its close relative, offline imitation-augmented RL, to train and evaluate planning policies against enormous fleets' worth of logged driving data before any on-road testing. The conservative value-estimation ideas developed here (pessimism under uncertainty) also echo directly into Imitation and Inverse RL, where an agent likewise must generalize from a fixed set of expert demonstrations rather than free interaction, and into offline policy evaluation, the sister problem of estimating a policy's value from logs without deploying it at all.

Ask the tutor: How does the choice of the conservative penalty weight α in CQL trade off safety against how much the learned policy can improve over the behavior policy?