Dreamer: Learning World Models

Dreamer (and its successors DreamerV2, DreamerV3) is a model-based reinforcement learning agent that learns a latent dynamics model—called a world model—entirely from high-dimensional sensory observations, then trains both actor and critic entirely inside imagined rollouts of that model, never touching the real environment during policy optimization. It matters because it achieves data efficiency orders of magnitude beyond leading model-free algorithms on visual control tasks, demonstrating that sufficiently accurate world models can substitute for environment interaction during learning. Remarkably, DreamerV3 is the first algorithm to collect diamonds in Minecraft from scratch without human demonstrations or domain-specific rewards—a benchmark considered intractable for prior methods.

The core insight

The key insight is the decoupling of world model learning from behavior learning: the agent learns a compact, predictive latent representation of the world from real data, and then treats that learned simulator as a cheap, differentiable environment in which actor-critic gradients can be computed through time via backpropagation through imagined trajectories. This means policy gradients flow through the dynamics model itself rather than being estimated purely from Monte Carlo rollouts, dramatically reducing variance. The "aha moment" is realizing that if your world model is good enough, you can learn an excellent policy while spending almost no additional real-world interaction budget—imagination becomes the primary training signal.

How It Works

Dreamer decomposes learning into three interleaved phases: (1) fitting the Recurrent State Space Model (RSSM) world model to a replay buffer of experience, (2) imagining multi-step trajectories from the learned latent dynamics, and (3) training actor and critic on those imagined trajectories via backpropagation through time (BPTT).

The Recurrent State Space Model (RSSM)

The RSSM maintains a latent state $h_t$ (deterministic GRU hidden state) and a stochastic component $z_t$ , separating deterministic memory from stochastic uncertainty. The joint state $(h_t, z_t)$ forms the full model state.

\quad h_t = f_\phi(h_{t-1},\, z_{t-1},\, a_{t-1})

Explanation: The deterministic recurrent function $f_\phi$ (a GRU) updates the hidden state by conditioning on the previous latent state $z_{t-1}$ and action $a_{t-1}$ , creating a temporal memory that carries information across time steps.

Symbols: $h_t \in \mathbb{R}^d$ is the GRU hidden state at step $t$ ; $z_{t-1} \in \mathbb{R}^k$ is the previous stochastic latent; $a_{t-1}$ is the previous action; $\phi$ denotes world model parameters. The stochastic posterior (used during training) and prior (used during imagination) are:

\quad z_t \sim q_\phi(z_t \mid h_t,\, x_t), \qquad \hat{z}_t \sim p_\phi(\hat{z}_t \mid h_t)

Explanation: During training the posterior $q_\phi$ refines the latent using the actual observation $x_t$ , while during imagination the prior $p_\phi$ generates the next stochastic state from the recurrent state alone, enabling rollouts without real data.

Symbols: $q_\phi$ is the encoder/posterior network; $p_\phi$ is the transition prior; $x_t$ is the high-dimensional observation (e.g., image). In DreamerV2/V3 the stochastic latent $z_t$ is a vector of categorical distributions (straight-through gradients), not a Gaussian—this empirically reduces posterior collapse.

World Model Training Objective

The world model is trained by maximizing the Evidence Lower Bound (ELBO) over sequence data:

\quad \mathcal{L}_{\text{WM}} = \mathbb{E}_{q_\phi}\!\left[\sum_{t=1}^{T} \underbrace{\ln p_\phi(x_t \mid h_t, z_t)}_{\text{reconstruction}} + \underbrace{\ln p_\phi(r_t \mid h_t, z_t)}_{\text{reward}} - \underbrace{\beta\, D_{\mathrm{KL}}\!\left[q_\phi(z_t \mid h_t, x_t) \;\|\; p_\phi(z_t \mid h_t)\right]}_{\text{KL regularization}}\right]

Explanation: The objective simultaneously trains a decoder to reconstruct observations and rewards from latent states and pushes the posterior toward the prior, balancing representation fidelity against the compactness of learned dynamics.

Symbols: $p_\phi(x_t \mid h_t, z_t)$ is the observation decoder; $p_\phi(r_t \mid h_t, z_t)$ is the reward predictor; $\beta$ is the KL balance coefficient (DreamerV3 uses free bits and dynamic balancing); $D_{\mathrm{KL}}$ is the Kullback–Leibler divergence.

Behavior Learning via Imagined Rollouts

Starting from states in the replay buffer, the actor $\pi_\theta(a_\tau \mid h_\tau, z_\tau)$ is unrolled for $H$ imagination steps entirely within the world model, producing imagined trajectories $\{(h_\tau, z_\tau, a_\tau, \hat{r}_\tau)\}_{\tau=t}^{t+H}$ . The critic $V_\psi(h_\tau, z_\tau)$ estimates $\lambda$ -returns (a TD( $\lambda$ ) target combining bootstrapped and Monte Carlo estimates):

\quad V_\tau^\lambda = \hat{r}_\tau + \gamma \left[(1-\lambda)\, V_\psi(s_{\tau+1}) + \lambda\, V_{\tau+1}^\lambda\right], \quad V_{t+H}^\lambda = V_\psi(s_{t+H})

Explanation: The $\lambda$ -return interpolates between one-step TD (low variance, high bias) and full Monte Carlo returns (high variance, low bias) within the imagination horizon, giving a balanced advantage estimate for the actor gradient.

Symbols: $\hat{r}_\tau$ is the imagined reward; $\gamma$ is discount factor; $\lambda \in [0,1]$ controls bias-variance tradeoff (typically 0.95); $s_\tau = (h_\tau, z_\tau)$ is the imagined latent state. The actor is updated by maximizing:

\quad \mathcal{L}_\pi = \mathbb{E}_{\pi_\theta}\!\left[\sum_{\tau=t}^{t+H} V_\tau^\lambda\right]

Explanation: Actor gradients are backpropagated through the entire imagined trajectory—including through the differentiable world model dynamics—enabling low-variance, gradient-based policy improvement without returning to the real environment.

Symbols: The expectation is over imagined actions sampled from $\pi_\theta$ ; gradients flow through $h_\tau, z_\tau$ back through $f_\phi$ via straight-through estimation for the categorical latents.

DreamerV3 Innovations

DreamerV3 (2023, Hafner et al.) introduced several robustness improvements enabling single-hyperparameter training across wildly different domains:

Symlog encoding: Replaces raw reward/return magnitudes with $\text{symlog}(x) = \text{sign}(x)\ln(|x|+1)$ , allowing a single learning rate to handle rewards spanning many orders of magnitude.
Two-hot encoding of returns: The critic predicts a categorical distribution over a fixed support, avoiding scale sensitivity entirely.
KL balancing: Separately scales the representation loss and dynamics loss within the KL term to prevent posterior collapse.

researcher · Danijar Hafner | b. 1993 | Principal architect of Dreamer, DreamerV2, DreamerV3

Danijar Hafner completed his PhD at the University of Toronto and Google Brain, working at the intersection of latent variable models and reinforcement learning. He is the lead author of the entire Dreamer series—introducing the RSSM architecture in the original Dreamer paper (2019), scaling it with categorical latents and actor-critic improvements in DreamerV2 (2020), and achieving domain-agnostic general intelligence benchmarks in DreamerV3 (2023). Remarkably, Hafner developed much of the early Dreamer codebase as a solo researcher, and DreamerV3's Minecraft diamond result emerged from a system trained with identical hyperparameters across all 150+ benchmark tasks including Atari, DMControl, and ProcGen simultaneously.

researcher · Timothy Lillicrap | b. 1980 | Foundational model-based RL and actor-critic infrastructure

Timothy Lillicrap (DeepMind) is best known as a co-author of DDPG and the biological plausibility of backpropagation, but his earlier work on continuous control and latent-space imagination directly influenced the actor-critic framework Dreamer builds upon. His collaboration with David Silver on value-gradient methods helped establish the theoretical grounding for differentiating through learned dynamics models. Lillicrap trained as a computational neuroscientist before transitioning to deep RL, and his interest in biologically plausible learning algorithms informs an unconventional angle: he views world models as mechanistically analogous to the predictive coding framework hypothesized in the mammalian neocortex.

Figure

Dreamer Agent Architecture: World Model + Behavior Learning

Figure

RSSM Temporal Unrolling: Posterior vs Prior Pathways

Intuition

The Mental Model

Imagine a chess grandmaster who has played thousands of games and has built a deeply internalized mental model of how the board evolves. To improve, they no longer need to play real games; instead they close their eyes and simulate thousands of positions in their head, running through imagined continuations and learning which strategies succeed. Dreamer is precisely this: the RSSM is the grandmaster's internal chess engine (learned from real games stored in the replay buffer), and the actor-critic is trained entirely by watching the grandmaster's imagination unfold—never touching the real board during policy improvement. The critical subtlety is that this inner world model is learned, not hand-coded: it starts blank and bootstraps its own physics, rewards, and dynamics purely from pixels or sensor data. The categorical latent $z_t$ acts like a compressed "situation code"—a few dozen discrete categories capturing the essential state of the world, analogous to recognizing a chess position as belonging to a "Sicilian defense middle-game" rather than memorizing every piece position individually.

Historical Development

Origins · 1 minhow this was discovered

History

1990

Sutton introduces Dyna—the first principled framework for using a learned model to generate synthetic experience for policy improvement, conceptually anticipating imagination-based RL.

2013

Wierstra et al. develop the DRAW architecture and Kingma & Welling introduce VAEs, providing the variational inference tools that Dreamer's RSSM inherits.

2015

Oh et al. (DeepMind) demonstrate video prediction models for Atari, showing neural networks can predict high-dimensional game frames, though using them for planning proves difficult.

2018

World Models (Ha & Schmidhuber) trains a VAE + MDN-RNN world model and demonstrates that a linear policy trained entirely inside the dream can solve CarRacing—the direct conceptual ancestor of Dreamer.

2019

Dreamer (Hafner et al., arXiv:1912.01603) introduces the RSSM with joint deterministic-stochastic states and full actor