Video Generation Models

Video generation models are deep generative systems that learn to synthesize temporally coherent sequences of image frames conditioned on text, image, or other modalities, representing one of the most computationally and architecturally challenging frontiers in generative AI. They matter because they compress the full complexity of visual perception — spatial structure, motion dynamics, physical plausibility, and semantic consistency — into a single learned model, enabling applications from film production to world-model simulation. A counterintuitive fact: the hardest problem in video generation is not generating realistic individual frames, but enforcing temporal consistency — a model that generates perfect frames independently will still produce flickering, incoherent video unless it explicitly models inter-frame dependencies.

The core insight

Video generation is fundamentally a problem of learning a probability distribution over high-dimensional spatiotemporal tensors — a video is not a sequence of independent images, but a single object in $\mathbb{R}^{T \times H \times W \times C}$ space where time, height, width, and channel dimensions are all jointly dependent. The "aha moment" is realizing that modern video generators succeed by factorizing this intractable joint distribution into manageable pieces: a spatial model (typically a 2D or 3D UNet / transformer) combined with a temporal model (causal attention, recurrent layers, or temporal convolutions) trained together end-to-end, often inside a compressed latent space to keep compute tractable. Every architectural choice — 3D convolutions vs. pseudo-3D, full 3D attention vs. spatial-then-temporal — is really a different bet on how to factorize space and time efficiently.

Common trapCommon Misconception

The most common mistake practitioners make is treating video generation as "image generation applied frame-by-frame" — running a diffusion model T times independently and stitching the results. This is wrong in two compounding ways: first, each frame samples independently from the noise distribution, so there is no mechanism to enforce inter-frame coherence, producing visible flicker and object teleportation; second, it ignores the fact that motion itself is a signal that must be modeled, not just appearance. The correct mental model is that the denoising network must receive the entire video tensor (or a temporal window of it) simultaneously, with attention or convolution kernels explicitly spanning the time axis so that frame $t$ is denoised conditioned on frames $t-1$ and $t+1$ . Temporal consistency is not a post-processing step — it must be baked into the forward pass.

How It Works

The Video as a Spatiotemporal Tensor

A raw video clip is a tensor $\mathbf{x} \in \mathbb{R}^{T \times H \times W \times C}$ where $T$ is the number of frames, $H \times W$ is the spatial resolution, and $C$ is the number of channels (typically 3 for RGB). Even at modest resolution — say $16$ frames of $256 \times 256$ — this is a 3.1M-dimensional object, making direct pixel-space generation computationally catastrophic. The near-universal solution is to first encode video into a compressed latent space using a pretrained video VAE, then generate in that latent space.

Video Latent Diffusion

The dominant paradigm today is latent diffusion applied to the spatiotemporal tensor. A video encoder $\mathcal{E}$ maps $\mathbf{x}$ to $\mathbf{z} = \mathcal{E}(\mathbf{x}) \in \mathbb{R}^{T' \times h \times w \times d}$ , typically with $8\times$ spatial downsampling and optional temporal compression. The diffusion forward process adds Gaussian noise:

\quad \mathbf{z}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{z}_0 + \sqrt{1 - \bar{\alpha}_t}\,\boldsymbol{\epsilon}, \quad \boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})

Explanation: At diffusion timestep $t$ , the clean latent $\mathbf{z}_0$ is interpolated toward pure Gaussian noise; $\bar{\alpha}_t$ is the cumulative noise schedule controlling the signal-to-noise ratio. Here $\bar{\alpha}_t = \prod_{s=1}^{t}(1 - \beta_s)$ is the cumulative product of the noise schedule $\{\beta_s\}$ . The reverse process learns to denoise by training a neural network $\boldsymbol{\epsilon}_\theta(\mathbf{z}_t, t, \mathbf{c})$ where $\mathbf{c}$ is the conditioning signal (e.g., CLIP text embeddings):

\quad \mathcal{L}_{\text{LDM}} = \mathbb{E}_{\mathbf{z}_0, \boldsymbol{\epsilon}, t, \mathbf{c}}\bigl[\|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta(\mathbf{z}_t, t, \mathbf{c})\|_2^2\bigr]

Explanation: The model is trained to predict the noise added to the latent at each diffusion timestep; minimizing this MSE loss is equivalent to maximizing a variational lower bound on the data log-likelihood.

Temporal Attention and 3D Architectures

The core architectural challenge is extending 2D spatial operations to handle the time axis. Two dominant strategies exist:

Pseudo-3D (factorized) attention decomposes full 3D self-attention into separate spatial and temporal passes. For a feature map $\mathbf{F} \in \mathbb{R}^{T \times h \times w \times d}$ , we reshape and apply:

\quad \text{Attn}_{\text{spatial}}(\mathbf{F}) = \text{softmax}\!\left(\frac{\mathbf{Q}_s \mathbf{K}_s^\top}{\sqrt{d_k}}\right)\mathbf{V}_s, \quad \mathbf{Q}_s, \mathbf{K}_s, \mathbf{V}_s \in \mathbb{R}^{(T \cdot hw) \times d_k}

Explanation: Spatial attention flattens the spatial dimensions while treating each frame as a separate batch element, so each pixel attends only to other pixels within the same frame. A causal or bidirectional temporal attention layer then operates across frames at each spatial location:

\quad \text{Attn}_{\text{temporal}}(\mathbf{F}) = \text{softmax}\!\left(\frac{\mathbf{Q}_t \mathbf{K}_t^\top}{\sqrt{d_k}} + \mathbf{M}\right)\mathbf{V}_t, \quad \mathbf{Q}_t, \mathbf{K}_t, \mathbf{V}_t \in \mathbb{R}^{(hw \cdot T) \times d_k}

Explanation: Temporal attention lets each spatial position attend across all frames; the mask $\mathbf{M}$ (set to $-\infty$ for future frames in autoregressive settings) enforces causality when needed.

Classifier-Free Guidance for Video

Video generators universally use classifier-free guidance (CFG) to boost prompt adherence. During training, the conditioning $\mathbf{c}$ is randomly dropped with probability $p_{\text{uncond}}$ . At inference, the effective score is:

\quad \hat{\boldsymbol{\epsilon}}_\theta(\mathbf{z}_t, t, \mathbf{c}) = \boldsymbol{\epsilon}_\theta(\mathbf{z}_t, t, \varnothing) + w\bigl[\boldsymbol{\epsilon}_\theta(\mathbf{z}_t, t, \mathbf{c}) - \boldsymbol{\epsilon}_\theta(\mathbf{z}_t, t, \varnothing)\bigr]

Explanation: The unconditional prediction is extrapolated toward the conditional prediction by guidance scale $w > 1$ , trading sample diversity for fidelity to the text prompt — higher $w$ produces sharper, more prompt-faithful but less diverse videos.

PyTorch Sketch: Temporal Attention Block

```python import torch import torch.nn as nn class TemporalAttention(nn.Module): """Causal temporal self-attention over T frames at each spatial position.""" def __init__(self, dim: int, n_heads: int = 8): super().__init__() self.n_heads = n_heads self.head_dim = dim // n_heads self.qkv = nn.Linear(dim, 3 * dim, bias=False) self.proj = nn.Linear(dim, dim) def forward(self, x: torch.Tensor) -> torch.Tensor:

# x: (B, T, H, W, C) B, T, H, W, C = x.shape x_flat = x.reshape(B H W, T, C) # spatial pos as batch qkv = self.qkv(x_flat) # (BHW, T, 3*C) q, k, v = qkv.chunk(3, dim=-1)

# Reshape for multi-head attention def reshape_heads(t): return t.view(B H W, T, self.n_heads, self.head_dim).transpose(1, 2) q, k, v = map(reshape_heads, (q, k, v))

# Causal mask mask = torch.triu( torch.full((T, T), float('-inf'), device=x.device), diagonal=1 ) attn = (q @ k.transpose(-2, -1)) / self.head_dim**0.5 + mask attn = attn.softmax(dim=-1) out = (attn @ v).transpose(1, 2).reshape(B H W, T, C) out = self.proj(out) return out.reshape(B, T, H, W, C) ```

researcher · Jonathan Ho | b. 1990 | Denoising Diffusion Probabilistic Models (DDPM)

Jonathan Ho is a research scientist at Google DeepMind who completed his PhD at UC Berkeley under Pieter Abbeel, specializing in deep generative models and reinforcement learning. His 2020 paper "Denoising Diffusion Probabilistic Models" established the modern DDPM framework that virtually all video diffusion models build upon, and he directly extended it to video in the 2022 paper "Video Diffusion Models" which introduced the 3D UNet architecture for spatiotemporal denoising. Surprisingly, Ho's original DDPM paper was initially rejected from ICLR 2020 before becoming one of the most influential generative modeling papers of the decade with thousands of citations.

researcher · Uriel Singer | b. ~1995 | Make-A-Video: Text-to-Video without Text-Video Data

Uriel Singer is a research scientist at Meta AI whose 2022 paper "Make-A-Video" introduced a landmark approach to text-to-video generation by bootstrapping from a pretrained text-to-image model (trained on image-text pairs) and then learning motion priors separately from unlabeled video data — elegantly sidestepping the scarcity of text-video paired data. The key insight was that spatial appearance and semantic grounding could be inherited from the image model, while only temporal dynamics needed to be learned from video. This decoupled training strategy has influenced nearly every subsequent large-scale video generation system.

Figure

Video Latent Diffusion Architecture

Figure

Factorized Spatial + Temporal Attention

Intuition

The Mental Model

Imagine you are a film director with total amnesia between shots — if you could only look at one frame at a time while directing, your film would be visually beautiful but completely incoherent: actors would teleport, physics would reset, objects would change color randomly. Video generation models face exactly this problem if they process frames independently. The solution is like giving the director a storyboard that shows all frames simultaneously — the model can then ensure that the character's hand reaching toward the door in frame 8 actually grasps the handle in frame 12. In diffusion terms, this means the noise being removed from frame $t$ is informed by what neighboring frames look like, so all frames are denoised together in a coordinated choreography rather than independent performances. The VAE compression is like rehearsing the storyboard in thumbnail sketches before committing to full resolution — it's the same story, just cheaper to work with.

Historical Development

Origins · 1 minhow this was discovered

History

2014

Variational Autoencoders (Kingma & Welling) and GANs (Goodfellow et al.) establish foundational deep generative frameworks that video models would later inherit.

2016

Video Pixel Network (Kalchbrenner et al.) generates video autoregressively pixel-by-pixel using PixelCNN, demonstrating feasibility but at impractically slow inference speeds.

2018

MoCoGAN (Tulyakov et al.) decomposes video generation into separate content and motion generators using GANs, pioneering the idea of disentangled spatiotemporal representations.

2020

DDPM (Ho et al.) re-establishes diffusion models as state-of-the-art generative models for images, laying the direct algorithmic foundation for all subsequent video diffusion work.

2022

Video Diffusion Models (Ho et al., Google Brain) introduces the 3D UNet for video, achieving the first high-quality diffusion-based video generation with factorized space-time attention.

2022

Make-A-Video (Singer et al., Meta AI) and Imagen Video (Ho et al., Google) demonstrate billion-parameter text-to-video generation, establishing the latent diffusion + text conditioning paradigm at scale.

2023

Stable Video Diffusion (Blattmann et al.,