Video Understanding and Temporal Modeling

A video is a tensor of shape $(T, H, W, C)$: $T$ frames, each an $H \times W$ image with $C$ channels. The central design question in video understanding is how to let a model's computation depend jointly on the spatial and temporal axes, rather than treating $T$ as an afterthought (e.g., simply averaging per-frame CNN outputs, a strategy usually called "late fusion" or a frame-pooling baseline). Three ideas dominate, corresponding to three different notions of "spatiotemporal receptive field":

3D convolution. Extend the 2D convolution kernel $k_h \times k_w$ used in image CNNs to a 3D kernel $k_t \times k_h \times k_w$ that slides over all three axes at once. The filter directly learns spatiotemporal patterns — e.g., an edge that rotates over 5 frames — the same way a 2D filter learns spatial edges. This is the most direct generalization of Convolutional Neural Networks to video and is the basis of architectures like C3D and I3D (Inflated 3D ConvNet).

Two-stream fusion. Instead of asking one network to learn appearance and motion simultaneously, split the problem. A spatial stream is a standard 2D CNN applied to RGB frames, capturing what objects and scenes look like. A temporal stream is a 2D CNN applied to a stack of precomputed optical flow fields, capturing how pixels are moving independent of appearance. The two streams are trained (semi-)independently and their class-score predictions are fused, typically by weighted averaging. This decomposition exploits the empirical fact that motion patterns (a golf swing) generalize across appearances (different clothes, backgrounds) better than raw pixels do, and it lets each stream specialize.

Temporal attention. Convolution — whether 2D or 3D — has a bounded receptive field per layer; long-range temporal dependencies (an action that begins in frame 1 and pays off in frame 90) require stacking many layers or heavy temporal downsampling to reach. Attention, following the transformer design used elsewhere in deep learning, computes a weighted combination of all other positions (frames, or spatiotemporal patches) for every query position, with weights learned from content similarity. This gives an $O(1)$-hop path between any two frames regardless of distance, at the cost of a computation that grows quadratically with the number of tokens attended over. These three ideas are not mutually exclusive — modern video transformers often use 3D-convolutional "tokenizers" (patch embeddings) feeding into an attention backbone, and two-stream ideas persist as an inductive bias (motion vs. appearance) inside single-stream models via features like frame differencing.

We walk through each family's mechanism precisely, since the differences are subtle and the exam-relevant details live in the specifics.

1. 3D Convolution. A 2D convolution kernel of shape $(C_{in}, k_h, k_w)$ produces one output channel by

A 3D convolution kernel of shape $(C_{in}, k_t, k_h, k_w)$ does the same over three axes:

Stacking such layers (as in C3D, using $3\times3\times3$ kernels throughout) builds a network whose receptive field grows in time as well as space with depth — early layers see a few adjacent frames, deep layers integrate information across the whole clip.

I3D (Inflated 3D ConvNet) solves a practical problem with training 3D CNNs from scratch: video datasets are far smaller than image datasets like ImageNet, so a network with $k_t \times k_h \times k_w$ parameters per filter (roughly $k_t\times$ as many as a 2D filter) is prone to overfitting and slow to converge. I3D's trick is to inflate a pretrained 2D CNN (e.g., Inception): take every 2D $k_h \times k_w$ filter, replicate it $k_t$ times along a new time axis, and divide by $k_t$ so the output magnitude on a constant-in-time input matches the original 2D filter's output. This gives the 3D network a principled ImageNet-pretrained initialization, then fine-tunes on video. ``` function INFLATE_2D_TO_3D(filter2d, k_t):

# filter2d has shape (C_out, C_in, k_h, k_w) filter3d = repeat filter2d along a new time axis, k_t times filter3d = filter3d / k_t # preserve output scale return filter3d # shape (C_out, C_in, k_t, k_h, k_w) function I3D_FORWARD(clip, network3d):

# clip: tensor (C, T, H, W) x = clip for layer in network3d.layers: x = conv3d(x, layer.weights, stride=layer.stride) x = batchnorm(x) x = relu(x) if layer.has_pool: x = pool3d(x, layer.pool_kernel) # pools space and/or time logits = global_average_pool(x) # collapse (T,H,W) -> vector logits = fully_connected(logits) return softmax(logits) ```

2. Two-Stream Networks. ``` function TWO_STREAM_PREDICT(rgb_frame, flow_stack, spatial_cnn, temporal_cnn):

# spatial_cnn: standard 2D CNN, input = one RGB frame (H,W,3) p_spatial = spatial_cnn(rgb_frame) # softmax scores

# temporal_cnn: 2D CNN whose first-layer input channels = 2L

# flow_stack: L consecutive optical-flow (dx,dy) fields stacked -> (H,W,2L) p_temporal = temporal_cnn(flow_stack) # softmax scores p_final = alpha p_spatial + (1 - alpha) p_temporal return p_final ``` Key details that matter:

• Optical flow input. For each pair of consecutive frames, an optical flow algorithm (classical, e.g. TV-L1, or a learned network) produces a horizontal displacement field $d_x$ and vertical field $d_y$, each the same size as the frame. Stacking $L$ (commonly $L=10$) consecutive flow pairs gives a $2L$-channel input that encodes short-term motion history, letting the temporal stream see motion dynamics, not just instantaneous velocity.
• Independent training. Each stream is trained with its own cross-entropy loss on the same action labels; the temporal stream in particular benefits from heavy data augmentation and sometimes multi-task pretraining because flow datasets are small.
• Late fusion. The simplest, and historically the most common, way to combine streams is a weighted average of softmax scores at test time (or an SVM over both streams' penultimate features). Later variants fuse features earlier ("early fusion" via a shared convolutional block) at the cost of losing the streams' ability to specialize independently.

3. Temporal Attention. Self-attention over a sequence of $N$ tokens computes, for each token, a weighted sum over all tokens' value vectors, with weights derived from query–key similarity:

In video, tokens can be (a) whole frames (frame-level temporal attention), or (b) spatiotemporal patches, as in video transformers (e.g., ViViT, TimeSformer) that tokenize a clip into $T \times H' \times W'$ patches. Because full attention over every spatiotemporal patch pair costs $O((T\cdot H' \cdot W')^2)$, practical designs factorize attention into cheaper passes:

• Divided (space-then-time) attention: each patch first attends only to patches at the same time step (spatial attention), then attends only to patches at the same spatial location across time (temporal attention). Cost drops from $O((THW)^2)$ to $O(T(HW)^2 + HW\cdot T^2)$.
• Non-local blocks: a lighter-weight attention module inserted into a CNN backbone, computing pairwise affinities between all spatiotemporal positions in a feature map, used to augment (not replace) convolutional features with long-range context.
• Temporal Shift Module (TSM): an attention-adjacent trick that shifts a fraction of channels forward/backward in time before a standard 2D convolution, giving the 2D conv an effective temporal receptive field at zero extra FLOPs — a cheap approximation to temporal mixing.

``` function DIVIDED_SPACE_TIME_ATTENTION(tokens):

# tokens: array indexed [t][s], t in 0..T-1 (time), s in 0..HW-1 (space) for t in 0..T-1: tokens[t][:] = SelfAttention(tokens[t][:]) # attend within frame t for s in 0..HW-1: column = [tokens[t][s] for t in 0..T-1] column = SelfAttention(column) # attend across time for t in 0..T-1: tokens[t][s] = column[t] return tokens ```

3D convolution. A layer with $C_{in}$ input channels, $C_{out}$ output channels, kernel $k_t\times k_h\times k_w$, applied to a clip of size $T\times H\times W$, costs

multiply–adds — exactly the 2D convolution cost multiplied by an extra factor of $k_t$ (the temporal kernel extent) and applied over an extra factor of $T$ output positions. This is linear in $T$: doubling clip length doubles compute, which is the expected, efficient scaling for a local operator. The parameter count per filter is also $k_t\times$ that of the corresponding 2D filter, which is precisely why I3D's inflation trick (reusing pretrained 2D weights) is valuable — training a $k_t$-times-larger filter bank from scratch on a comparatively small video dataset invites overfitting; inflation supplies a strong prior at zero extra labeled data. Correctness of inflation. If the input clip is temporally constant (every frame identical, mimicking a still image seen for $T$ frames), then a 3D filter built by replicating a 2D filter $k_t$ times and dividing by $k_t$ produces exactly the same output as applying the original 2D filter to a single frame. Proof: for a temporally constant input $x[c,t,h,w] = x_0[c,h,w]$ for all $t$,

(the $\tau$-sum over $k_t$ identical terms cancels the $1/k_t$). This guarantees the inflated network reproduces the pretrained 2D network's behavior on static input exactly, so early training only has to learn to deviate from that baseline to capture motion — a much easier optimization problem than learning 3D filters from a random initialization.

Two-stream networks. Each stream costs exactly as much as its underlying 2D CNN, run either once per frame (spatial stream) or once per flow stack (temporal stream); the extra cost is optical flow computation itself, which for classical methods (TV-L1) is comparable to or exceeds the CNN forward pass and is not end-to-end differentiable through the flow algorithm in the classic formulation (a key limitation — flow quality is a fixed, external hyperparameter rather than something the loss can improve, unless a learned differentiable flow network is substituted). Memory is modest since each stream is a standard 2D CNN; the design's efficiency advantage is that it reuses image-pretrained backbones for the spatial stream directly.

Temporal attention. Full self-attention over $N$ spatiotemporal tokens costs $O(N^2 d)$ for the $QK^\top$ product and softmax, where $d$ is the feature dimension; with $N = T\cdot H' \cdot W'$ patch tokens this is quadratic in clip length and frame resolution — the dominant practical bottleneck of video transformers. Divided space-time attention reduces this to

attending within-frame ($O((H'W')^2)$ per frame, $T$ frames) and across-time-at-fixed-location ($O(T^2)$ per spatial position, $H'W'$ positions) separately — asymptotically cheaper than joint attention whenever $T$ and $H'W'$ are both reasonably large, since the joint cost $O(T^2(H'W')^2)$ dominates both terms. Why attention captures long-range dependence that convolution does not (reasoning, not assertion). A stack of $L$ convolutional layers with temporal kernel extent $k_t$ has a temporal receptive field of $O(L\cdot k_t)$ — linear in depth. To relate frame 1 to frame 100 with $k_t=3$ requires roughly 33 layers stacked purely for temporal reach, forcing depth to be spent on connectivity rather than representational abstraction. A single self-attention layer connects any pair of tokens directly (path length 1, by construction of the $QK^\top$ product being computed over all pairs), so long-range dependency modeling does not compete with depth budget — this is the same argument that motivates attention's advantage over recurrence and convolution in sequence modeling generally.

Tradeoff summary, stated honestly. 3D convolution is FLOP-efficient (linear in $T$) but has a bounded per-layer temporal reach and a large parameter count; two-stream is simple, modular, and reuses pretrained image backbones, but depends on a fixed, non-differentiable, hand-designed motion representation and pays for two forward passes; temporal attention has unbounded reach per layer but pays quadratic cost in tokens, which factorized/local variants trade back down at the cost of some modeling generality.

Consider classifying a 16-frame, $112\times112$ clip of someone either "pouring" or "stacking" cups — two actions that look nearly identical in any single frame (a hand near cups) but differ entirely in motion trajectory.

Step 1 — 3D CNN path (I3D-style). The clip tensor is $(3, 16, 112, 112)$. A first 3D conv layer with kernel $3\times7\times7$, stride $(1,2,2)$, 64 output channels, produces an activation map $(64, 16, 56, 56)$ — spatial downsampling has begun, temporal extent is preserved so early motion cues survive. After several inflated Inception blocks and periodic $2\times$ temporal pooling, by the final block the tensor might be $(1024, 2, 4, 4)$: only 2 temporal "moments" remain, each a coarse spatial summary, but each moment's 1024-dim feature already encodes a few frames' worth of local motion via the accumulated $k_t$ reach. Global average pooling over $(2,4,4)$ gives a single 1024-dim vector fed to a softmax over {pouring, stacking}. Because pouring involves downward-then-sideways motion of the flow while stacking involves discrete vertical taps, the accumulated spatiotemporal filters (not any single frame) carry the discriminative signal.

Step 2 — Two-stream path. The spatial stream sees a single RGB frame — hand, cup, table — and, alone, cannot distinguish the two actions reliably: both softmax outputs might read (pouring 0.55, stacking 0.45), reflecting only weak scene priors (e.g., a cup with liquid slightly visible). The temporal stream sees a $10$-flow stack ($20$ channels): pouring flow fields show a smooth, sustained downward-then-tilt vector field near the cup rim; stacking flow fields show short, repeated vertical bursts. The temporal stream's softmax might read (pouring 0.92, stacking 0.08) — much sharper, because motion pattern is the actually discriminative feature here. Fusing with $\alpha = 0.4$ (weighting the sharper temporal stream more):

correctly predicting "pouring," illustrating precisely why two-stream fusion helps: the spatial stream alone is near-chance, but motion resolves the ambiguity.

Step 3 — Temporal attention path. Tokenize the clip into $16$ time-steps $\times\ 8\times8$ spatial patches ($1024$ tokens total, using $14\times14$ pixel patches on the $112\times112$ frame). Divided attention first lets patches within frame 6 (mid-pour) attend to each other, sharpening the "hand + tilted cup + falling liquid" pattern into a strong local feature. Then, for the spatial location containing the cup, temporal attention lets that location's token in frame 6 attend strongly to the same location's token in frames 4, 5, 7, 8 — directly linking "cup starting to tilt" to "liquid mid-air" to "cup upright again," regardless of how many frames separate them, and regardless of network depth. The resulting cup-location feature at the final layer effectively encodes the full pour trajectory, which a classification head maps to "pouring" with high confidence — and does so without requiring 30+ stacked conv layers to reach that temporal span.

The lineage of video temporal modeling runs from classical motion estimation through deep two-stream design to the transformer era.

Two-stream and 3D-CNN backbones (I3D in particular) remain standard pretraining targets and feature extractors for action recognition benchmarks (Kinetics, Something-Something), directly foreshadowing the more applied treatment in Action Recognition. Temporal Shift Module and similar low-cost temporal-mixing tricks are deployed in production video classification systems where inference latency budgets rule out full 3D convolution or full attention. Video transformers with factorized spatiotemporal attention underlie modern video captioning, video question-answering, and video retrieval systems, where long-range temporal reasoning (relating an early event to a late consequence) is essential. Optical-flow-based motion streams persist in video compression-adjacent and sports-analytics pipelines that need explicit, interpretable motion fields rather than opaque learned features. And the spatiotemporal-token abstraction developed for video transformers now underlies text-to-video and video-generation models, which must model temporal coherence across generated frames — connecting this article's mechanisms directly to the frontier of generative video systems and situating it within the broader arc from Image Generation Fundamentals to full video generation.

Ask the tutor: How would you design a hybrid architecture that uses 3D-convolutional patch embeddings feeding into a factorized temporal-attention transformer, and what compute budget tradeoffs would that involve?