Optical Flow Fundamentals

Optical flow is the apparent velocity field $\mathbf{v}(x,y) = (u,v)$ describing, for each pixel $(x,y)$ in an image, how far and in which direction its brightness pattern appears to have moved between time $t$ and time $t+\delta t$. It is important to distinguish optical flow from the true motion field — the projection of actual 3D scene velocities onto the image plane. The two coincide under ideal conditions (Lambertian surfaces, no lighting changes) but can diverge dramatically: a perfectly uniform, rotating sphere under fixed lighting produces zero optical flow despite having non-zero 3D motion, while a stationary striped barber pole under moving light can produce apparent flow with no physical motion at all. Optical flow is what we can actually measure from pixel intensities; the motion field is what we usually want to know. To make estimation tractable, essentially every classical flow algorithm — including feature trackers built on Feature Detection Fundamentals and Image gradients — adopts the brightness constancy assumption: the observed intensity $I(x,y,t)$ of a physical point does not change as that point moves, only its image-plane position does. Formally, if a point at $(x,y)$ at time $t$ moves to $(x+dx, y+dy)$ at time $t+dt$, constancy requires

This assumption is an idealization. It fails whenever shading changes with viewing or lighting angle (specular highlights, shadows), whenever objects are transparent or translucent, and at occlusion boundaries where surfaces appear or disappear. Nonetheless it holds well enough locally, over short time intervals and for diffuse (Lambertian) surfaces, to be the workhorse assumption of the field. Linearizing brightness constancy with a first-order Taylor expansion produces the central result of this article: the optical flow constraint equation (also called the brightness constancy constraint equation, BCCE), a single linear relationship between the spatial and temporal image gradients and the unknown velocity $(u,v)$. Because this one equation cannot pin down two unknowns, every pixel's local evidence constrains motion to a line in velocity space rather than a point — the aperture problem — motivating the neighborhood-pooling strategies used by later algorithms such as Lucas-Kanade Optical Flow.

Step 1 — State brightness constancy

Assume the point at $(x,y)$ moves by $(dx, dy)$ over a small time step $dt$, and its brightness is preserved:

Step 2 — Taylor-expand the right-hand side

Assuming $I$ is differentiable, expand the left side to first order around $(x,y,t)$:

Substituting into the constancy equation and cancelling $I(x,y,t)$ from both sides leaves

Step 3 — Divide by $dt$ to get velocities

Let $u = dx/dt$ and $v = dy/dt$ be the horizontal and vertical components of optical flow. Dividing through by $dt$:

where $I_x = \partial I/\partial x$, $I_y = \partial I/\partial y$, and $I_t = \partial I/\partial t$ are the spatial and temporal image gradients, all computable directly from the image sequence using finite differences (the same derivatives used in Image gradients for edge and corner detection). This is the optical flow constraint equation:

Step 4 — Recognize the equation is underdetermined

This is one scalar linear equation in two unknowns $(u,v)$. Geometrically, it defines a line in $(u,v)$-velocity space:

Any $(u,v)$ satisfying this line is consistent with the observed local brightness change. The only fully determined quantity is the component of flow along the gradient direction $\nabla I / \|\nabla I\|$ — the normal flow:

The component of true velocity parallel to the edge (tangential to the local iso-brightness contour) is invisible to this single equation — it can be anything without violating brightness constancy. This is the aperture problem: viewed through a small aperture (a local patch), a moving edge's true velocity cannot be recovered, only its perpendicular component.

Step 5 — Where does full 2D constraint come from?

A single equation cannot resolve $(u,v)$, but if a local neighborhood contains two or more differently-oriented gradients (i.e. a corner or textured patch, not a single straight edge), then each pixel contributes a differently-oriented line constraint in $(u,v)$-space, and the intersection of these lines pins down a unique $(u,v)$. This is precisely the structural condition — a well-conditioned second-moment ("Harris") matrix — used by corner detectors in Feature Detection Fundamentals, and it is the reason optical flow is estimated reliably at corners and textured regions but not along smooth edges or inside flat regions. The following pseudocode summarizes the full pipeline: compute gradients, form the per-pixel constraint, and classify each pixel by how well-constrained its flow is. ```text ALGORITHM ComputeOpticalFlowConstraint(I_prev, I_curr):

# I_prev, I_curr: grayscale images at time t and t+dt, same size (H x W)

# Returns: Ix, Iy, It (gradient fields) and Constraint (per-pixel line) Ix = zeros(H, W) Iy = zeros(H, W) It = zeros(H, W) for each pixel (x, y) in I_curr:

# spatial derivatives via central differences on the current frame Ix[x,y] = ( I_curr[x+1, y] - I_curr[x-1, y] ) / 2 # edge pixels: use one-sided difference Iy[x,y] = ( I_curr[x, y+1] - I_curr[x, y-1] ) / 2

# temporal derivative between the two frames It[x,y] = I_curr[x, y] - I_prev[x, y] for each pixel (x, y): g = (Ix[x,y], Iy[x,y]) if norm(g) < EPSILON:

# flat region: no gradient, no constraint at all -- flow is fully undetermined mark (x,y) as UNCONSTRAINED else:

# the single linear constraint: Ixu + Iyv = -It Constraint[x,y] = Line(normal_direction = g, offset = -It[x,y])

# normal-flow magnitude is always recoverable: normal_speed = abs(It[x,y]) / norm(g) mark (x,y) as APERTURE_LIMITED with normal_speed and direction g return Ix, Iy, It, Constraint

ALGORITHM ClassifyAmbiguity(Ix, Iy, window_radius r):

# Determines, per pixel, whether local neighborhood constrains BOTH flow components

# (this anticipates the structure tensor used by Lucas-Kanade) for each pixel (x, y): M = [[0,0],[0,0]] # 2x2 second-moment (structure) matrix for (i, j) in window centered at (x,y) with radius r: M += [[Ix[i,j]^2, Ix[i,j]Iy[i,j]], [Ix[i,j]Iy[i,j], Iy[i,j]^2 ]] lambda1, lambda2 = eigenvalues(M) # lambda1 >= lambda2 >= 0 if lambda2 is small: # only one dominant gradient direction label(x,y) = "APERTURE PROBLEM (edge or flat region)" else: # two well-separated gradient directions label(x,y) = "WELL-CONSTRAINED (corner / texture)" return labels ``` The second routine does not yet solve for flow — that is the job of Lucas-Kanade Optical Flow — but it makes precise, using the same eigenvalue analysis as corner detection, exactly which pixels the aperture problem afflicts.

Time and space complexity. For an $H \times W$ image, computing $I_x$, $I_y$, and $I_t$ via finite differences takes $O(HW)$ time and $O(HW)$ space, since each pixel requires a constant number of arithmetic operations on its neighbors, and the outputs are stored as $H\times W$ arrays. Classifying ambiguity with a window of radius $r$ costs $O(HW r^2)$ time in the naive implementation (an $r \times r$ sum per pixel), which reduces to $O(HW)$ using running-sum (integral image / box-filter) tricks that make the per-pixel window sum $O(1)$ amortized. Nothing here yet solves for $(u,v)$; this article establishes the constraint, and the estimation cost is deferred to algorithms like Lucas-Kanade Optical Flow that must additionally solve a $2\times 2$ linear system per pixel, adding only $O(1)$ per pixel and hence $O(HW)$ overall.

Correctness of the linearization. The optical flow equation is only an approximation, valid in the limit of small motion. Formally, brightness constancy states $I(x+dx, y+dy, t+dt) = I(x,y,t)$ exactly; the Taylor expansion introduces error terms of order $O(\|(dx,dy,dt)\|^2)$, i.e., proportional to the square of the displacement and second derivatives of $I$. The approximation is therefore accurate when: 1. The displacement $(u,v)\,dt$ is small relative to the scale over which $I_x, I_y$ change appreciably (i.e., small motion relative to image structure scale), and 2. $I$ is smooth enough that a first-order Taylor expansion is meaningful (textureless regions with near-zero gradient are numerically ill-conditioned but not literally "wrong" — the constraint $0\cdot u + 0\cdot v + I_t \approx 0$ degenerates, forcing $I_t\approx 0$, which conveys no directional information). When displacements are large (fast motion, low frame rate), the linear approximation breaks down; this is why practical systems use coarse-to-fine pyramids, warping, or larger search ranges — refinements built on top of, not replacing, the constraint equation derived here.

Correctness of the aperture-problem claim. This is not merely empirical — it follows from linear algebra. The constraint $I_x u + I_y v = -I_t$ is one linear equation in two unknowns, so its solution set is a 1-dimensional affine subspace (a line) of $\mathbb{R}^2$, unless a second, linearly independent equation is supplied. Decompose the true velocity $\mathbf v = \mathbf v_\parallel + \mathbf v_\perp$ into components parallel and perpendicular to the local edge (i.e. parallel and perpendicular to $\nabla I$). Since $\nabla I \cdot \mathbf v_\parallel = 0$ by construction (motion along the edge does not change local brightness to first order), the constraint equation depends only on $\mathbf v_\perp$:

This is a proof, not a heuristic: any two velocities differing only by a vector parallel to the local iso-brightness contour produce identically zero change in the linearized brightness equation, hence are indistinguishable from one pixel's local evidence alone. Resolving $\mathbf v_\parallel$ strictly requires additional information — either spatial context (multiple differently-oriented gradients in a neighborhood, as in Lucas-Kanade Optical Flow), a global smoothness prior (as in Horn–Schunck-style methods), or a learned data prior (as in modern deep flow networks) — none of which is available from the single-pixel differential constraint itself.

Honest tradeoff. The brightness constancy linearization buys a clean, closed-form, per-pixel linear constraint at essentially zero computational cost, but it is fundamentally a local and first-order model: it says nothing about occlusion, large motion, non-Lambertian shading, or how to combine constraints across pixels. Every subsequent classical or deep optical flow method is, in some sense, a different strategy for patching these three gaps (aggregation, robustness, and scale) while keeping this equation as the measurement model at its core.

Consider a tiny $5\times 5$ synthetic image with a diagonal dark/light edge, sampled at two consecutive time steps, where the edge shifts by exactly one pixel to the right (true motion $u=1, v=0$). Frame at $t$ (intensity 0 = dark, 1 = bright), edge running along the anti-diagonal: ``` row\col 0 1 2 3 4 0 0.0 0.0 0.0 1.0 1.0 1 0.0 0.0 1.0 1.0 1.0 2 0.0 1.0 1.0 1.0 1.0 3 1.0 1.0 1.0 1.0 1.0 4 1.0 1.0 1.0 1.0 1.0 ``` Frame at $t+1$, shifted one column to the right ($u=1,\,v=0$): ``` row\col 0 1 2 3 4 0 0.0 0.0 0.0 0.0 1.0 1 0.0 0.0 0.0 1.0 1.0 2 0.0 0.0 1.0 1.0 1.0 3 0.0 1.0 1.0 1.0 1.0 4 1.0 1.0 1.0 1.0 1.0 ``` Take pixel $(x,y) = (2,2)$ (row 2, col 2), which sits exactly on the edge in the first frame. Using central differences on the current frame ($t+1$) and a temporal difference between frames:

Plugging into the optical flow constraint $I_x u + I_y v + I_t = 0$:

This is a line through the origin in $(u,v)$-space, not a point — exactly the aperture problem in action, since $(2,2)$ sits on a straight diagonal edge with a single dominant gradient direction. The normal flow — the only recoverable quantity — is

which is misleading here only because $I_t$ happened to be exactly zero at this pixel (the shifted edge still passes through the same intensity value at $(2,2)$) — a reminder that single-pixel evidence can be degenerate. Now examine pixel $(1,2)$ instead:

Constraint: $0.5u + 0.5v - 1.0 = 0 \implies u+v = 2$. The normal-flow magnitude is $|{-1.0}|/\sqrt{0.5}\approx 1.41$, correctly signaling substantial motion, but the equation is satisfied equally by $(u,v)=(1,1)$, $(2,0)$, $(0,2)$, or infinitely many other pairs — even though the true motion is $(1,0)$. Only by pooling this constraint with neighboring pixels that have differently-oriented gradients (present near the corner of the diagonal pattern, but absent along its straight middle) can the true $(1,0)$ be isolated — precisely the mechanism Lucas-Kanade Optical Flow formalizes.

The mathematical formulation of optical flow as a differential brightness-constancy constraint was crystallized in the early 1980s, converging computer vision with ideas from psychophysics (motion perception in biological vision) and fluid dynamics (the notion of a "flow field").

Optical flow underlies video compression standards (motion-compensated prediction in MPEG/H.26x codecs estimates block-wise flow to encode only residual differences between frames). It is central to autonomous driving and robotics, where flow fields feed into obstacle detection, time-to-collision estimation, and visual odometry pipelines that also rely on Feature Detection Fundamentals for sparse tracking. Video frame interpolation (slow-motion generation, frame-rate up-conversion in TVs and streaming) synthesizes intermediate frames by warping along estimated flow. Action recognition models historically used stacked optical flow fields as an input modality alongside RGB frames to capture motion cues explicitly. Medical imaging uses optical-flow-style deformable registration to track tissue motion (e.g., cardiac wall motion in echocardiography). Even modern learned flow networks such as RAFT and PWC-Net, despite being trained end-to-end, are typically evaluated against — and partly interpretable through — the same brightness constancy constraint and aperture problem introduced here, making this article's content the conceptual substrate for the entire field, both classical and deep.

Ask the tutor: How exactly does Lucas-Kanade combine multiple pixels' constraint lines to solve the aperture problem in closed form?