The Image Formation Model

The image formation model describes the transformation from a scene, a continuous distribution of light-emitting and light-reflecting surfaces in 3D space, to an image, a discrete 2D array of intensity values. This transformation is usually decomposed into a geometric projection and a radiometric response. The geometric part answers: given a 3D point $\mathbf{P} = (X, Y, Z)$ in the world, at which pixel $(u, v)$ does it appear? The simplest and most widely used answer is the pinhole (perspective) projection: light from $\mathbf{P}$ travels in a straight line through a single point (the optical center) and strikes an image plane at distance $f$ (the focal length) behind it. This produces the projection equations

where $(x, y)$ are coordinates on the image plane, later converted to pixel coordinates $(u, v)$ via the sensor's pixel size and principal point offset. The key qualitative fact — objects farther away ($Z$ larger) project smaller — is what makes monocular depth ambiguous and multi-view geometry necessary. The radiometric part answers: given that a ray from $\mathbf{P}$ reaches the sensor, what numerical value does that pixel record? This depends on the scene radiance $L$ leaving the surface toward the camera (itself a function of illumination, surface orientation, and material reflectance), the image irradiance $E$ that radiance produces at the sensor after passing through the lens aperture, and finally the sensor's conversion of irradiance, integrated over exposure time, into a digital number through gain and quantization. Radiance is preserved along a ray in free space (ignoring atmospheric effects), but irradiance at the sensor is not equal to radiance — it is attenuated by the lens geometry, most importantly by a factor of $\cos^4\alpha$ where $\alpha$ is the angle between the ray and the optical axis, which is the physical origin of vignetting (darkening toward image corners). Putting the two halves together gives the full image formation equation used throughout the field:

where $E$ is irradiance at pixel $(u,v)$ derived from scene radiance via the geometric mapping and lens aperture, $\Delta t$ is exposure time, $\tau$ is a sensor gain, and $g(\cdot)$ is a (possibly nonlinear) camera response function converting accumulated charge to a digital pixel value. Every vision algorithm that assumes "brighter surface $\Rightarrow$ brighter pixel" or "closer object $\Rightarrow$ different pixel location" is implicitly relying on some version of this equation.

The image formation pipeline can be broken into five explicit stages, each with a well-defined mathematical operation. This decomposition is what lets camera calibration, photometric correction, and 3D reconstruction be solved as separate, tractable sub-problems rather than one intractable black box.

Stage 1 — Scene radiance. For each visible surface point, compute the radiance $L$ leaving that point toward the camera. This depends on incoming illumination $L_i$ from all directions, the surface's bidirectional reflectance distribution function (BRDF) $f_r$, and the surface normal $\mathbf{n}$:

Stage 2 — Geometric projection. Map the 3D point to a 2D location on the image plane using the pinhole model (extended later with lens distortion). This uses the camera's extrinsic parameters (rotation $R$, translation $\mathbf{t}$, placing the point in camera coordinates) and intrinsic parameters (focal length, pixel scale, principal point).

Stage 3 — Lens aperture and irradiance. Convert the radiance arriving along each ray into image irradiance $E$ at the corresponding pixel, accounting for the lens aperture diameter $d$, focal length $f$, and the off-axis angle $\alpha$:

Stage 4 — Sensor integration. Integrate irradiance over the pixel's finite area and the exposure time $\Delta t$, then apply gain $\tau$ (ISO-like sensitivity) to produce an analog signal.

Stage 5 — Quantization and response. Apply the sensor/ISP response function $g(\cdot)$ (often approximately linear for RAW data, nonlinear such as gamma-corrected for JPEG output), add sensor noise, and quantize to a fixed bit depth (e.g., 8 or 12 bits). The pseudocode below implements this pipeline for a synthetic scene made of point samples with known position, normal, and material, which is exactly how physically based renderers and vision simulators generate labeled training data. ``` function FORM_IMAGE(scene_points, camera):

# scene_points: list of (P_world, normal, brdf, light_sources)

# camera: {R, t, f, pixel_size, principal_point, aperture_d, exposure_dt, gain, response_fn} image_irradiance = zeros(height, width) depth_buffer = full(height, width, +infinity) for each point in scene_points:

# --- Stage 1: radiance leaving the surface --- L = compute_radiance(point.normal, point.brdf, point.light_sources)

# --- Stage 2: geometric projection (pinhole) --- P_cam = camera.R point.P_world + camera.t # world -> camera coords if P_cam.Z <= 0: continue # behind the camera, skip x = camera.f P_cam.X / P_cam.Z y = camera.f * P_cam.Y / P_cam.Z (u, v) = to_pixel_coords(x, y, camera.pixel_size, camera.principal_point) if not in_bounds(u, v, height, width): continue

# --- occlusion: keep only the nearest point per pixel --- if P_cam.Z >= depth_buffer[u, v]: continue depth_buffer[u, v] = P_cam.Z

# --- Stage 3: radiance -> irradiance via lens aperture, cos^4 falloff --- alpha = angle_between(ray_direction(u, v, camera), camera.optical_axis) E = L (pi / 4) (camera.aperture_d / camera.f)^2 * cos(alpha)^4 image_irradiance[u, v] = E

# --- Stage 4: integrate over exposure time and apply gain --- signal = image_irradiance camera.exposure_dt camera.gain

# --- Stage 5: response function, noise, quantization --- signal = signal + sample_sensor_noise(signal) pixel_values = quantize(camera.response_fn(signal), bit_depth=8) return pixel_values ``` The loop over `scene_points` with a depth buffer is the standard way to correctly resolve occlusion: multiple 3D points can project to the same pixel, and only the nearest (smallest camera-space depth $Z$) should contribute, since it physically blocks light from farther points along that ray.

Time and space complexity. For a scene represented by $N$ point samples (or, in a rasterizer, $N$ triangle vertices) and an image of $P = H \times W$ pixels, the algorithm above performs $O(1)$ work per point for radiance computation and projection, and $O(1)$ amortized work per depth-buffer update, giving overall time complexity

— linear in the number of scene primitives plus a constant pass to initialize and finalize the image buffer. Space complexity is $O(P)$ for the irradiance image and depth buffer, plus $O(N)$ to store the scene description if it must be kept in memory. This is asymptotically optimal: any correct renderer must at minimum write to every pixel it touches and read every scene primitive at least once, so $\Omega(N + P)$ is a lower bound whenever primitives can independently affect arbitrary pixels. Real rasterizers using triangles rather than points add a per-triangle rasterization cost proportional to the triangle's projected pixel area, but the point-sample analysis captures the essential linear scaling.

Correctness argument. The correctness of the pipeline rests on three invariants that must each be maintained: 1. Projection invariant: a 3D point $\mathbf{P}$ with camera-frame depth $Z > 0$ projects to the unique pixel $(u,v)$ predicted by similar triangles under the pinhole assumption — this follows directly from Euclidean geometry: the ray from the origin through $\mathbf{P}$ intersects the plane $Z = f$ at $(fX/Z, fY/Z, f)$, which is exactly the perspective divide used in Stage 2. Points with $Z \le 0$ are behind the camera and must be discarded, otherwise the projection formula would place them incorrectly in front. 2. Occlusion invariant: at the end of processing, `depth_buffer[u,v]` holds the minimum camera-space depth among all points that projected to $(u,v)$, and `image_irradiance[u,v]` corresponds to that same nearest point. This is maintained inductively: each new candidate point is accepted only if its depth is strictly less than the current minimum, so after processing all $N$ points the buffer reflects the true visible surface — exactly the physical fact that opaque nearer surfaces block farther light along the same ray. 3. Radiometric invariant: the irradiance formula $E = L \cdot \frac{\pi}{4}(d/f)^2 \cos^4\alpha$ is not an approximation invented for convenience but follows from integrating radiance over the solid angle subtended by a lens aperture of diameter $d$ at distance $f$, projected through angle $\alpha$ from the axis — the four factors of $\cos\alpha$ arise from foreshortening of the aperture as seen from the point (one factor), foreshortening of the pixel as seen from the lens (one factor), and the inverse-square falloff of solid angle with the two remaining factors of $\cos\alpha$ absorbed into the distance term. This derivation is standard in radiometry and explains why corners of wide-angle images are reliably darker even for a perfectly uniform scene.

Honest tradeoffs. The pinhole model ignores lens aberrations, diffraction, and finite aperture blur (depth of field), all of which require extending Stage 3 with a more detailed optical model — the subject of lens-based image formation. The point-sampled radiance model also ignores interreflection and global illumination unless the radiance computation in Stage 1 itself recursively accounts for light bouncing between surfaces, which is expensive. Finally, treating the sensor response $g(\cdot)$ as known and fixed is itself an approximation; real cameras apply proprietary, often nonlinear in-camera processing (white balance, tone mapping, denoising) that must be estimated or inverted (radiometric calibration) before radiance-level reasoning is valid.

Consider a small gray Lambertian sphere of radius $0.05$ m sitting at world position $(0, 0, 2)$ m, illuminated by a point light of intensity such that the topmost surface point facing the camera has outgoing radiance $L = 120\ \text{W}/(\text{sr}\cdot\text{m}^2)$. The camera has focal length $f = 0.05$ m (50 mm), aperture diameter $d = 0.01$ m ($f/5$), pixel size $5\ \mu\text{m}$, exposure time $\Delta t = 1/500\ \text{s}$, gain $\tau = 1$, and a linear response $g(x) = 255 \cdot \min(x / x_{\max}, 1)$.

The nearly-off-axis point ($\alpha \approx 0.58^\circ$) suffers negligible vignetting here, but repeating the same computation for a surface point projecting to the far image corner — say $u = 1000$, giving $x \approx 0.0244$ m and $\alpha = \arctan(0.0244/0.05) \approx 26.0^\circ$ — yields $\cos^4(26.0^\circ) \approx 0.653$, roughly a 35% darkening purely from geometry, even though the physical surface radiance is identical. This is precisely the vignetting effect that flat-field calibration must remove before, for example, using pixel brightness as a proxy for surface reflectance in Photometric Stereo.

The image formation model synthesizes three historically separate lineages: the geometry of perspective from Renaissance art and optics, the physics of radiometric measurement, and the computational formalization of both within computer vision starting in the 1970s–1980s.

Camera calibration pipelines, used in every robotics and augmented-reality system, exist precisely to estimate the intrinsic and extrinsic parameters of the geometric half of this model, so that pixel coordinates can be converted back into 3D rays. Multi-view 3D reconstruction, autonomous vehicle perception stacks, and photogrammetry software (used for mapping, archaeology, and visual effects) all depend on the pinhole projection equation being an accurate — or accurately corrected — description of how points land in the image. On the radiometric side, high dynamic range (HDR) imaging, photometric stereo for industrial inspection and film-quality 3D scanning, and physically based rendering engines used in game and movie production all rely on recovering or simulating the sensor's radiometric response function. Even modern learned approaches — neural radiance fields, differentiable rendering, and self-supervised depth estimation from video — bake the same pinhole-and-radiance equations directly into their loss functions and forward models, showing that this classical formulation remains foundational even inside deep learning systems. Consumer smartphone cameras apply real-time vignetting correction and exposure fusion that are direct engineering responses to the $\cos^4\alpha$ falloff and dynamic-range limits described here.

Ask the tutor: How does adding real lens distortion change the geometric projection equations derived here?