The Camera Matrix

A pinhole camera models image formation as light rays passing through a single point, the center of projection (or optical center), and striking a flat image plane behind it. Every 3D point, its projection ray, and its image are collinear. This idealization ignores lenses and finite apertures but captures the essential geometry of almost every real camera to good approximation, and lens distortion can be modeled as a correction layered on top. Formally, a 3D point in the world, $\mathbf{X} = (X, Y, Z)^\top$, is mapped to a 2D pixel $\mathbf{x} = (u, v)^\top$. In Euclidean coordinates this map is a nonlinear rational function (perspective division by depth $Z$). Lifting both spaces to Homogeneous Coordinates — $\tilde{\mathbf{X}} = (X,Y,Z,1)^\top \in \mathbb{P}^3$ and $\tilde{\mathbf{x}} = (x,y,w)^\top \in \mathbb{P}^2$ with $u = x/w,\ v = y/w$ — turns projection into a linear map:

P is the camera matrix (also projection matrix). It has rank 3 and is defined only up to an overall nonzero scale, since $\mathbf{P}$ and $\lambda\mathbf{P}$ give the same pixel after dividing by $w$. This yields 11 degrees of freedom in its 12 entries. The camera matrix factors cleanly as

where $[\mathbf{R}\mid\mathbf{t}]$ are the extrinsic parameters — a rotation $\mathbf{R} \in SO(3)$ and translation $\mathbf{t} \in \mathbb{R}^3$ that convert world coordinates into the camera's own reference frame — and $\mathbf{K}$ holds the intrinsic parameters — focal length, pixel aspect ratio, skew, and principal point — that convert a point in front of the camera into pixel coordinates. This split is the conceptual heart of the model: extrinsics describe where the camera is, intrinsics describe how the camera itself works, and together they fully determine image formation.

Step 1: The camera coordinate frame

Attach a right-handed coordinate frame to the camera: origin at the optical center, $Z$-axis along the viewing direction (the principal axis), $X$ and $Y$ axes parallel to the image plane's rows and columns. A world point $\mathbf{X}_w$ is converted to camera coordinates $\mathbf{X}_c$ by a rigid transform:

Here $\mathbf{R}$ is the rotation of the world axes relative to the camera, and $\mathbf{t} = -\mathbf{R}\,\mathbf{C}$, where $\mathbf{C}$ is the camera center expressed in world coordinates. In homogeneous form,

Step 2: Ideal perspective projection

In camera coordinates, the pinhole model says a point $(X_c, Y_c, Z_c)$ projects onto the image plane at distance $f$ (focal length) from the center by similar triangles:

This is exactly what the matrix

computes when applied to $(X_c, Y_c, Z_c, 1)^\top$ and dehomogenized: the third row extracts $Z_c$ as the homogeneous scale $w$, and dividing the first two rows by $w$ recovers exactly $fX_c/Z_c$ and $fY_c/Z_c$. This is the crucial linearization: perspective division is deferred to a single final step rather than baked into the matrix entries.

Step 3: From optical units to pixels — the intrinsic matrix

Real sensors measure discrete pixels, not metric distances on the image plane, and pixels may be non-square, skewed, or offset from the principal axis. The intrinsic matrix converts:

• $\alpha_x = f m_x$, $\alpha_y = f m_y$: focal length in pixels along each axis ($m_x, m_y$ = pixels per unit length).
• $u_0, v_0$: the principal point, where the optical axis pierces the image (near the image center, but not exactly, in general).
• $s$: skew, nonzero only if the pixel grid axes are not perpendicular (essentially zero for virtually all modern sensors).

Step 4: Assembling the full camera matrix

Composing steps 1–3:

$\mathbf{P}$ is $3\times 4$, has rank 3, and its null space — the unique (up to scale) $\tilde{\mathbf{X}}$ with $\mathbf{P}\tilde{\mathbf{X}} = \mathbf{0}$ — is precisely the homogeneous coordinate of the camera center $\mathbf{C}$ (since every ray through the center projects to no well-defined pixel, i.e., it maps to the zero vector, undefined in $\mathbb{P}^2$).

Step 5: Using the matrix (forward projection algorithm)

``` function PROJECT(P, X_world):

# X_world: 3-vector (X, Y, Z) in world coordinates

# P: 3x4 camera matrix X_h = [X_world[0], X_world[1], X_world[2], 1.0] # homogeneous lift x_h = P @ X_h # 3x4 * 4x1 -> 3x1 if x_h[2] == 0: return UNDEFINED # point at infinity / in image plane through center u = x_h[0] / x_h[2] v = x_h[1] / x_h[2] return (u, v) ``` Correctness of this routine is immediate from the derivation above: `x_h` equals $w\tilde{\mathbf{x}}$ by construction, and dividing by the third (homogeneous) coordinate recovers Euclidean pixel coordinates whenever $w \neq 0$, i.e., whenever the point lies strictly in front of the camera ($Z_c > 0$) rather than in the plane through the center parallel to the image plane.

Step 6: Estimating the matrix from correspondences (the Direct Linear Transform)

Given $n \ge 6$ known pairs $(\mathbf{X}_i, \mathbf{x}_i)$ (3D points with measured pixel projections), $\mathbf{P}$ can be recovered up to scale. Each correspondence gives $\mathbf{x}_i \times (\mathbf{P}\tilde{\mathbf{X}}_i) = \mathbf{0}$ (the cross product vanishes because the vectors are parallel), which is linear in the 12 unknown entries of $\mathbf{P}$. Stacking two independent equations per point yields a homogeneous linear system $\mathbf{A}\mathbf{p} = \mathbf{0}$. ``` function DLT_CALIBRATE(points3D, points2D):

# points3D: list of n >= 6 world points (X, Y, Z)

# points2D: matching list of n pixel points (u, v) A = empty matrix with 2n rows, 12 columns for i in range(n): X, Y, Z = points3D[i] u, v = points2D[i] row1 = [X, Y, Z, 1, 0, 0, 0, 0, -uX, -uY, -uZ, -u] row2 = [0, 0, 0, 0, X, Y, Z, 1, -vX, -vY, -vZ, -v] append row1, row2 to A

# Solve A p = 0 in the least-squares sense: U, S, Vt = SVD(A) p = Vt[last row] # singular vector for smallest singular value P = reshape(p, 3, 4) return P ``` The solution is the right singular vector of $\mathbf{A}$ corresponding to its smallest singular value — this is the unit vector minimizing $\|\mathbf{A}\mathbf{p}\|^2$ subject to $\|\mathbf{p}\| = 1$, which is exactly the standard variational characterization of the SVD's trailing singular vector. This gives the camera matrix directly; recovering the individual intrinsic/extrinsic factors afterward requires an RQ decomposition of the leading $3\times 3$ block of $\mathbf{P}$, covered under camera calibration.

Forward projection. Projecting one point costs one $3\times4$ by $4\times1$ matrix-vector product ($O(1)$ — 12 multiplications, 8 additions) plus one division: $O(1)$ time and space per point, $O(n)$ for $n$ points. This is essentially free compared to any image-processing step downstream, which is why perspective projection is embedded directly inside real-time rendering and vision pipelines.

Correctness of the linear formulation. The justification rests on the ratio identity of homogeneous coordinates: $(x, y, w)$ and $(\lambda x, \lambda y, \lambda w)$ represent the same 2D point for any $\lambda \neq 0$. Since $\mathbf{P}\tilde{\mathbf{X}}$ scales linearly with any rescaling of $\tilde{\mathbf{X}}$'s homogeneous representative, and physical points are only defined up to that rescaling (a homogeneous 3D point $(\lambda X, \lambda Y, \lambda Z, \lambda)$ represents the same Euclidean point for $\lambda \ne 0$), the map $\tilde{\mathbf{X}} \mapsto \mathbf{P}\tilde{\mathbf{X}}$ descends to a well-defined map on the corresponding equivalence classes — i.e., a genuine map $\mathbb{P}^3 \to \mathbb{P}^2$. The final division by $w$ is not an approximation of perspective; it is perspective division, performed exactly. Hence the model is exact for an ideal pinhole (no approximation error, aside from real lenses' deviation from the pinhole assumption, handled separately by distortion models).

Rank and degrees of freedom. $\mathbf{P}$ must have rank exactly 3 (not 4) because a rank-4 map would be invertible and could not collapse the 3D world onto a 2D image — projection is fundamentally lossy, forgetting depth. $\mathbf{K}$ contributes 5 free parameters ($\alpha_x, \alpha_y, s, u_0, v_0$), $\mathbf{R}$ contributes 3 (rotations in $SO(3)$), and $\mathbf{t}$ contributes 3, totaling 11 — matching the 12 raw entries of $\mathbf{P}$ minus 1 for the unrecoverable global scale. This parameter count is exactly what tells us the DLT needs at least $\lceil 11/2 \rceil = 6$ point correspondences (each contributing 2 independent linear equations) to constrain $\mathbf{P}$ up to scale, with more points used in practice to average out noise via least squares.

DLT correctness and cost. Solving $\mathbf{A}\mathbf{p}=\mathbf{0}$ via SVD costs $O(n \cdot 12^2) = O(n)$ for the matrix construction and $O(\min(2n,12)^2 \cdot \max(2n,12))$ for the SVD itself — in practice dominated by $O(n)$ for large $n$ since the matrix has only 12 columns. Correctness follows from the Rayleigh–Ritz variational characterization: minimizing $\|\mathbf{A}\mathbf{p}\|^2$ over unit vectors is minimized exactly by the eigenvector of $\mathbf{A}^\top\mathbf{A}$ with smallest eigenvalue, which is the last right-singular vector of $\mathbf{A}$'s SVD. This is a genuine algebraic minimum, not a heuristic — but it minimizes an algebraic residual ($\mathbf{A}\mathbf{p}$), not the true geometric reprojection error, which is why DLT solutions are typically refined afterward by nonlinear least squares (bundle adjustment) that directly minimizes pixel distances.

Tradeoffs. The pinhole/DLT model is linear, fast, and globally solvable without initialization — a major advantage over nonlinear optimization, which needs a good starting guess. Its cost is that it ignores lens distortion entirely and optimizes an algebraic rather than geometric error, so in practice DLT output is used as an initializer for a subsequent nonlinear refinement rather than a final answer.

Take a simple, deliberately asymmetric setup so no coincidental cancellation hides an error.

Intrinsics: $\alpha_x = 800$, $\alpha_y = 800$, $s=0$, principal point $(u_0, v_0) = (320, 240)$ (a $640\times480$ image).

Extrinsics: the camera sits at world position $\mathbf{C} = (0, 0, -5)$, looking straight along $+Z$ with no rotation, so $\mathbf{R} = \mathbf{I}$ and $\mathbf{t} = -\mathbf{R}\mathbf{C} = (0,0,5)^\top$.

Camera matrix:

(Row 1 last entry: $320 \times 5 = 1600$; row 2 last entry: $240\times 5 = 1200$; row 3: $0,0,1,5$.)

Project the world point $\mathbf{X} = (1, 2, 3)$:

So the world point $(1,2,3)$ lands at pixel $(420, 440)$.

Sanity check via direct geometry. In camera coordinates, $\mathbf{X}_c = \mathbf{R}\mathbf{X} + \mathbf{t} = (1, 2, 3+5) = (1,2,8)$. Ideal perspective: $x = f X_c/Z_c = 800(1/8) = 100$, $y = 800(2/8) = 200$. Converting to pixels with the principal point offset: $u = 100 + 320 = 420$, $v = 200 + 240 = 440$. This matches the matrix computation exactly, confirming the factorization $\mathbf{P}=\mathbf{K}[\mathbf{R}\mid\mathbf{t}]$ is correctly assembled.

Recovering the camera center from P alone. Solve $\mathbf{P}\tilde{\mathbf{C}} = 0$. With $\mathbf{P}$ as above, one can verify $\tilde{\mathbf{C}} = (0,0,-5,1)^\top$ satisfies row 3: $0+0+1(-5)+5(1) = 0$ ✓; row 1: $800(0)+0+320(-5)+1600(1)=-1600+1600=0$ ✓; row 2: $0+800(0)+240(-5)+1200(1) = -1200+1200 = 0$ ✓. This recovers the camera position $(0,0,-5)$ purely from the null space of $\mathbf{P}$, exactly as claimed.

The mathematics of perspective projection predates computing by centuries, rooted in Renaissance perspective drawing, but its formalization as linear algebra over homogeneous coordinates belongs to projective geometry and later to photogrammetry and computer vision.

The camera matrix is the geometric backbone of structure-from-motion and visual SLAM systems (used in AR headsets, drones, and mobile mapping), where thousands of camera matrices and 3D points are jointly refined via bundle adjustment. Camera calibration procedures, from Zhang's checkerboard method to industrial photogrammetry rigs, exist specifically to estimate $\mathbf{K}$ and $[\mathbf{R}\mid\mathbf{t}]$ for a physical camera. Epipolar geometry and stereo vision, used in self-driving perception stacks and depth-sensing cameras, build directly on relating two camera matrices for the same scene. Computer graphics rendering pipelines use the same $\mathbf{K}[\mathbf{R}\mid\mathbf{t}]$ structure (as a "view-projection matrix") to draw 3D scenes onto 2D screens, and augmented-reality systems rely on accurate camera matrices to align virtual objects with real video convincingly. Robotics grasping and navigation systems use inverse projection (back-projecting a pixel to a 3D ray) built entirely on the same matrix, once combined with depth or multi-view constraints.

Ask the tutor: How does the fundamental matrix in epipolar geometry get constructed from two camera matrices $\mathbf{P}_1$ and $\mathbf{P}_2$?