Camera Calibration

A camera maps a 3D point in the world onto a 2D pixel through a chain of transformations. First, a rigid-body transform (extrinsics) expresses the world point in the camera's own coordinate frame: rotation matrix $R \in SO(3)$ and translation vector $t$. Second, the point is projected through the pinhole model onto the normalized image plane. Third, a nonlinear lens distortion function warps that normalized point to account for real optical imperfections. Fourth, the intrinsic matrix $K$ converts the (now distorted) normalized coordinates into pixel coordinates, accounting for focal length, principal point offset, and sensor skew.

Here $f_x, f_y$ are focal lengths in pixel units (horizontal and vertical, which differ if pixels are non-square), $c_x, c_y$ is the principal point (where the optical axis pierces the sensor), and $s$ is a skew term, nonzero only if sensor rows and columns are not perfectly perpendicular — negligible on virtually all modern sensors but included for completeness. Extrinsics $[R \mid t]$ vary per photograph: each image of the target was taken from a different pose, so calibration must estimate one $(R_i, t_i)$ pair per view in addition to the single shared $K$ and distortion coefficients.

Lens distortion is modeled separately from the linear camera matrix because it is not representable as a matrix multiplication. The dominant term is radial distortion, caused by the lens bending light more sharply away from its center:

where $(x_n, y_n)$ is the ideal (undistorted) normalized image coordinate, $r^2 = x_n^2 + y_n^2$, and $k_1, k_2, k_3$ are radial coefficients (barrel distortion for $k_1<0$, pincushion for $k_1>0$). A smaller tangential distortion term, caused by imperfect lens-sensor alignment, adds two more parameters $p_1, p_2$:

Calibration is the joint estimation of $K$, distortion coefficients $(k_1,k_2,k_3,p_1,p_2)$, and per-view extrinsics $(R_i, t_i)$ that minimizes the total squared pixel discrepancy between observed feature points and their model-predicted projections, across all views and all points.

Zhang's plane-based calibration is the standard practical recipe, chosen because it needs no specialized 3D calibration rig — a printed checkerboard on a flat surface is enough. The key insight: for a planar target, the mapping from target-plane coordinates to image pixels is a homography, a $3\times3$ projective transform, and the intrinsic matrix leaves a specific fingerprint in the algebra of that homography that can be extracted linearly.

Step 1 — Detect correspondences. For each of $N \geq 3$ images of the checkerboard (in different orientations), locate all interior corners with sub-pixel accuracy. Each detected corner $(u,v)$ pairs with a known target-plane coordinate $(X,Y,0)$ — the checkerboard's own metric grid, with $Z=0$ by construction since the target is flat.

Step 2 — Estimate a homography per view. With the target plane at $Z=0$, the projection simplifies:

where $r_1, r_2$ are the first two columns of $R$ (the third column, $r_3$, drops out because $Z=0$). Each view gives a homography $H_i$, estimated by the standard linear DLT algorithm (using ≥4 non-collinear correspondences, refined with all detected corners), the same machinery underlying planar-scene mosaicing.

Step 3 — Extract linear constraints on $K$. Write $H = [h_1\ h_2\ h_3] = \lambda K [r_1\ r_2\ t]$. Since $r_1, r_2$ are orthonormal columns of a rotation matrix, they satisfy $r_1^\top r_2 = 0$ and $\|r_1\| = \|r_2\|$. Substituting $r_1 = \lambda^{-1}K^{-1}h_1$ and $r_2 = \lambda^{-1}K^{-1}h_2$ turns these two geometric facts into two linear equations in the entries of $B = K^{-\top}K^{-1}$ (a symmetric $3\times3$ matrix with 6 unknowns):

Each of the $N$ views contributes 2 such equations. Stacking them gives a linear system $Vb = 0$ where $b$ is the 6-vector of unknowns in $B$; with $N \geq 3$ views this system is over-determined and solved by taking the eigenvector of $V^\top V$ with smallest eigenvalue (equivalently, the last column of the right singular vectors from an SVD of $V$). This is precisely why at least 3 views are required — 2 views give only 4 equations for 6 unknowns, an underdetermined system.

Step 4 — Recover $K$ in closed form. Because $B = K^{-\top}K^{-1}$, a Cholesky-style factorization of $B$ (with a sign/scale normalization) yields $K$ directly, with explicit formulas for $c_y, f_y, s, f_x, c_x$ derived from the entries of $B$. This gives an excellent initial estimate — accurate to within a fraction of a percent for typical lenses — but it assumes zero distortion, which real lenses never have exactly.

Step 5 — Recover extrinsics per view. With $K$ known, back out $r_1 = \lambda^{-1}K^{-1}h_1$, $r_2 = \lambda^{-1}K^{-1}h_2$, $r_3 = r_1\times r_2$ (enforcing orthonormality via the closest rotation, e.g. through SVD projection), and $t = \lambda^{-1}K^{-1}h_3$, giving an initial $(R_i, t_i)$ for every view.

Step 6 — Nonlinear refinement (bundle adjustment). The closed-form solution ignores distortion and treats each step's least-squares fit as independent; it is a good starting point, not the final answer. All parameters — $K$, distortion coefficients, and every $(R_i, t_i)$ — are refined jointly by minimizing total reprojection error using Levenberg–Marquardt:

where $\mathbf{u}_{ij}$ is the observed pixel location of target point $j$ in view $i$, and $\hat{\mathbf{u}}(\cdot)$ is the full forward model: rotate/translate the 3D point, project to normalized coordinates, apply distortion, then apply $K$. This is the same nonlinear least-squares machinery as bundle adjustment in Structure from Motion, specialized to a known, rigid, planar 3D structure. ``` function CALIBRATE_CAMERA(images, board_size, square_size): object_points = [] # list of arrays of known 3D board points, one array per view image_points = [] # list of arrays of detected 2D corner pixels, one array per view board_pts_3d = generate_grid(board_size, square_size) # Z = 0 for all points for img in images: found, corners = find_checkerboard_corners(img, board_size) if not found: continue # skip unusable views refined = refine_corners_subpixel(img, corners) object_points.append(board_pts_3d) image_points.append(refined) assert len(image_points) >= 3 # need >=3 views for a well-posed linear system

# --- Linear initialization (Zhang's method) --- H_list = [] for pts3d, pts2d in zip(object_points, image_points): H = estimate_homography_DLT(pts3d[:, :2], pts2d) # RANSAC-guarded against mis-detected corners H_list.append(H) V = [] for H in H_list: h1, h2, h3 = H[:, 0], H[:, 1], H[:, 2] V.append(v_ij(h1, h2)) # h1^T B h2 = 0 V.append(v_ij(h1, h1) - v_ij(h2, h2)) # h1^T B h1 = h2^T B h2 V = stack_rows(V) b = smallest_right_singular_vector(V) # solves V b = 0 in least-squares sense B = reshape_symmetric(b) K_init = recover_K_from_B(B) # closed-form Cholesky-style extraction extrinsics_init = [] for H in H_list: R, t = recover_extrinsics(H, K_init) # includes SVD projection onto SO(3) extrinsics_init.append((R, t)) dist_init = zeros(5) # k1, k2, p1, p2, k3 assumed 0 initially

# --- Nonlinear refinement --- params = pack(K_init, dist_init, extrinsics_init) params = levenberg_marquardt( residual_fn = reprojection_residuals, params0 = params, data = (object_points, image_points), max_iters = 100, tol = 1e-8 ) K, dist, extrinsics = unpack(params) rms_error = compute_rms_reprojection_error(params, object_points, image_points) return K, dist, extrinsics, rms_error

function reprojection_residuals(params, object_points, image_points): K, dist, extrinsics = unpack(params) residuals = [] for (R, t), pts3d, pts2d_obs in zip(extrinsics, object_points, image_points): pts2d_pred = project_with_distortion(pts3d, K, dist, R, t) residuals.extend(pts2d_obs - pts2d_pred) return residuals ```

Time complexity. Let $N$ be the number of views and $M$ the number of correspondences per view (e.g. $M =$ 54 for a 9×6 checkerboard). Corner detection is $O(N \cdot P)$ where $P$ is pixels per image — linear scan plus local sub-pixel refinement, negligible compared to the rest. Homography estimation via DLT is $O(M)$ per view to build the linear system plus $O(1)$ (constant $9\times9$ or smaller) for its SVD, so $O(NM)$ total. Building the intrinsic-constraint matrix $V$ is $O(N)$ (2 rows per view, 6 columns), and its SVD is $O(N)$ as well since $V$ has fixed column width — cheap regardless of $N$. The dominant cost is the nonlinear refinement: each Levenberg–Marquardt iteration solves a normal-equations system of size roughly $6N + 8$ (six extrinsic parameters per view plus shared intrinsics/distortion), costing $O((6N)^3)$ for a dense solve or, exploiting the sparse block structure (each 3D point's residual touches only one view's extrinsics and the shared intrinsics), $O(N \cdot M)$ per iteration using Schur-complement tricks analogous to sparse bundle adjustment. With a bounded number of iterations to convergence (typically 10–30), total time is close to $O(N M \cdot \text{iters})$ in a well-implemented solver, dominated in practice by corner detection and image I/O rather than the algebra.

Space complexity is $O(NM)$ to hold all correspondences, plus $O(N)$ parameters for extrinsics and $O(1)$ for shared intrinsics/distortion — modest even for hundreds of views.

Correctness of the linear step rests on two facts, both provably exact under the ideal pinhole-without-distortion model. First, a planar scene under perspective projection is related to its image by an exact homography — this is a direct consequence of the camera matrix restricted to a plane ($Z=0$ kills the third column of the rotation, collapsing the $3\times4$ projection to a $3\times3$ matrix). Second, rotation matrices are orthonormal by definition, so $r_1^\top r_2 = 0$ and $\|r_1\|=\|r_2\|$ are exact algebraic identities, not approximations — they hold for any true camera pose, which is why they can be used as hard constraints to pin down $K$. The only source of error entering the linear step is corner-detection noise and outlier correspondences (guarded against via RANSAC Robust Estimation when fitting each homography), and lens distortion, which biases $H$ estimation slightly since real correspondences don't perfectly satisfy the undistorted pinhole homography.

Correctness of the nonlinear step. Levenberg–Marquardt is a damped Gauss–Newton method: it minimizes a sum of squared residuals by iteratively solving

where $J$ is the Jacobian of the residual vector with respect to all parameters and $\mu$ is a damping term interpolating between Gauss–Newton (fast, can diverge) and gradient descent (slow, always improves locally). Because the closed-form linear step supplies an initialization already close to the true minimum — reprojection error typically starts within a few pixels — the optimization landscape near the solution is well-approximated as locally convex (quadratic in residuals), and LM converges quickly and reliably to a local minimum that, for a well-designed calibration set, coincides with the global minimum. There is no closed-form guarantee of global optimality for the nonlinear stage — this is the honest tradeoff: the linear stage is provably exact under its (idealized) assumptions, while the nonlinear stage is only guaranteed to find a local optimum, made trustworthy in practice by a good starting point and diverse, well-conditioned view geometry.

Degeneracy and conditioning. The linear system $Vb=0$ becomes ill-conditioned — near rank-deficient — if all homographies share nearly the same rotation (e.g., the target always held frontal to the camera, never tilted), because tilting the plane is precisely what generates independent constraints on $B$. This is why calibration guides insist on varying orientation substantially across views: it is not a stylistic suggestion but a direct requirement for the linear system's rank.

Consider calibrating a webcam using 8 photographs of a 9×6 inner-corner checkerboard (25 mm squares), detected and refined to sub-pixel corners. The table traces representative intermediate quantities (values are realistic order-of-magnitude figures for a typical 720p webcam):

The reprojection error drops from 1.84 px (linear-only, distortion ignored) to 0.31 px after joint nonlinear refinement — a roughly 6× improvement, entirely attributable to modeling the negative $k_1$ (barrel distortion typical of wide-angle webcam lenses). The recovered focal lengths $f_x \approx f_y \approx 910$ are consistent (a near-square-pixel sensor), and the principal point sits close to but not exactly at the image center $(640, 360)$ for a 1280×720 sensor — a small offset ($\sim$10 px) that is normal and expected due to sensor mounting tolerances, not a sign of error. A second, cautionary case: if all 8 views were taken with the board nearly parallel to the sensor (tilt under 5° in every shot), the same pipeline would produce a $K_{\text{init}}$ with wildly inflated focal length uncertainty — the constraint matrix $V$ becomes nearly rank-1, and even a good nonlinear refinement cannot fully compensate, because the information needed to separate focal length from target distance was never captured in the images. This is why calibration guides recommend tilts spanning at least 30°–45° across the view set, as in the example above.

Camera calibration draws on photogrammetry, a discipline that predates digital computer vision by over a century, developed for aerial surveying and mapmaking using stereo film photographs. The specific closed-form, plane-based method that dominates modern practice — usable with an ordinary printed checkerboard and no specialized 3D calibration jig — was formalized by Zhengyou Zhang in the late 1990s, making calibration accessible to any lab with a printer and a camera rather than requiring machined 3D calibration objects.

Every stereo camera rig — from industrial 3D scanners to autonomous vehicle perception stacks — depends on calibration to rectify image pairs into a common epipolar geometry, the essential preprocessing step feeding Stereo Vision and Depth algorithms. Structure-from-motion and SLAM pipelines, covered under Structure from Motion, use calibrated intrinsics as fixed, trusted inputs so that the harder unknowns (3D structure, camera trajectory) are the only free variables in bundle adjustment; an uncalibrated or poorly calibrated camera injects unrecoverable ambiguity into the whole reconstruction. Augmented reality systems calibrate a phone or headset camera once at manufacturing time (or via an in-app checkerboard capture) so that virtual objects render with pixel-accurate perspective and do not visibly swim as the device moves. Robotics hand-eye calibration extends the same machinery to jointly recover the rigid transform between a camera and a robot arm's end effector, letting a robot translate pixel detections into commands in its own coordinate frame. Multi-camera systems — surveillance networks, motion-capture stages, multi-view 3D scanners — calibrate every camera individually and then calibrate their relative extrinsics jointly, so that observations across cameras can be fused into one consistent 3D scene.

Ask the tutor: How does stereo camera calibration extend this single-camera pipeline to recover the relative pose between two cameras?