Neural Radiance Fields

A radiance field is a function that assigns, to every point in space and every outgoing direction, the amount of light emitted or reflected toward that direction. Classical computer graphics represents such fields implicitly via surfaces plus a bidirectional reflectance model; Neural Radiance Fields instead represent the field directly as a continuous, learned function $F_\theta$, parameterized by a neural network with weights $\theta$. The network takes a 3D position $\mathbf{x}=(x,y,z)$ and a viewing direction $\mathbf{d}=(\theta,\phi)$ (or a unit vector) and outputs an emitted RGB color $\mathbf{c}$ and a volume density $\sigma(\mathbf{x})$, a scalar describing how opaque space is at that point, independent of viewing direction. Density depending only on position (not direction) encodes the physical fact that occupancy of space is not observer-dependent, while color depending on direction captures view-dependent effects such as specular highlights. Rendering a pixel means casting a ray $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$ from the camera center $\mathbf{o}$ through the pixel, and computing the expected color received along that ray via the classical volume rendering integral:

Here $T(t)$ is the accumulated transmittance, the probability that a photon travels from $t_n$ (near bound) to $t$ unobstructed. Intuitively, each infinitesimal segment of the ray absorbs and emits light in proportion to its density, and $T(t)$ discounts contributions from points that are occluded by denser material closer to the camera. This is exactly the emission-absorption model used in medical volume rendering and physically-based smoke rendering, adapted here so that the entire field $(\sigma, \mathbf{c})$ is a differentiable neural network rather than a voxel grid measured by a scanner. The key insight that made this practical is that the rendering integral is differentiable end-to-end with respect to $\theta$: if predicted pixel colors are compared to ground-truth photograph pixels with a squared-error loss, gradients can be backpropagated through the discretized integral into the MLP weights. No 3D ground truth, depth maps, or explicit correspondences are required — only a set of images with known camera poses (obtained beforehand via 3D Vision and Point Clouds techniques like structure-from-motion). Two engineering ingredients are essential to make a simple MLP actually work in this role: positional encoding, which lifts low-dimensional coordinates into a high-dimensional space of sinusoids so the network can represent high-frequency detail (a manifestation of the spectral bias of MLPs toward smooth functions), and hierarchical volume sampling, a coarse-to-fine importance sampling scheme that concentrates ray samples near surfaces instead of wasting them in empty space.

Step 1 — Ray generation. For each pixel in the target image, use the known camera intrinsics and pose to compute the ray origin $\mathbf{o}$ and direction $\mathbf{d}$.

Step 2 — Positional encoding. Raw coordinates are mapped through a fixed, non-learned sinusoidal embedding before entering the MLP:

This is applied per coordinate of $\mathbf{x}$ (typically $L=10$) and per coordinate of $\mathbf{d}$ (typically $L=4$), turning a 3-vector into a $3(2L)$-dimensional vector.

Step 3 — Network evaluation. An 8-layer MLP takes $\gamma(\mathbf{x})$, with a skip connection re-injecting $\gamma(\mathbf{x})$ at a middle layer, and outputs a 256-dim feature plus density $\sigma$ (density is made to depend only on position by branching it off before direction is introduced). The feature is concatenated with $\gamma(\mathbf{d})$ and passed through a small additional head to produce the view-dependent RGB color $\mathbf{c}$.

Step 4 — Stratified sampling and quadrature. The continuous integral is approximated by sampling $N$ points $\{t_i\}$ along each ray, drawn via stratified sampling (one uniform random sample per bin) so that the network sees a continuum of positions during training rather than a fixed discrete set — this is what lets the trained field be continuous rather than griddy. The integral is discretized as a Riemann-sum-like quadrature (derived below) using the interval lengths $\delta_i = t_{i+1}-t_i$:

Step 5 — Hierarchical (coarse-to-fine) sampling. A "coarse" network is evaluated at uniformly stratified samples first; the resulting per-sample weights $w_i = T_i(1-e^{-\sigma_i\delta_i})$, normalized, form a piecewise-constant probability density along the ray. A second, denser set of samples is then drawn from this density (inverse-transform sampling) and combined with the coarse samples to evaluate a "fine" network — concentrating the finer, more expensive computation near surfaces (where $w_i$ is large) instead of empty space or fully-occluded regions.

Step 6 — Loss and optimization. Both the coarse and fine rendered pixel colors are compared to ground truth via squared error, summed over a random batch of rays per iteration, and backpropagated with Adam to update all network weights. No explicit geometric regularizer is required, though later variants add depth or distortion losses.

Step 7 — Novel-view synthesis (inference). Given a new camera pose, repeat steps 1–5 (using only the fine network, or both, averaged) for every pixel to synthesize the output image. ``` function RENDER_RAY(o, d, t_near, t_far, N_coarse, N_fine, MLP_coarse, MLP_fine):

# --- Coarse pass: uniform stratified sampling --- bins <- partition([t_near, t_far] into N_coarse equal bins) t_c <- [uniform_random_sample(bin) for bin in bins] # stratified samples sigma_c, color_c <- [], [] for t in t_c: x <- o + t * d gamma_x <- positional_encode(x, L=10) gamma_d <- positional_encode(d, L=4) sigma, feat <- MLP_coarse.density_head(gamma_x) c <- MLP_coarse.color_head(feat, gamma_d) sigma_c.append(sigma); color_c.append(c) C_coarse, weights <- VOLUME_RENDER(t_c, sigma_c, color_c)

# --- Importance resampling from coarse weights --- pdf <- normalize(weights) # piecewise-constant density along ray t_f <- inverse_transform_sample(pdf, N_fine) # new samples concentrated near surfaces t_all <- sort(t_c UNION t_f)

# --- Fine pass: dense evaluation --- sigma_f, color_f <- [], [] for t in t_all: x <- o + t * d gamma_x <- positional_encode(x, L=10) gamma_d <- positional_encode(d, L=4) sigma, feat <- MLP_fine.density_head(gamma_x) c <- MLP_fine.color_head(feat, gamma_d) sigma_f.append(sigma); color_f.append(c) C_fine, _ <- VOLUME_RENDER(t_all, sigma_f, color_f) return C_coarse, C_fine

function VOLUME_RENDER(t, sigma, color): N <- length(t) T <- 1.0 C <- (0, 0, 0) weights <- array of size N for i in 1..N: delta_i <- (i < N) ? t[i+1] - t[i] : 1e10 # last interval extends to infinity alpha_i <- 1 - exp(-sigma[i] delta_i) # opacity contributed by segment i w_i <- T alpha_i C <- C + w_i color[i] weights[i] <- w_i T <- T (1 - alpha_i) # update transmittance return C, weights

function TRAIN(images, poses, intrinsics, MLP_coarse, MLP_fine, iters): for step in 1..iters: batch <- sample_random_rays(images, poses, intrinsics, batch_size=4096) loss <- 0 for (o, d, C_gt) in batch: C_coarse, C_fine <- RENDER_RAY(o, d, t_near, t_far, N_coarse=64, N_fine=128, MLP_coarse, MLP_fine) loss <- loss + ||C_coarse - C_gt||^2 + ||C_fine - C_gt||^2 backpropagate(loss) Adam_step(MLP_coarse.params, MLP_fine.params) ```

Why the quadrature is a correct approximation. The continuous transmittance $T(t) = \exp(-\int_{t_n}^t \sigma(s)\,ds)$ is the solution of the linear ODE $T'(t) = -\sigma(t)T(t)$, $T(t_n)=1$ — a first-order absorption law identical to the Beer–Lambert law in optics. Discretizing the ray into $N$ segments of density assumed piecewise constant equal to $\sigma_i$ on segment $i$, the exact transmittance across that segment is $\exp(-\sigma_i \delta_i)$, and the exact expected radiance contributed by segment $i$ (accounting for emission along the segment and outgoing attenuation) is $T_i(1-e^{-\sigma_i\delta_i})c_i$ under the assumption that color $c_i$ is also constant across the segment. Summing these gives exactly the estimator $\hat C(\mathbf r)$ used above — it is not a crude Euler approximation but the exact solution of the volume rendering equation under the piecewise-constant assumption, which becomes arbitrarily accurate as $N\to\infty$ and $\delta_i \to 0$ by the mesh-refinement limit of Riemann sums. Stratified (jittered) sampling additionally makes $\hat C(\mathbf r)$ an unbiased Monte Carlo estimator of the true integral in expectation over the random offsets, since each stratum contributes its own independent uniform sample rather than a fixed grid point — this is precisely why the network is trained on a continuum of positions rather than memorizing $N$ fixed depths.

Why hierarchical sampling helps without introducing bias. The fine samples are drawn from the empirical density induced by the coarse weights $w_i$, which by construction is large exactly where the ray's coarse pass estimated substantial radiance contribution (i.e., near opaque surfaces) and near zero in free space or fully-occluded regions. This is textbook importance sampling: allocating more samples where the integrand is large reduces the variance of the finite-sample estimate for a fixed sample budget, without biasing the estimator, since the final compositing still uses the actual (position-dependent) $\sigma, c$ evaluated at the resampled points, not the coarse network's stale predictions.

Why positional encoding is necessary, not cosmetic. A neural tangent kernel argument explains the effect: standard MLPs with smooth activations have kernels that decay rapidly at high frequencies, exhibiting a strong spectral bias that makes them converge slowly on and often fail to represent fine spatial detail (sharp edges, thin structures) when the raw low-dimensional coordinate is the input. Composing the input with fixed sinusoids of exponentially increasing frequency reshapes the effective kernel to have significant support at high frequencies, letting the same MLP architecture fit high-frequency signals — an empirically and theoretically documented effect, not a heuristic add-on.

Time complexity. Let $R$ be rays per training batch, $N = N_{\text{coarse}} + N_{\text{fine}}$ samples per ray, and $C$ the cost of one MLP forward/backward pass. One training iteration costs $O(R \cdot N \cdot C)$ — every sample along every ray requires an independent network evaluation, since the field has no shared spatial structure to exploit (this is the classic implicit-representation tradeoff: compactness of parameters versus cost of querying). With $R \approx 4096$, $N \approx 192$, and a modest 8-layer, 256-wide MLP, a single scene commonly requires $10^5$–$3\times10^5$ iterations, translating to one to two GPU-days on a single high-end GPU in the original formulation. Rendering one $800\times800$ novel view costs $O(HW \cdot N \cdot C)$ — proportional to total pixels times samples times per-query cost, with no reuse across pixels, which is the main practical bottleneck addressed by later accelerated variants.

Space complexity. The learned representation itself is remarkably compact: $O(P)$ where $P$ is the MLP parameter count, typically a few megabytes, independent of scene resolution or extent — a sharp contrast to explicit voxel grids or point clouds whose memory scales with $O(\text{resolution}^3)$ or $O(\text{point count})$. This compactness is a direct benefit of the implicit-function representation, at the cost of the query-time expense above.

Honest tradeoffs. NeRF assumes a static scene with fixed lighting and accurately known camera poses; it has no native mechanism for dynamic content, relighting, or pose uncertainty (all addressed by later extensions). Per-scene optimization means a trained NeRF does not generalize to new scenes without retraining, unlike a feed-forward image-to-image network. The reliance on volumetric density rather than an explicit surface can create "floaters" (small suspended blobs of density) when views are sparse or inconsistent, since the optimization problem is under-constrained by too few rays.

Consider a single ray passing near a red opaque sphere, sampled at five points along the ray with the network (hypothetically, post-training) producing the following outputs. Distances $t_i$ are in scene units; density $\sigma_i$ in units of inverse length; colors are RGB in $[0,1]$.

Step through the transmittance recursion explicitly:

• $T_1 = 1.0$ (nothing occludes the first segment).
• After segment 1: $T_2 = T_1(1-\alpha_1) = 1.0 \times 0.99501 = 0.99501$.
• After segment 2: $T_3 = T_2(1-\alpha_2) = 0.99501 \times 0.99005 = 0.98511$.
• After segment 3 (this is where the sphere surface sits — density jumps to 8.0): $T_4 = T_3(1-\alpha_3) = 0.98511 \times 0.01832 = 0.01805$ (matches table to rounding).
• After segment 4: $T_5 = T_4(1-\alpha_4) = 0.01805 \times 0.02346 = 0.000424$.

The rendered pixel color is the weighted sum:

The result is a nearly saturated red, exactly as expected: samples 3 and 4 (which straddle the sphere's surface, where density spikes) dominate the weighted sum because they carry almost all of the accumulated transmittance-times-opacity mass ($w_3+w_4\approx 0.984$ of the total), while the low-density background samples (blue sky, color (0.1,0.1,0.9)) contribute almost nothing — precisely because $T$ has already collapsed to near zero by the time the ray reaches them. This traces exactly why hierarchical sampling concentrates fine-pass samples around $t\approx 3.0$–$3.5$: that is where $w_i$ peaks, and it is where additional samples reduce estimator variance the most.

The volume rendering equation NeRF relies on predates deep learning by decades, originating in physically-based computer graphics for rendering smoke, clouds, and medical CT/MRI volumes. What NeRF contributed was recognizing that this classical, fully differentiable integral could serve as the renderer inside an end-to-end learned pipeline, with a coordinate-based MLP as the scene representation being differentiated through.

NeRF-style representations underpin modern novel-view synthesis products for real-estate walkthroughs, cultural-heritage digitization, and e-commerce product visualization, where a handful of photographs must become a fully explorable 3D asset. Robotics and autonomous-driving teams use radiance-field reconstructions of street scenes for simulation and sensor-data augmentation, since a trained field can render synthetic camera views from arbitrary novel trajectories for testing perception stacks. Visual effects and virtual production pipelines use accelerated NeRF variants to convert on-set footage into relightable, camera-path-free 3D scene proxies. The broader research trajectory — encoding a signal as the weights of a small network, queried continuously rather than stored on a grid — reappears throughout implicit neural representations for signed distance functions, textures, and even physical fields, and its differentiable-rendering recipe (render, compare to real pixels, backpropagate) is a direct conceptual ancestor of vision-conditioned generative systems and the fusion of 3D scene understanding with language, foreshadowing the broader convergence explored under Multimodal Models. It also builds directly on techniques from Image Generation Fundamentals (photometric losses, coordinate-based networks) and complements point-cloud-based geometry pipelines from 3D Vision and Point Clouds by supplying the missing photorealistic, view-dependent appearance model.

Ask the tutor: How does 3D Gaussian Splatting replace NeRF's implicit MLP with an explicit, rasterizable representation while keeping differentiable rendering?