Neural 3D, from radiance fields and Gaussian splatting to learned geometry

Why this subject matters now

Two papers reset the field in three years. NeRF (Mildenhall et al., 2020) showed that a five-layer MLP mapping a 3D coordinate and a viewing direction to color and density, trained by nothing more than rendering rays and comparing them to photographs, produces novel views of photographic quality, with view-dependent highlights and correct occlusion, from a few dozen input images. It did this with no mesh, no explicit geometry, no correspondence step. The whole scene lived in the weights of a network smaller than a single texture. Three years later, 3D Gaussian Splatting (Kerbl et al., 2023) kept the differentiable image-formation idea but threw out the MLP and the ray marching, representing the scene as a few million anisotropic Gaussians rasterized with a tile-based splatting kernel, and reached real-time rendering (well over 100 frames per second at 1080p) with training in minutes rather than the day a vanilla NeRF took. Between and around those two results, an entire subfield reorganized. The multi-view stereo pipelines that shipped in every photogrammetry product were suddenly a baseline rather than the state of the art, and the vocabulary of graphics, transmittance, alpha compositing, anti-aliasing, merged with the vocabulary of deep learning.

A practitioner today is expected to understand several things that were niche or nonexistent five years ago. Why does a coordinate network need positional encoding to represent anything sharp, and what exactly is the spectral-bias argument that explains it. What is the precise relationship between NeRF's alpha-compositing sum and the physical volume rendering integral, and where does every term come from. Why is a hash grid faster than a dense voxel grid without being much larger. Why can Gaussian splatting rasterize in real time when NeRF cannot, expressed not as folklore but as a FLOP and memory-access argument. How does score distillation turn a 2D image diffusion model into a 3D generator, and why does its gradient drop the diffusion U-Net's Jacobian. And, cutting across all of it, why PSNR looking good is nearly uncorrelated with the geometry being correct. This page derives each of these rather than asserting them, and works the arithmetic so the numbers are real.

The classical baseline is anchored on Hartley and Zisserman, Multiple View Geometry in Computer Vision, the standard reference for projective camera geometry, the fundamental and essential matrices, and bundle adjustment. The neural material is anchored on the primary papers, cited inline. The image-formation physics is not re-derived here where the companion rendering pages already own it. This page owns the learned representations and the differentiable fitting of them.

The classical baseline of multi-view geometry

To understand what NeRF replaced, and what it did not, one has to hold the classical pipeline in view. The problem is inverse. Given a set of photographs of a static scene from unknown viewpoints, recover both the camera poses and the 3D geometry. For decades the answer was a chain of geometric estimators, each solving a well-posed subproblem, and the neural methods inherited the front half of that chain (pose estimation) while replacing the back half (the scene representation and its rendering).

The pinhole camera and projection

A pinhole camera maps a 3D world point \( \mathbf{X} = (X, Y, Z)^{\top} \) to a 2D image point by central projection through the optical center. In homogeneous coordinates the map is linear,

The extrinsics \( [\,R \mid \mathbf{t}\,] \), a rotation and translation, place the world in the camera's frame. The intrinsic matrix \( K \) then encodes focal lengths \( f_x, f_y \) in pixels, the principal point \( (c_x, c_y) \), and a skew \( s \) (almost always zero). The scalar \( \lambda \) is the depth along the optical axis, and dividing through by it, the perspective division, is the only nonlinear step. It is what makes distant objects small and what makes the map non-invertible from a single image, because every point on a ray projects to the same pixel. This projective camera, its calibration, and the meaning of \( K \) are derived in full on rendering foundations. What matters here is that the forward map is exactly differentiable, and its Jacobian will reappear when Gaussians are projected.

The essential and fundamental matrices of two views

Two cameras looking at the same scene are constrained. If a 3D point projects to \( \mathbf{x} \) in the first image and \( \mathbf{x}' \) in the second (both in normalized camera coordinates, so \( K \) has been divided out), then the two rays and the baseline between the camera centers are coplanar. Coplanarity of three vectors is a vanishing scalar triple product, and in normalized coordinates it becomes the epipolar constraint

where \( E \), the essential matrix, packages the relative rotation \( R \) and the skew-symmetric cross-product matrix \( [\mathbf{t}]_{\times} \) of the baseline. In pixel coordinates, before dividing out the intrinsics, the same relation reads \( \mathbf{x}'^{\top} F\, \mathbf{x} = 0 \) with the fundamental matrix \( F = K'^{-\top} E\, K^{-1} \). The practical content is that \( E\mathbf{x} \) is the epipolar line in the second image on which the match must lie, collapsing the correspondence search from two dimensions to one. \( E \) can be estimated from as few as five point correspondences (Nister's five-point algorithm) or, more simply, eight (the normalized eight-point algorithm), and factored into \( R \) and \( \mathbf{t} \) up to a scale and a four-fold ambiguity resolved by requiring reconstructed points to lie in front of both cameras.

Structure from motion, bundle adjustment, and MVS

Scaling from two views to hundreds is structure from motion (SfM). Detect and match features (SIFT and its successors) across all images, chain pairwise pose estimates into a global set of camera poses and a sparse point cloud, and then refine everything jointly. That joint refinement is bundle adjustment, the nonlinear least-squares problem at the heart of the pipeline,

where \( \pi \) is the projection above, \( \mathbf{x}_{ij} \) is the observed image location of point \( i \) in image \( j \), and \( v_{ij} \) is one when that point is visible there. This is minimized by Levenberg-Marquardt, exploiting the bipartite sparsity between cameras and points (the Schur complement trick) so that thousands of cameras and millions of points remain tractable. The derivation of Gauss-Newton, Levenberg-Marquardt, and the Schur complement lives on numerical methods. The output is accurate camera poses and a sparse cloud. Multi-view stereo (MVS) then densifies it, estimating a depth for nearly every pixel by photo-consistency across views and fusing the depth maps into a dense point cloud or a mesh via Poisson surface reconstruction or marching cubes.

The crucial observation for everything that follows is that neural methods did not replace SfM. NeRF and Gaussian splatting both begin from camera poses produced by COLMAP, the standard SfM tool. What they replaced is the back end, the mesh-plus-texture scene representation and its rasterizer, with a continuous, differentiable representation fit directly to pixels. The front-end pose estimation and the back-end representation had always been separate concerns. The neural revolution was entirely in the second.

Implicit representations, occupancy and signed-distance fields

Before radiance fields, the deep-learning-for-3D community had already settled on a powerful idea. Represent a shape not as a mesh or voxel grid but as a continuous function, evaluated by a neural network, that classifies or measures every point in space. Two forms dominate, and both matter for understanding NeRF, because NeRF's density field is a close cousin.

Occupancy networks

Mescheder et al. (2019, Max Planck and Autonomous Vision Group) represent a solid shape by its indicator function, learned as a classifier \( f_\theta : \R^3 \times \mathcal{Z} \to [0,1] \) that maps a query point (and a shape latent code \( z \)) to the probability that the point is inside the object. The surface is then the decision boundary, the level set \( \{ \mathbf{x} : f_\theta(\mathbf{x}, z) = \tau \} \) at some threshold \( \tau \). The representation is continuous and resolution-free. Memory is the size of the network, not the cube of a grid resolution, and the surface can be extracted at any resolution by running marching cubes on a sampled grid of occupancy values. Training is a binary cross-entropy on points sampled in space with known inside/outside labels. The limitation is that occupancy is a discontinuous target (a hard step at the surface), so gradients near the boundary are uninformative, which motivates the smoother signed-distance formulation.

Signed distance fields and the eikonal constraint

A signed distance field (SDF) assigns to every point the signed distance to the nearest surface, negative inside, positive outside, zero on the surface. DeepSDF (Park et al., 2019, UC San Diego and collaborators) learns \( f_\theta(\mathbf{x}, z) \approx \mathrm{SDF}(\mathbf{x}) \) by regressing distance values, and the surface is again a level set, now the zero set \( \{ \mathbf{x} : f_\theta(\mathbf{x}) = 0 \} \). An SDF carries strictly more information than occupancy. Its gradient is the outward surface normal, and its magnitude tells you how far you are from the surface, which lets sphere-tracing renderers take large steps in empty space.

A true signed distance function is not an arbitrary scalar field. It satisfies a partial differential equation. Move a small step \( \delta \) along the direction to the nearest surface point and the distance changes by exactly \( \delta \). Move perpendicular to that direction and, to first order, the nearest-surface distance is unchanged. So the directional derivative of the distance along its own gradient is one and there is no faster direction of change, which is precisely the statement that the gradient is a unit vector everywhere it is differentiable,

This is the eikonal equation. It is the defining property that separates a genuine distance function from any old level-set representation of the same surface, and modern SDF fitters (IGR, Gropp et al. 2020, and the neural-SDF branch of NeRF successors such as VolSDF and NeuS) enforce it as a soft penalty during training,

with \( \nabla_{\mathbf{x}} f_\theta \) computed by automatic differentiation through the network. Without this regularizer, a network fit only to zero-crossings can represent the same surface with a wildly non-metric field whose gradient is useless as a normal and whose level sets bunch and stretch. The eikonal term is what makes the learned field an actual distance, and therefore what makes its gradient a usable normal. The connection to NeRF is that both an SDF and a NeRF density are continuous fields queried by a coordinate network, and the same spectral-bias problem afflicts both, which is the next derivation.

Coordinate MLPs and the need for positional encoding

Every representation above evaluates a network at a raw spatial coordinate. It is a notable empirical fact, and the most important trick in the area, that a standard ReLU MLP fed raw coordinates cannot represent high-frequency detail. Fit it to a sharp image or a detailed shape and it produces a blurry, over-smoothed result no matter how long you train. NeRF's authors found the same, and the fix, mapping coordinates through a bank of sinusoids before the network, came with a theory from Tancik et al. (2020, Berkeley) that explains exactly why.

The spectral-bias argument

The training dynamics of a wide network are governed, to leading order, by its neural tangent kernel (NTK). For a network \( f_\theta \) trained by gradient descent on a squared loss, the evolution of the function values on the training inputs is, in the infinite-width limit, linear,

where \( K \) is the NTK Gram matrix, constant in this limit. Diagonalize \( K = \sum_k \eta_k\, \mathbf{v}_k \mathbf{v}_k^{\top} \). In the eigenbasis, the error component along eigenvector \( \mathbf{v}_k \) decays as \( e^{-\eta_k t} \). Large-eigenvalue components are learned fast, small-eigenvalue components slowly. The problem is that for a plain coordinate MLP the NTK is a rapidly decaying function of frequency, its eigenvalues fall off like a power of the frequency, so the high-frequency components of the target have tiny \( \eta_k \) and are learned so slowly that within any realistic training budget they are never learned at all. This is the spectral bias. A coordinate MLP is a low-pass filter on the target.

Why Fourier features fix it

Tancik et al. show that composing the network with a fixed feature map of sinusoids reshapes the NTK into a stationary kernel whose bandwidth you control directly. Map each input coordinate through

with frequencies \( b_1, \dots, b_m \). The reason this works is a one-line trigonometric identity. The inner product of two encoded points, which is what the composed kernel sees, is

using \( \cos(A - B) = \cos A \cos B + \sin A \sin B \). The right-hand side depends only on the difference \( x - x' \), so the induced kernel is stationary, a sum of cosines whose frequencies are exactly the \( b_j \) you chose. Composing a stationary kernel with the network's NTK gives a composed kernel that is again stationary and whose effective bandwidth is set by the largest \( b_j \). By including high frequencies in \( \gamma \), you inject large-eigenvalue directions at those frequencies, so the high-frequency components of the target are now learned at a comparable rate to the low ones, and the network can represent sharp detail. NeRF uses the axis-aligned geometric schedule \( b_j = 2^{j} \) for \( j = 0, \dots, L-1 \), which is the special "positional encoding" case, with \( L = 10 \) bands for position and \( L = 4 \) for viewing direction.

NeRF and the volume rendering integral

NeRF represents a scene as a function \( F_\theta(\mathbf{x}, \mathbf{d}) = (\mathbf{c}, \sigma) \) mapping a 3D position and a unit viewing direction to an emitted RGB color \( \mathbf{c} \) and a scalar volume density \( \sigma \). Density depends only on position (geometry is view-independent). Color depends on both (to capture specular highlights). To render a pixel, cast the camera ray \( \mathbf{r}(t) = \mathbf{o} + t\mathbf{d} \) through it and integrate the emitted color along the ray, weighted by how likely the ray is to have survived to each point. That weighting is the physics of a participating medium, and it is worth deriving rather than asserting.

From radiative transfer to the rendering integral

Treat the scene as an emissive, absorbing medium with no scattering, the emission-absorption model. A photon travelling along the ray is absorbed with probability \( \sigma(\mathbf{r}(t))\, dt \) per unit length. Let \( T(t) \) be the transmittance, the probability that the ray travels from its start to \( t \) without being absorbed. Over a further step \( dt \), the survival probability multiplies by \( (1 - \sigma\, dt) \), so

a linear ODE whose solution is the Beer-Lambert law,

The exponent \( \int \sigma\, ds \) is the optical depth. Now, the radiance reaching the camera is the sum over all points of the color emitted there times the probability the ray both reaches that point (factor \( T(t) \)) and is absorbed exactly there to deposit its energy toward the sensor (factor \( \sigma(\mathbf{r}(t))\, dt \)). Integrating,

This is exactly the emission-only case of the volume rendering equation derived on physically based rendering, with the medium's emission and extinction coefficients replaced by the network's \( \mathbf{c} \) and \( \sigma \). NeRF is not a new physics. It is the classical volume rendering integral with a learned integrand. The one liberty NeRF takes is that \( \sigma \) plays the role of both the extinction that attenuates the ray and the source strength that emits toward the camera, which is the emission-absorption model, appropriate because a surface point should both block what is behind it and contribute its own color.

Quadrature, the alpha-compositing sum

The integral has no closed form for a general network, so NeRF evaluates it by numerical quadrature. Partition \( [t_n, t_f] \) into \( N \) intervals at sample points \( t_1 < \dots < t_N \) and assume the density and color are piecewise constant, taking values \( \sigma_i, \mathbf{c}_i \) on the \( i \)-th interval of length \( \delta_i = t_{i+1} - t_i \). Over one interval where \( \sigma \) is constant, the transmittance drop is analytic. From the ODE, \( T \) decays by the factor \( \exp(-\sigma_i \delta_i) \) across the interval. Define the interval's opacity, the probability of absorption within it given the ray reached its start, as the complement of that survival,

The accumulated transmittance up to the start of interval \( i \) is the product of survivals over all earlier intervals,

which is the discrete Beer-Lambert law. The integral \( \int T\sigma\mathbf{c}\, dt \) over interval \( i \), with \( \mathbf{c}_i \) constant, is \( \mathbf{c}_i T_i \int_0^{\delta_i} e^{-\sigma_i s}\sigma_i\, ds = \mathbf{c}_i T_i (1 - e^{-\sigma_i \delta_i}) = T_i \alpha_i \mathbf{c}_i \). Summing over intervals gives NeRF's rendering equation,

Every term is now accounted for. \( \alpha_i \) comes from integrating the constant-density ODE over an interval, \( T_i \) is the running product of survival probabilities, and the weight \( w_i = T_i \alpha_i \) is the probability that the ray is absorbed exactly in interval \( i \). This is precisely the "over" alpha-compositing operator from graphics, front to back, which is not a coincidence. Alpha compositing has always been an approximation to this same transport integral. If the network puts all density in one interval (\( \alpha_k \to 1 \), others zero), the sum collapses to \( \mathbf{c}_k \), a hard surface. A soft \( \sigma \) profile gives a soft, semi-transparent edge.

Stratified and hierarchical sampling

Where the samples \( t_i \) go matters a great deal, because a surface occupies a thin slab of the ray and uniform sampling wastes almost all samples in empty space. NeRF addresses this in two ways. The first is stratified sampling. Rather than fixed sample locations, it partitions \( [t_n, t_f] \) into \( N \) equal bins and draws one sample uniformly within each bin, \( t_i \sim \mathcal{U}[t_n + \tfrac{i-1}{N}(t_f - t_n), t_n + \tfrac{i}{N}(t_f - t_n)] \). This makes the quadrature an unbiased estimator of the integral and, because the sample positions vary every iteration, lets the continuous field be supervised everywhere rather than at a fixed grid, preventing the network from overfitting to sample locations.

Second, hierarchical sampling with two networks, a coarse and a fine. The coarse network is rendered with \( N_c \) stratified samples, producing weights \( w_i \). Those weights, normalized to a probability distribution \( \hat{w}_i = w_i / \sum_j w_j \), are a piecewise-constant estimate of where the ray's opacity actually lives. The fine network then draws \( N_f \) additional samples from this distribution by inverse-transform sampling of its CDF, concentrating them near the surface, and renders with all \( N_c + N_f \) samples combined. This is importance sampling of the transport integral, the same variance-reduction principle derived on physically based rendering. Put samples where the integrand is large. In the original NeRF, \( N_c = 64 \) and \( N_f = 128 \).

The training objective and the cost

Training minimizes the squared photometric error between rendered and observed pixels, summed over both coarse and fine renders,

with \( \mathcal{R} \) a minibatch of rays sampled across all training images. There is no explicit geometry supervision at all. The density field emerges purely from the constraint that the composited colors match the photographs from every view, and multi-view consistency is what forces \( \sigma \) to concentrate on the true surface. This is the entire training signal, and it is notable that it suffices.

The cost is the problem. Rendering one pixel requires \( N_c + N_f \approx 192 \) network evaluations, and an \( 800 \times 800 \) image is 640,000 pixels, so a single frame is on the order of \( 10^8 \) forward passes through an eight-layer MLP. Training to convergence on one scene took the original implementation one to two days on a single GPU, and rendering a frame took tens of seconds. The bottleneck is intrinsic to the design. A query is a full MLP evaluation, the samples are dense because the network gives no hint where the surface is until it is trained, and nothing is cached between rays. Every acceleration method that follows attacks one of these three costs.

From density to surfaces, the SDF-as-density bridge

A NeRF density field is convenient for rendering but a poor surface. The level set of \( \sigma \) that best matches the images is ambiguous, and extracting a clean mesh from it is unreliable, which is why raw NeRFs produce noisy geometry even at high PSNR. The fix that unified the two halves of this page is to represent geometry as a signed distance field and derive the volume density from it, so that the eikonal-regularized surface is what gets rendered. VolSDF (Yariv et al., 2021, Weizmann) and NeuS (Wang et al., 2021, Hong Kong and Max Planck) do exactly this. VolSDF sets the density to a scaled Laplace cumulative distribution of the signed distance \( d(\mathbf{x}) = f_\theta(\mathbf{x}) \),

so that far outside the surface (\( d \gg 0 \)) the density is near zero, far inside (\( d \ll 0 \)) it saturates to \( \alpha \), and the transition sharpens as the learned scale \( \beta \to 0 \). The parameter \( \beta \) controls how surface-like the medium is. Large \( \beta \) is a soft cloud that renders stably early in training, and annealing \( \beta \) down turns it into a hard surface as optimization converges. The payoff is that the geometry being optimized is a genuine, eikonal-constrained distance function whose zero set is a clean surface, while rendering still goes through the same volume-rendering quadrature. This is the reconciliation of the implicit-surface line (DeepSDF, occupancy) with the radiance-field line (NeRF). The surface is an SDF, the appearance is volume-rendered, and the eikonal constraint from earlier is what keeps the recovered geometry metric.

Acceleration with hash grids and explicit voxels

The insight shared by the fast methods is that the expensive part of NeRF is not the volume rendering, it is the per-sample MLP evaluation, and most of what the MLP is doing is memorizing a smooth spatial field that a lookup table could store directly. Replace most of the network with a learned spatial data structure and the per-query cost collapses.

Instant-NGP and the multiresolution hash grid

Muller et al. (2022, NVIDIA) encode position not with fixed sinusoids but with a learned, multiresolution hash grid, and pair it with a tiny MLP. The encoding has \( L \) levels of resolution arranged geometrically between a coarsest \( N_{\min} \) and a finest \( N_{\max} \), with growth factor

At each level, space is a grid of resolution \( N_\ell \). Each grid vertex holds a learnable feature vector of length \( F \) (typically \( F = 2 \)). To encode a point \( \mathbf{x} \) at level \( \ell \), find the \( 2^3 = 8 \) surrounding grid vertices, look up each one's feature, and trilinearly interpolate them by the point's fractional position within the cell. Concatenating across all \( L \) levels gives the encoding, of length \( L \cdot F \), which feeds the small MLP.

The trick is how vertex features are stored. At coarse levels the grid has few enough vertices to store densely, one feature per vertex. At fine levels a dense grid would be enormous (a \( 2048^3 \) grid has \( 8.6 \) billion vertices), so instead each level has a hash table of a fixed capacity \( T \) (typically \( 2^{19} = 524{,}288 \)) entries, and a vertex at integer coordinate \( \mathbf{v} \) maps to a slot by a spatial hash

where \( \oplus \) is bitwise XOR and \( \pi_k \) are large fixed primes (Instant-NGP uses \( \pi_1 = 1 \), \( \pi_2 = 2\,654\,435\,761 \), \( \pi_3 = 805\,459\,861 \)). At fine levels many distinct vertices collide into the same slot, and Instant-NGP does not resolve the collisions at all. It lets gradient descent sort them out. Because the important, high-density surface regions occupy a tiny fraction of the fine grid, the features that matter mostly do not collide with each other, and where they do, the coarser levels and the MLP disambiguate. This is the whole reason it is fast. A query is \( L \) hash lookups plus \( L \) trilinear interpolations plus a two-layer MLP, all of which fit in cache and run in microseconds, versus a deep MLP over a 60-dimensional Fourier encoding. Instant-NGP trains a NeRF-quality scene in seconds to minutes rather than a day.

Plenoxels and DVGO, no MLP at all

Fridovich-Keil et al. (2022, Berkeley, "Plenoxels: radiance fields without neural networks") take the argument to its conclusion and drop the MLP entirely. Store the scene as a sparse voxel grid where each occupied voxel holds a density and a set of spherical-harmonic coefficients for view-dependent color, and render by trilinearly interpolating those values at each sample and applying the same alpha-compositing sum. Because the representation is now a plain grid of numbers, there is nothing to forward-pass. The entire model is the grid, optimized directly by gradient descent on the photometric loss with a total-variation regularizer for smoothness. Plenoxels matches NeRF's quality and trains two orders of magnitude faster, which established that the neural network in NeRF was never doing the essential work. The differentiable volume rendering was. DVGO (Sun et al., 2022) reaches the same conclusion with a hybrid density-voxel-plus-tiny-MLP design. These results reframed NeRF. The representation is interchangeable, and what carries the method is the image-formation model you differentiate through.

3D Gaussian Splatting

Gaussian splatting (Kerbl et al., 2023, INRIA and Max Planck) is the representation that made radiance-field rendering real-time. It keeps the differentiable image-formation idea and the alpha-compositing, but replaces both the MLP and the ray marching with an explicit set of 3D Gaussian primitives that are rasterized, not ray-marched. The difference between marching a ray (asking, for each pixel, what is along it) and rasterizing primitives (asking, for each primitive, which pixels it covers) is the difference between an offline and a real-time renderer, and the whole design follows from choosing the second.

The 3D Gaussian primitive

Each primitive is an anisotropic 3D Gaussian with a mean (center) \( \boldsymbol{\mu} \in \R^3 \), a \( 3\times3 \) covariance \( \Sigma \), an opacity \( o \in [0,1] \), and a set of spherical-harmonic coefficients encoding view-dependent color. The density it contributes at a point is \( \exp\!\big( -\tfrac12 (\mathbf{x} - \boldsymbol{\mu})^{\top} \Sigma^{-1} (\mathbf{x} - \boldsymbol{\mu}) \big) \). To keep \( \Sigma \) positive semidefinite under gradient descent, it is not optimized directly but parameterized by a rotation and a scale,

with \( R \) a rotation stored as a unit quaternion and \( S \) a diagonal of positive scales (stored in log space). This factorization is exactly an ellipsoid's principal-axes decomposition. \( S \) sets the axis lengths, \( R \) orients them, and \( SS^{\top} \) conjugated by \( R \) is guaranteed a valid covariance for any parameter values, so the optimizer can move freely.

Projection to 2D via the EWA splatting Jacobian

Rasterizing a 3D Gaussian requires its 2D footprint on the image, and a Gaussian's image under the camera is not exactly a Gaussian because the perspective projection is nonlinear. The elliptical weighted average (EWA) framework (Zwicker et al., 2001) handles this by linearizing the projection at the Gaussian's center. To first order, an affine map sends a Gaussian to a Gaussian, with the covariance transformed by the map's Jacobian. Let \( W \) be the viewing transform (world to camera) and \( J \) the Jacobian of the perspective projection evaluated at the camera-space mean. Then the projected 2D covariance is

For a camera-space mean \( (x, y, z) \) with focal lengths \( f_x, f_y \), the projection is \( (u, v) = (f_x x / z, f_y y / z) \), and its Jacobian is

the derivatives of the perspective-divided coordinates, the same projection Jacobian from the pinhole camera above. \( \Sigma' \) is \( 2\times2 \). Dropping the third row and column of the projected covariance keeps only the image-plane extent. In practice a small isotropic term is added to \( \Sigma' \) (a low-pass or dilation filter, \( \Sigma' \leftarrow \Sigma' + \epsilon I \)) so that Gaussians never shrink below one pixel and alias. This is the anti-aliasing role the EWA filter played in its original texture-mapping context. The inverse \( \Sigma'^{-1} \) is the conic that evaluates the 2D Gaussian's falloff at each pixel offset.

View-dependent color with spherical harmonics

A surface point does not have one color. A specular highlight makes it depend on the viewing direction. NeRF captures this by feeding the direction \( \mathbf{d} \) into the color network. Gaussian splatting and Plenoxels, having no per-query network, instead store the color of each primitive as a set of spherical-harmonic coefficients and evaluate the color as a function of direction on the fly. Spherical harmonics \( Y_\ell^m(\mathbf{d}) \) are an orthonormal basis for functions on the sphere, the angular analogue of a Fourier series, and a band-limited color function is the truncated sum

with learnable coefficient vectors \( \mathbf{k}_\ell^m \in \R^3 \) (one per color channel). The \( \ell = 0 \) term is a constant, the diffuse base color. Higher bands add view-dependence, and 3DGS uses up to \( L = 3 \), which is \( (L+1)^2 = 16 \) coefficients per channel, or 48 numbers of color per Gaussian. This is why a Gaussian is roughly 59 floats, 3 for position, 4 for the rotation quaternion, 3 for scale, 1 for opacity, and 48 for third-order spherical-harmonic color. Storing color as a fixed low-order basis rather than a network is what keeps per-primitive evaluation to a handful of multiply-adds, which the real-time rasterizer needs.

Tile-based differentiable rasterization

Given every Gaussian's 2D mean, 2D covariance, opacity, and view-dependent color, the renderer composites them front to back per pixel with the same alpha formula NeRF uses, but arrived at by rasterization rather than ray marching. The pixel color is

where \( \mathbf{p} \) is the pixel, \( \boldsymbol{\mu}'_i \) and \( \Sigma_i' \) are the projected mean and covariance, \( o_i \) is the learned opacity, and \( \mathcal{N} \) is the list of Gaussians overlapping the pixel, sorted by depth. The structural difference from NeRF is that \( \alpha_i \) here comes from evaluating a closed-form 2D Gaussian at the pixel, not from a network query at a sample point, so there is no MLP in the inner loop and no sampling along a ray at all.

The speed comes from the tile scheme. The image is divided into \( 16\times16 \)-pixel tiles. Each Gaussian is assigned to every tile its \( 2D \) footprint touches, and within a tile the Gaussians are sorted once by depth (a single global radix sort keyed by tile and depth). Every pixel in a tile then walks the same sorted list, accumulating \( \alpha \) and stopping early once transmittance drops below a threshold. Sorting once per tile rather than per pixel, and evaluating a closed-form Gaussian rather than an MLP, is what puts rendering in the real-time regime. The whole thing is differentiable. The backward pass walks the same per-tile lists in reverse, accumulating gradients to each Gaussian's position, covariance, opacity, and color, so the primitives are fit by the same photometric loss as NeRF.

Adaptive densification and why it hits real-time

The set of Gaussians is not fixed. Optimization starts from the sparse SfM point cloud (one Gaussian per point) and periodically adapts the population by adaptive density control. Gaussians whose accumulated positional gradient is large, meaning the region is under-reconstructed, are either cloned (in under-populated regions, to add coverage) or split into two smaller ones (in over-large regions, to add detail), and Gaussians whose opacity falls below a threshold are pruned. This grows the representation from thousands to a few million primitives, placing detail exactly where the loss demands it, the explicit-primitive analogue of NeRF's hierarchical sampling.

The real-time result is a FLOP-and-memory-access argument, not folklore. NeRF pays, per pixel, on the order of 192 evaluations of a deep MLP over a high-dimensional encoding, and the samples are dense because the ray marcher does not know where the surface is. Nothing is shared between pixels. Gaussian splatting pays, per pixel, a walk over the handful of sorted Gaussians overlapping that pixel, each a single exponential and a dot product, with the depth sort amortized across all \( 256 \) pixels of a tile and the Gaussians' parameters read once from memory. The arithmetic intensity is far higher and the memory access pattern far more regular. On modern hardware this is the gap between tens of seconds per frame and well over 100 frames per second. Kerbl et al. report real-time rendering at 1080p with training in tens of minutes, versus a NeRF's hours to a day, on comparable scenes. The cost is memory. A few million Gaussians with color coefficients can occupy hundreds of megabytes, versus a NeRF's few megabytes of weights, which is the trade the representation makes.

Point-cloud and mesh networks

Not every learned 3D representation is a field. When the data is a raw point cloud, a set of 3D points with no connectivity, the central difficulty is that the representation has no canonical order. The same shape can be listed in any of \( N! \) orderings, and a network that consumes it must give the same answer for all of them. This permutation invariance is the defining constraint of point-cloud learning, and PointNet (Qi et al., 2017) solved it with a clean argument.

PointNet and the symmetric-function argument

A function \( f \) of a set is permutation invariant if \( f(\{x_{\pi(1)}, \dots, x_{\pi(N)}\}) = f(\{x_1, \dots, x_N\}) \) for every permutation \( \pi \). PointNet's construction is to build \( f \) as

where \( h \) is a per-point MLP applied identically and independently to each point, \( \rho \) is a second MLP, and \( \operatorname{pool} \) is a symmetric aggregation, in PointNet the element-wise maximum over the \( N \) points. Invariance is immediate and exact. \( h \) is applied point-wise so permuting the points permutes the arguments of the pool, and the max (like sum or mean) is a symmetric function, unchanged by reordering its arguments. Composing with \( \rho \), which sees only the pooled vector, preserves invariance. The deeper result Qi et al. prove is a universal-approximation statement. Any continuous permutation-invariant set function can be approximated arbitrarily well by this max-pool form with a sufficiently high-dimensional \( h \), so nothing is given up by insisting on the symmetric structure. The max, they argue, effectively selects a sparse set of "critical points" that determine the output, giving robustness to outliers and to non-critical points being missing. PointNet++ (Qi et al., 2017) then applies this hierarchically over local neighborhoods to capture local geometric structure, the analogue of a convolution's receptive field.

Generative 3D by score distillation and feed-forward reconstruction

The methods so far reconstruct a specific scene from photographs of it. The generative problem is to synthesize a 3D asset that never existed, from a text prompt or a single image. The obstacle is data. There is no web-scale corpus of 3D models comparable to the billions of images that train 2D diffusion models. The dominant idea sidesteps this by distilling a 2D image prior into 3D.

Score distillation sampling

DreamFusion (Poole et al., 2022, Google) optimizes the parameters \( \theta \) of a 3D representation (a NeRF) so that its renderings, from random viewpoints, look like samples from a pretrained text-to-image diffusion model conditioned on the prompt. The 3D representation is never trained on 3D data. Its only supervision is the 2D diffusion model's opinion of its renders. The difficulty is defining a usable gradient. Let \( g(\theta) \) be a rendered image, \( x = g(\theta) \). A diffusion model provides a denoiser \( \epsilon_\phi(x_t; y, t) \) that predicts the noise added to a noised image \( x_t = \alpha_t x + \sigma_t \epsilon \) (the forward process and the noise-prediction parameterization are derived on diffusion and large vision models). The natural objective is the diffusion training loss on the rendered image,

Differentiating through it by the chain rule produces a term containing the Jacobian of the U-Net, \( \partial \epsilon_\phi / \partial x_t \), which is enormous (a full backward pass through the diffusion network per gradient step) and, empirically, poorly conditioned. The key move of score distillation sampling (SDS) is to drop that Jacobian term. Differentiating the loss,

and the SDS gradient discards the middle Jacobian factor, keeping

using \( \partial x_t / \partial \theta = \alpha_t\, \partial g / \partial \theta \). The interpretation justifies the omission. The quantity \( \epsilon_\phi(x_t) - \epsilon \) is, up to scale, the difference between the model's predicted noise and the actual noise added, which by the score-epsilon relationship is proportional to the score of the noised data distribution at \( x_t \). It points from the current render toward the region of high-probability images. SDS is thus gradient ascent on the diffusion model's log-density of the render, treating the denoiser as a fixed vector field (a score) rather than a function to backpropagate through. Dropping the Jacobian is what makes it cheap (one forward pass of the U-Net per step, no backward through it) and, because a well-trained denoiser's output already is the score, well-motivated rather than a hack. The cost is that SDS with the large guidance weights it requires tends to produce over-saturated, over-smoothed, low-diversity results, the mode-seeking behavior that ProlificDreamer later diagnosed.

Variational score distillation and feed-forward generation

ProlificDreamer (Wang et al., 2023, Tsinghua) reframes SDS as a special case of a variational problem. Rather than driving a single 3D asset toward the mode, treat the renders as a distribution and minimize its KL divergence to the diffusion prior, variational score distillation (VSD). The VSD gradient replaces the fixed noise target \( \epsilon \) with the score of the current render distribution, estimated by a second, LoRA-fine-tuned diffusion model tracking the renders. SDS is the degenerate case where that second score is taken to be the noise itself. VSD produces sharper, more diverse, more realistic 3D than SDS, at the cost of training the auxiliary network. Separately, the feed-forward line abandons per-scene optimization entirely. LRM (Hong et al., 2023, Adobe and collaborators), the large reconstruction model, trains a large transformer on a big corpus of 3D objects to map a single image directly to a triplane NeRF in a single forward pass, in seconds, with no optimization loop at all. These are the two poles of generative 3D as of this writing. One is slow, prior-distilling optimization (SDS/VSD) that needs no 3D data, and the other is fast feed-forward inference (LRM and its successors, including image-to-Gaussian-splat models) that needs a large 3D training set.

The evaluation-metric trap

View-synthesis papers report PSNR, SSIM, and LPIPS on held-out views, and it is essential to understand why all three can look excellent while the recovered geometry is wrong. PSNR is peak signal-to-noise ratio, a monotone function of pixel-wise mean squared error,

It rewards matching pixel values on the test views and nothing else. A method can composite density in physically wrong places, floaters in mid-air, a smeared cloud instead of a surface, and still reproduce every training and test view's pixels almost perfectly, because the alpha-compositing sum is many-to-one. Many density fields render to the same image from the observed viewpoints. High PSNR certifies that the renders match where you looked. It says nothing about the density field being a correct surface, and NeRFs are notorious for high-PSNR reconstructions riddled with floaters that only reveal themselves from unobserved angles. SSIM (structural similarity, comparing local luminance, contrast, and structure) and LPIPS (a learned perceptual distance in the feature space of a pretrained network, Zhang et al. 2018) correlate better with human judgments of image quality than PSNR, but they are still image metrics computed on rendered views. None of the three measures geometric accuracy at all. Evaluating geometry requires a different instrument, a depth or mesh error against ground-truth scans (Chamfer distance, normal consistency), which most view-synthesis benchmarks do not report. The practical rule is that a PSNR number is a claim about interpolating between your cameras, not a claim about the 3D structure, and the two come apart exactly when the cameras do not densely surround the object.

Implementation

The three code blocks below are the load-bearing computations of the page, the alpha-compositing quadrature (PyTorch and JAX side by side), positional encoding (PyTorch and JAX), and the geometry and hash-grid arithmetic (plain Python, the verification for Problems 4 and 5). Every number printed matches the worked solutions above.

Volume rendering compositing

This is the core of any NeRF renderer. Given per-sample densities, colors, and inter-sample distances, it produces the composited pixel, the accumulated opacity, and the expected depth. The subtlety is computing \( T_i = \prod_{j<i}(1 - \alpha_j) \) as an exclusive cumulative product, which both frameworks do by shifting a cumulative product so the first transmittance is one. This reproduces Problem 3 exactly.

import torch

def volume_render(sigma, color, delta):
    # sigma: (R, N) densities, color: (R, N, 3), delta: (R, N) sample spacings
    alpha = 1.0 - torch.exp(-sigma * delta)              # (R, N) per-sample opacity
    # exclusive cumulative product of (1 - alpha): T_i = prod_{j<i}(1 - alpha_j)
    one_minus = 1.0 - alpha + 1e-10                       # guard against log(0)
    T = torch.cumprod(one_minus, dim=-1)                 # inclusive product
    T = torch.cat([torch.ones_like(T[..., :1]), T[..., :-1]], dim=-1)  # shift -> exclusive
    w = T * alpha                                        # (R, N) compositing weights
    rgb = (w.unsqueeze(-1) * color).sum(dim=-2)          # (R, 3) rendered color
    acc = w.sum(dim=-1)                                  # (R,) accumulated opacity
    return rgb, acc, w

sigma = torch.tensor([[0.1, 2.0, 5.0, 1.0]])
color = torch.tensor([[0.2, 0.8, 0.5, 0.9]]).unsqueeze(-1).repeat(1, 1, 3)
delta = torch.full((1, 4), 0.5)
rgb, acc, w = volume_render(sigma, color, delta)
print(rgb[0, 0].item())   # 0.661566  (red channel, matches Problem 3)
print(acc[0].item())      # 0.982578

import jax, jax.numpy as jnp

def volume_render(sigma, color, delta):
    # sigma: (R, N), color: (R, N, 3), delta: (R, N)
    alpha = 1.0 - jnp.exp(-sigma * delta)                     # per-sample opacity
    one_minus = 1.0 - alpha + 1e-10
    T_incl = jnp.cumprod(one_minus, axis=-1)                  # inclusive product
    # exclusive: prepend 1, drop last -> T_i = prod_{j<i}(1 - alpha_j)
    T = jnp.concatenate([jnp.ones_like(T_incl[..., :1]), T_incl[..., :-1]], axis=-1)
    w = T * alpha                                             # compositing weights
    rgb = jnp.sum(w[..., None] * color, axis=-2)              # (R, 3)
    acc = jnp.sum(w, axis=-1)                                 # (R,)
    return rgb, acc, w

sigma = jnp.array([[0.1, 2.0, 5.0, 1.0]])
color = jnp.repeat(jnp.array([[0.2, 0.8, 0.5, 0.9]])[..., None], 3, axis=-1)
delta = jnp.full((1, 4), 0.5)
rgb, acc, w = volume_render(sigma, color, delta)
print(float(rgb[0, 0]))   # 0.661566
print(float(acc[0]))      # 0.982578

Positional encoding

The Fourier feature map \( \gamma \) with the geometric frequency schedule \( b_j = 2^j \). Both implementations concatenate the raw input and then, per band, the sine and cosine, matching the dimension count of Problem 2.

import torch

def positional_encoding(x, num_bands=10, include_input=True):
    # x: (..., D) coordinates in a normalized range
    freqs = 2.0 ** torch.arange(num_bands, device=x.device) * torch.pi  # (L,)
    xb = x[..., None] * freqs                              # (..., D, L)
    enc = torch.cat([torch.sin(xb), torch.cos(xb)], dim=-1)  # (..., D, 2L)
    enc = enc.flatten(start_dim=-2)                       # (..., D*2L)
    if include_input:
        enc = torch.cat([x, enc], dim=-1)                 # (..., D + D*2L)
    return enc

x = torch.rand(5, 3) * 2 - 1        # positions in [-1, 1]
gamma = positional_encoding(x, num_bands=10)
print(gamma.shape)                  # (5, 63)  = 3 + 3*2*10, matches Problem 2

import jax.numpy as jnp

def positional_encoding(x, num_bands=10, include_input=True):
    # x: (..., D) coordinates in a normalized range
    freqs = 2.0 ** jnp.arange(num_bands) * jnp.pi          # (L,)
    xb = x[..., None] * freqs                              # (..., D, L)
    enc = jnp.concatenate([jnp.sin(xb), jnp.cos(xb)], axis=-1)  # (..., D, 2L)
    enc = enc.reshape(*x.shape[:-1], -1)                   # (..., D*2L)
    if include_input:
        enc = jnp.concatenate([x, enc], axis=-1)           # (..., D + D*2L)
    return enc

import jax
x = jax.random.uniform(jax.random.PRNGKey(0), (5, 3)) * 2 - 1
gamma = positional_encoding(x, num_bands=10)
print(gamma.shape)                  # (5, 63)

Geometry and hash-grid arithmetic

The plain-Python verification for the hash-grid memory count (Problem 4) and the Gaussian projection (Problem 5). The 2x2 inverse and eigen-decomposition are small enough to be unaffected by the LAPACK issue noted in the authoring guide, and the inverse residual is checked to confirm it.

import numpy as np

# --- Instant-NGP hash-grid parameter count (Problem 4) ---
L, F, T, Nmin, Nmax = 16, 2, 2**19, 16, 2048
b = np.exp((np.log(Nmax) - np.log(Nmin)) / (L - 1))       # growth factor ~1.3819
params = 0
for l in range(L):
    Nl = int(np.floor(Nmin * b**l))
    dense = (Nl + 1) ** 3                                 # vertices of a resolution-Nl grid
    entries = min(T, dense)                               # dense until it exceeds T, then hashed
    params += entries * F
print(round(b, 4), params, round(params * 2 / 1e6, 1))    # 1.3819  12197850  24.4 (MB fp16)

# --- 3D Gaussian projection (Problem 5) ---
mu = np.array([0.4, -0.2, 2.0])                           # camera-space mean
s  = np.array([0.10, 0.05, 0.20])                         # scales
th = np.deg2rad(30.0)
R  = np.array([[np.cos(th), -np.sin(th), 0],
               [np.sin(th),  np.cos(th), 0],
               [0, 0, 1]])
M = R @ np.diag(s)
Sigma = M @ M.T                                           # 3x3 covariance
fx = fy = 1111.0
x, y, z = mu
J = np.array([[fx / z, 0, -fx * x / z**2],
              [0, fy / z, -fy * y / z**2]])               # projection Jacobian
Sigma2d = J @ Sigma @ J.T + 0.3 * np.eye(2)               # EWA + dilation
evals = np.linalg.eigvalsh(Sigma2d)                       # [1163.4, 3311.6] px^2
conic = np.linalg.inv(Sigma2d)
print(np.max(np.abs(Sigma2d @ conic - np.eye(2))))        # ~0  (inverse is healthy)
d = np.array([2.0, 1.0])
print(round(float(np.exp(-0.5 * d @ conic @ d)), 5))      # 0.99924  weight at (2,1) px

A small real fit, spectral bias on the H100

To make the spectral-bias argument concrete, two identical MLPs were trained to fit a 1D signal \( \sin(2\pi\cdot 8\, x) + \tfrac12 \sin(2\pi\cdot 32\, x) \) on 1024 points, one on the raw coordinate and one on an 8-band Fourier encoding, 3000 Adam steps each, on an NVIDIA H100 80GB. The plain-coordinate network plateaued at a mean squared error of roughly \( 1.8\times10^{-1} \), unable to fit the \( 32\times \) component, exactly the low-pass behavior the NTK predicts. The Fourier-feature network, given access to the high frequencies, reached about \( 8\times10^{-8} \), roughly six orders of magnitude lower. This is the entire empirical content of Tancik et al. in miniature. The architecture is identical, only the input encoding differs, and the difference is the difference between fitting sharp detail and not.

How it is done in practice

A production reconstruction pipeline in 2024-2025 looks less like a single paper and more like a toolchain. Poses come from COLMAP or a learned SfM front end. The representation is increasingly a Gaussian splat rather than a NeRF when real-time rendering or editing is required, and a hash-grid NeRF (Instant-NGP style, via nerfstudio's Nerfacto) when quality per parameter matters more than frame rate. The engineering gap between the derivation and a deployed system is mostly in three places, anti-aliasing, unbounded scenes, and the primitive count.

Anti-aliasing is where the clean quadrature above breaks down. A pixel is not a ray, it is a frustum, and integrating a high-frequency field along an infinitesimal ray aliases badly as the camera zooms or the resolution changes. Mip-NeRF (Barron et al., 2021, Google) replaces the point-sampled positional encoding with an integrated positional encoding that averages the Fourier features over the Gaussian approximation of the pixel's conical frustum, which is why it anti-aliases. High-frequency bands, whose integral over a wide frustum averages toward zero, are automatically attenuated at coarse scales. Mip-NeRF 360 (Barron et al., 2022) extends this to unbounded outdoor scenes with a scene-contraction warp that maps infinite space into a bounded domain and a proposal-network sampler that replaces the coarse MLP. For Gaussian splatting the analogue is Mip-Splatting and the various anti-aliased splatting variants, which add the 3D low-pass and 2D dilation filters that keep Gaussians from aliasing across scales.

The primitive-count and memory story is where measured hardware limits bite. A Gaussian splat of a room-scale scene is commonly one to five million Gaussians. At roughly 59 floats per Gaussian (position, rotation, scale, opacity, and third-order spherical harmonics for color) that is on the order of a few hundred megabytes, and the depth sort plus the tile-binning are the throughput bottleneck, both bandwidth-bound operations rather than compute-bound. The H100 measurements in this repository make the regime concrete. The card sustains roughly 3.0 TB/s of memory bandwidth (measured copy bandwidth 2992 GB/s from the H100 80GB benchmark), which is what a radix sort over millions of keys and a scatter over tile lists actually consume, while the closed-form Gaussian evaluation is cheap arithmetic that the same card can do at hundreds of TFLOP/s (bf16 matmul reaches 744 TFLOP/s at \( n = 4096 \) on this H100). The practical consequence is that Gaussian-splat rendering is limited by how fast you can sort and gather, not by the exponentials, which is why the tile scheme, sorting once per tile, is the load-bearing optimization rather than any arithmetic trick.

The current research frontier

The last two years have been a race along several axes, with different groups pushing different ones. On speed and quality of splatting, the INRIA original has been followed by anti-aliased variants (Mip-Splatting, Yu et al. 2024, from Max Planck and collaborators), 2D Gaussian splatting for accurate surface reconstruction (Huang et al. 2024, which flattens the Gaussians to oriented disks to recover clean meshes), and compression work driving the hundreds-of-megabytes cost down by an order of magnitude. On dynamic and 4D scenes, deformable and 4D Gaussian methods add a time axis, fitting a canonical set of Gaussians plus a deformation field to capture motion, a direct descendant of the dynamic-NeRF line (D-NeRF, Nerfies from Washington and Google).

On generative 3D, the field has moved quickly past pure SDS. The pole positions are held by feed-forward image-to-3D models. After LRM, a wave of large reconstruction models produce triplanes, meshes, or Gaussians in a single pass, and image-to-multiview-diffusion pipelines (Zero-1- to-3 from Columbia, MVDream from ByteDance, and successors) generate consistent multiple views that a reconstruction model fuses, which sidesteps the slow per-scene SDS loop while keeping the 2D prior. The score-distillation line itself continues through VSD (ProlificDreamer) and its faster variants. OpenAI's Point-E and Shap-E were early feed-forward point-cloud and implicit-function generators that established the direction. On foundation-model-scale reconstruction, DUSt3R and MASt3R (Naver Labs Europe, 2024) drop the SfM front end entirely, regressing dense pointmaps directly from image pairs with a transformer, which folds pose estimation and geometry into one learned model and is arguably the most significant recent change to the classical pipeline this page opened with. The through-line across all of it is that the differentiable image-formation model, whether ray marching, splatting, or direct pointmap regression, is now the common substrate, and the research contest is over the representation and the prior, not over whether to differentiate the renderer.

Open source to read

• nerfstudio-project/nerfstudio is the best starting point for the NeRF family, a modular framework with a clean renderer, many methods behind one interface, and a real-time viewer. Read nerfstudio/model_components/renderers.py for the alpha-compositing and field_components/encodings.py for the positional and hash encodings.
• NVlabs/instant-ngp is the reference multiresolution-hash-grid implementation. The CUDA kernel in src/testbed_nerf.cu and the encoding in the companion tiny-cuda-nn library show how the hash lookup and trilinear interpolation are fused for speed.
• graphdeco-inria/gaussian-splatting is the original 3DGS release. Start with the differentiable rasterizer submodule (diff-gaussian-rasterization). The forward kernel does the EWA projection and tile binning, and the backward kernel is where the gradients to covariance and opacity are derived.
• google/jaxnerf is a compact, readable JAX reimplementation of the original NeRF, good for seeing the coarse/fine hierarchical sampling and the rendering loop without the abstraction layers of a full framework.
• sxyu/svox2 is Plenoxels, a sparse voxel radiance field with no MLP, which makes the "the network was never the point" argument concrete in a few hundred lines of CUDA plus Python.
• threestudio-project/threestudio is the reference framework for generative 3D, with SDS, VSD, and the multiview-diffusion pipelines behind one interface. Read the guidance modules to see the SDS gradient (the Jacobian-dropping trick) implemented directly.
• facebookresearch/pytorch3d is Meta's differentiable 3D library, covering cameras, mesh and point rendering, and the geometry operators (Chamfer distance, marching cubes) that the geometry-evaluation section calls for.
• openai/shap-e (and its predecessor openai/point-e) are the early feed-forward 3D generators, useful for seeing how a diffusion model can be trained to emit implicit functions or point clouds directly rather than through per-scene optimization.

Common misconceptions

"NeRF is a rendering algorithm." NeRF is a scene representation (a coordinate network for color and density) fit by a rendering algorithm (volume-rendering quadrature). The rendering integral is classical, from participating-media transport. The novelty is representing the medium's coefficients with a network and fitting them by differentiating the render against photographs. Conflating the two obscures why the representation is interchangeable, Plenoxels swaps the network for a voxel grid and keeps the renderer.

"Positional encoding adds information the network did not have." The sinusoids are a fixed, deterministic function of the input coordinate. They add no information. What they change is the training dynamics, by reshaping the neural tangent kernel into a stationary kernel with controllable bandwidth, so that high-frequency components of the target are learned in a realistic number of steps instead of essentially never. The information was always there in the coordinate. The encoding makes it learnable.

"Gaussian splatting is faster because it avoids the volume rendering integral." It uses the same alpha-compositing sum as NeRF, and the front-to-back \( \sum \mathbf{c}_i \alpha_i \prod(1 - \alpha_j) \) is identical. What it avoids is the per-sample MLP query and the ray marching. \( \alpha_i \) comes from a closed-form 2D Gaussian rather than a network, and the primitives are rasterized (sorted once per tile) instead of sampled per pixel. The transport model is the same. The evaluation is what changes.

"The eikonal loss makes the SDF more accurate." The eikonal penalty does not improve the fit to the zero-level-set data. A network can fit the surface without it. What it enforces is that the field is a metric distance function, unit-gradient everywhere, which is what makes the gradient a valid surface normal and the magnitude a valid distance for sphere tracing. Without it the same surface is represented by a non-metric field whose gradient is not a usable normal.

"High PSNR means the geometry is correct." PSNR measures pixel error on the test views only, and the alpha-compositing map is many-to-one. Distinct density fields, including ones with floaters and smeared surfaces, can render nearly identically from the observed viewpoints. High PSNR certifies interpolation between your cameras, not geometric correctness. Verifying geometry requires a depth or mesh error against ground truth, which image metrics never provide.

"SDS optimizes the diffusion training loss on the renders." It does not. The true gradient of that loss contains the U-Net's Jacobian \( \partial \epsilon_\phi / \partial x_t \). SDS deliberately drops it, leaving a gradient that treats the denoiser as a fixed score field. This is why SDS needs only a forward pass of the diffusion model, and why its mode-seeking behavior (over- saturation, low diversity) differs from what minimizing the actual loss would give, the gap ProlificDreamer's VSD closes.

"Instant-NGP is fast because the hash avoids collisions." The opposite is true. At fine levels the hash table is far smaller than the number of grid vertices, so collisions are frequent and deliberately unresolved. Speed comes from replacing a deep MLP over a large encoding with a handful of cache-friendly table lookups and a tiny MLP. Correctness survives the collisions because the surface occupies a small fraction of the fine grid and gradient descent, aided by the coarser levels, disambiguates the few that matter.

Self-check

References

1. R. Hartley, A. Zisserman. Multiple View Geometry in Computer Vision, 2nd ed. Cambridge University Press, 2004. The standard reference for the projective camera, the fundamental and essential matrices, triangulation, and bundle adjustment.
2. B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, R. Ng. NeRF: representing scenes as neural radiance fields for view synthesis. ECCV, 2020. arXiv:2003.08934.
3. M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. T. Barron, R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS, 2020. arXiv:2006.10739.
4. J. J. Park, P. Florence, J. Straub, R. Newcombe, S. Lovegrove. DeepSDF: learning continuous signed distance functions for shape representation. CVPR, 2019. arXiv:1901.05103.
5. L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, A. Geiger. Occupancy networks: learning 3D reconstruction in function space. CVPR, 2019. arXiv:1812.03828.
6. A. Gropp, L. Yariv, N. Haim, M. Atzmon, Y. Lipman. Implicit geometric regularization for learning shapes. ICML, 2020. arXiv:2002.10099. Introduces the eikonal regularizer for neural SDFs.
7. T. Müller, A. Evans, C. Schied, A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (SIGGRAPH), 2022. arXiv:2201.05989.
8. S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, A. Kanazawa. Plenoxels: radiance fields without neural networks. CVPR, 2022. arXiv:2112.05131.
9. C. Sun, M. Sun, H.-T. Chen. Direct voxel grid optimization: super-fast convergence for radiance fields reconstruction. CVPR, 2022. arXiv:2111.11215.
10. B. Kerbl, G. Kopanas, T. Leimkühler, G. Drettakis. 3D Gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (SIGGRAPH), 2023. arXiv:2308.04079.
11. M. Zwicker, H. Pfister, J. van Baar, M. Gross. EWA volume splatting. IEEE Visualization, 2001. The elliptical weighted average framework underlying the projection of Gaussians to screen space.
12. C. R. Qi, H. Su, K. Mo, L. J. Guibas. PointNet: deep learning on point sets for 3D classification and segmentation. CVPR, 2017. arXiv:1612.00593.
13. C. R. Qi, L. Yi, H. Su, L. J. Guibas. PointNet++: deep hierarchical feature learning on point sets in a metric space. NeurIPS, 2017. arXiv:1706.02413.
14. B. Poole, A. Jain, J. T. Barron, B. Mildenhall. DreamFusion: text-to-3D using 2D diffusion. ICLR, 2023. arXiv:2209.14988. Introduces score distillation sampling.
15. Z. Wang, C. Lu, Y. Wang, F. Bao, C. Li, H. Su, J. Zhu. ProlificDreamer: high-fidelity and diverse text-to-3D generation with variational score distillation. NeurIPS, 2023. arXiv:2305.16213.
16. Y. Hong, K. Zhang, J. Gu, S. Bi, Y. Zhou, D. Liu, F. Liu, K. Sunkavalli, T. Bui, H. Tan. LRM: large reconstruction model for single image to 3D. ICLR, 2024. arXiv:2311.04400.
17. J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, P. P. Srinivasan. Mip-NeRF: a multiscale representation for anti-aliasing neural radiance fields. ICCV, 2021. arXiv:2103.13415.
18. J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, P. Hedman. Mip-NeRF 360: unbounded anti-aliased neural radiance fields. CVPR, 2022. arXiv:2111.12077.
19. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, O. Wang. The unreasonable effectiveness of deep features as a perceptual metric (LPIPS). CVPR, 2018. arXiv:1801.03924.
20. S. Wang, V. Leroy, Y. Cabon, B. Chidlovskii, J. Revaud. DUSt3R: geometric 3D vision made easy. CVPR, 2024. arXiv:2312.14132. Dense pointmap regression that folds pose and geometry into one learned model.
21. Z. Yu, A. Chen, B. Huang, T. Sattler, A. Geiger. Mip-Splatting: alias-free 3D Gaussian splatting. CVPR, 2024. arXiv:2311.16493.
22. L. Yariv, J. Gu, Y. Kasten, Y. Lipman. Volume rendering of neural implicit surfaces (VolSDF). NeurIPS, 2021. arXiv:2106.12052.
23. P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, W. Wang. NeuS: learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS, 2021. arXiv:2106.10689.
24. M. Tancik, E. Weber, E. Ng, R. Li, B. Yi, J. Kerr, T. Wang, A. Kristoffersen, J. Austin, K. Salahi, A. Ahuja, D. McAllister, A. Kanazawa. Nerfstudio: a modular framework for neural radiance field development. SIGGRAPH, 2023. arXiv:2302.04264.