Prepare for University Studies & Career Advancement

3D Vision and Gaussian Splatting

3D computer vision extends classical 2D planar image analysis into three-dimensional geometric space. While traditional image processing handles flat pixel arrays, 3D vision algorithms reconstruct, represent, and render volumetric geometry, spatial depth, camera motion, and view-dependent light transport.
Recent breakthroughs in implicit neural representations—most notably Neural Radiance Fields (NeRFs)—and explicit differentiable rendering techniques like 3D Gaussian Splatting (3DGS) have revolutionized spatial computing. These architectures enable real-time photorealistic novel view synthesis, allowing virtual cameras to navigate complex physical scenes from sparse collections of 2D photographs.
3D Vision and Differentiable Rendering Pipeline: From Multi-View Camera Pose Estimation and NeRF Ray Marching to Real-Time 3D Gaussian Splatting Tile Rasterization
From autonomous vehicle spatial perception and robotics motion planning to immersive augmented reality (AR/VR) digital twins, 3D computer vision bridges the gap between discrete camera sensors and continuous physical environments. Understanding the mathematical transitions from classical epipolar geometry to implicit neural functions and explicit Gaussian primitives is essential for modern spatial computing engineers.

Foundations of 3D Computer Vision

Pinhole Camera Model and Projection Geometry

The basic geometric model relating a 3D point in world coordinates Pw = [X, Y, Z]T to its 2D pixel projection p = [u, v]T on an image plane is the pinhole camera model.
Using homogeneous coordinates, perspective projection is modeled as:
$$ s \begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = \mathbf{K} \begin{bmatrix} \mathbf{R} & \mathbf{t} \end{bmatrix} \begin{bmatrix} X \\ Y \\ Z \\ 1 \end{bmatrix} $$
where:
  • s: non-zero projective scaling factor
  • K: 3×3 intrinsic calibration matrix containing focal lengths (fx, fy) and principal point offset (cx, cy)
  • R ∈ SO(3), t ∈ &mathbb;R3: extrinsic rotation matrix and translation vector transforming world coordinates to camera coordinates
$$ \mathbf{K} = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix} $$

Epipolar Geometry and Depth Estimation

When a 3D point P is observed by two cameras at different spatial positions, the point and the two camera centers form an epipolar plane. The intersection of this plane with image planes defines epipolar lines.
For rectified stereo camera pairs with baseline distance b and focal length f, spatial depth Z is inversely proportional to pixel disparity d = (uL – uR):
$$ Z = \frac{f \cdot b}{d} $$

Traditional 3D Geometric Representations

Classical 3D computer vision algorithms utilize discrete explicit structures to store spatial data:
  • Depth Maps: 2D image matrices where each pixel value represents scalar radial or z-distance to the surface.
  • Point Clouds: Unordered sets of 3D coordinate vectors {pi = (xi, yi, zi)} often generated via LiDAR scanning or Structure-from-Motion (SfM).
  • Voxel Grids: Regular 3D spatial grids partitioning volumes into discrete occupancy cells (voxels). Memory footprint scales cubically (O(N3)).
  • Polygon Meshes: Connected structures of vertices, edges, and planar triangular faces that approximate surface shells efficiently.

Interactive Quick Review: Camera Projection & Epipolar Depth

Test your understanding of pinhole camera projection and stereo depth triangulation before exploring neural representations:

Quick Review Verification

1. How does pixel disparity d relate mathematically to physical surface depth Z in a rectified stereo system?
Answer: Surface depth Z is inversely proportional to disparity d according to Z = (f ċ b) / d, where f is focal length and b is baseline distance.
2. What is the key structural drawback of voxel grids for high-resolution 3D scene representation?
Answer: Voxel grids suffer from cubic memory scaling (O(N3)), making high-resolution representation computationally prohibitive compared to point clouds or surface meshes.

Neural Implicit Representations: NeRFs

Introduced in 2020, Neural Radiance Fields (NeRF) replaced discrete 3D geometric meshes with continuous implicit functions parameterized by Multi-Layer Perceptrons (MLPs).

The Continuous Radiance Field Function

A NeRF represents a static 3D scene as a continuous vector function FΘ mapping a 5D coordinate input to a 4D output consisting of emitted color and volume density:
$$ F_{\Theta}: (\mathbf{x}, \mathbf{d}) \to (\mathbf{c}, \sigma) $$
where:
  • x = (x, y, z): 3D spatial position coordinates
  • d = (θ, φ): 2D viewing direction unit vector
  • c = (r, g, b): view-dependent emitted color
  • σ: view-invariant volume density (differential opacity)

Volumetric Ray Marching and Differentiable Rendering

To evaluate the color C(r) of a single camera ray r(t) = o + td passing through a scene from camera origin o between bounds tn (near) and tf (far), NeRF integrates volumetric light transport:
$$ C(\mathbf{r}) = \int_{t_n}^{t_f} T(t) \cdot \sigma(\mathbf{r}(t)) \cdot \mathbf{c}(\mathbf{r}(t), \mathbf{d}) \, dt $$
where T(t) represents accumulated transmittance (probability that the ray travels from tn to t without hitting an occluding particle):
$$ T(t) = \exp\left( -\int_{t_n}^{t} \sigma(\mathbf{r}(s)) \, ds \right) $$
In numerical implementation, this continuous integral is approximated using discrete quadrature along N sampled points ti along the ray:
$$ \hat{C}(\mathbf{r}) = \sum_{i=1}^{N} T_i \cdot \alpha_i \cdot \mathbf{c}_i, \quad \text{where } T_i = \prod_{j=1}^{i-1} (1 – \alpha_j) \quad \text{and } \alpha_i = 1 – \exp(-\sigma_i \delta_i) $$
δi = ti+1 – ti represents distance between adjacent sample points.

Positional Encoding

Standard MLPs exhibit a spectral bias toward learning low-frequency functions. To capture high-frequency visual details (sharp edges, textures), raw input coordinates are mapped into a higher-dimensional frequency space using Fourier positional encoding γ(p):
$$ \gamma(p) = \left( \sin(2^0 \pi p), \cos(2^0 \pi p), \sin(2^1 \pi p), \cos(2^1 \pi p), \dots, \sin(2^{L-1} \pi p), \cos(2^{L-1} \pi p) \right) $$
for each scalar coordinate in x and d.

Explicit Modern 3D Rendering: 3D Gaussian Splatting (3DGS)

While NeRFs achieve high reconstruction quality, querying an MLP millions of times per frame makes real-time rendering difficult. Introduced in 2023, 3D Gaussian Splatting (3DGS) overcomes this by combining explicit point-based primitives with fast, tile-based differentiable rasterization.

The 3D Gaussian Primitive

Instead of an implicit neural network, a scene is represented explicitly by millions of parameterized anisotropic 3D Gaussians. Each Gaussian primitive G(x) is defined by:
$$ G(\mathbf{x}) = \exp\left( -\frac{1}{2} (\mathbf{x} – \boldsymbol{\mu})^T \boldsymbol{\Sigma}^{-1} (\mathbf{x} – \boldsymbol{\mu}) \right) $$
A single 3D Gaussian primitive carries five trainable parameters:
  1. Center Position: Mean vector μ = [x, y, z]T ∈ &mathbb;R3
  2. 3D Covariance Matrix: Σ ∈ &mathbb;R3×3 defining physical shape, scale, and spatial orientation
  3. Opacity: Scalar transparency α ∈ [0, 1]
  4. Color / View-Dependence: Spherical Harmonics (SH) coefficients parameterizing view-dependent color c(d)

Covariance Factorization for Optimization

A valid covariance matrix Σ must remain positive semi-definite during gradient descent optimization. To enforce this, Σ is factorized into a 3D rotation matrix R (represented by a unit quaternion q) and a diagonal scaling matrix S (vector s = [sx, sy, sz]):
$$ \boldsymbol{\Sigma} = \mathbf{R} \mathbf{S} \mathbf{S}^T \mathbf{R}^T $$

2D Projection and Tile-Based Rasterization

To render a 3D Gaussian onto a 2D camera image plane, the 3D spatial covariance Σ is projected into a 2D image-space covariance Σ’ using viewing transformation W and Jacobian matrix J of the projective transformation:
$$ \boldsymbol{\Sigma}’ = \mathbf{J} \mathbf{W} \boldsymbol{\Sigma} \mathbf{W}^T \mathbf{J}^T $$
Rendering proceeds via high-speed tile-based rasterization:
  1. Tile Sorting: The screen is divided into 16×16 pixel tiles. Gaussians overlapping each tile are identified and sorted by depth (z-distance) using GPU Radix Sort.
  2. Alpha Blending: For each pixel, color is accumulated front-to-back using ordered N overlapping Gaussians:
$$ C_{\text{pixel}} = \sum_{i \in N} \mathbf{c}_i \alpha_i’ \prod_{j=1}^{i-1} (1 – \alpha_j’) $$
where α’i is evaluated by multiplying opacity αi by the projected 2D Gaussian density value at that pixel coordinate.

Adaptive Density Control

During optimization, the total count of 3D Gaussians is dynamically adapted:
  • Cloning: Small Gaussians in under-reconstructed high-variance regions are duplicated.
  • Splitting: Large Gaussians covering broad areas are split into two smaller Gaussians.
  • Pruning: Gaussians with opacity α dropping below a threshold are deleted to optimize memory footprint.

Architectural Comparison Matrix

The following table summarizes key technical trade-offs across 3D representations:
Feature / MetricClassical Point Clouds / MeshesNeural Radiance Fields (NeRF)3D Gaussian Splatting (3DGS)
Scene RepresentationExplicit surface geometryImplicit neural MLP functionExplicit anisotropic 3D primitives
Rendering SpeedReal-time (>100 FPS)Slow (0.1–15 FPS)Real-time (>100+ FPS)
Optimization / Training TimeFast (SfM: minutes)Slow (hours to days)Fast (10–30 minutes)
View-Dependent EffectsPoor (requires textures)Excellent (view directional MLP)Excellent (Spherical Harmonics)
Memory FootprintLow (10–50 MB)Compact weights (5–15 MB)Large primitives (0.5–2.0 GB)

Spatial Computing & Real-World Applications

3D vision algorithms power several expanding technological sectors:
  • Photorealistic Scene Digital Twins: High-fidelity capture of physical architectural spaces, cultural heritage sites, and real estate for virtual inspection.
  • Spatial Computing & AR/VR Headsets: Real-time novel view synthesis enables low-latency visual placement of virtual objects into physical environments with accurate perspective and lighting.
  • Autonomous Vehicle Perception: 3D occupancy networks predict 3D geometric boundaries and velocity vectors across road scenes from multi-camera arrays, replacing costly physical sensors.
  • Robotics & Sim-to-Real Manipulation: Generating realistic volumetric assets allows robots to train interaction policies inside synthetic 3D physics environments before physical deployment.

History, Challenges, and Future of 3D Vision

History and Evolution

3D vision evolved from multi-view geometry and photogrammetry in the 1980s and 1990s. Early landmark milestones included Structure-from-Motion (SfM) algorithms like COLMAP and Multi-View Stereo (MVS) surface reconstruction. The field underwent a major shift with NeRF in 2020, establishing neural implicit representations. In 2023, 3D Gaussian Splatting combined explicit primitives with tile-based rasterization, achieving real-time rendering frame rates.

Current Technical & Operational Challenges

Deploying modern 3D vision systems presents technical challenges:
  • Dynamic Motion and Transient Objects: Standard NeRFs and 3DGS assume static scenes. Uncalibrated moving vehicles or people create temporal blur artifacts.
  • Unbounded Outdoor Environments: Distant background skies and horizon geometry require large spatial bounding volumes, straining Gaussian primitive memory constraints.
  • Storage Size Overhead: Storing millions of 3D Gaussians per scene generates file sizes exceeding hundreds of megabytes, requiring vector quantization compression for streaming over networks.
  • Specular Reflection and Transparent Surfaces: Refractive objects (glass, water, polished chrome) violate multi-view color consistency assumptions, causing floating surface artifacts.

Future Potentials & Emerging Paradigms

Future developments focus on Generative 3D Foundation Models. Combining diffusion models with 3D Gaussian representations enables text-to-3D and image-to-3D asset generation in seconds. Furthermore, dynamic 4D representations incorporating temporal deformation vectors are enabling real-time streaming of volumetric video.

Engineering Trade-offs

Engineers must balance several design choices:
  • Rendering Latency (FPS) vs Storage Memory Size: 3DGS achieves 120+ FPS by storing millions of explicit primitives, whereas NeRF conserves storage memory at the expense of slow neural ray-marching compute times.
  • Implicit Continuity vs Explicit Geometry Editability: Implicit MLP weights cannot be directly sculpted or manipulated in standard CAD tools, whereas explicit 3D Gaussians and meshes can be individually transformed, cropped, and animated.
  • Training Time vs View Synthesis Quality: Fast 3DGS optimization completes in 15 minutes on a single GPU, whereas high-quality continuous NeRF variants require hours of iterative optimization.

Interactive Quick Review: 3DGS Parameters & Render Trade-offs

Reinforce key concepts regarding 3D Gaussian parameters, covariance matrix factorization, and rendering trade-offs before reviewing the FAQ:

Quick Review Verification

1. Why is a 3D Gaussian’s covariance matrix Σ factorized into rotation R and scale S (Σ = R S ST RT) during optimization?
Answer: Direct optimization of Σ can yield invalid non-positive semi-definite matrices. Factorizing into unit quaternion rotation R and diagonal scale S guarantees that Σ remains mathematically valid throughout gradient descent.
2. What spherical harmonic (SH) coefficients represent in a 3D Gaussian primitive?
Answer: Spherical Harmonics parameterize view-dependent color variations across different viewing direction angles, allowing primitives to model specular highlights and reflection effects.

External References and Further Reading

To explore multi-view geometry, neural radiance fields, and differentiable rendering, consult the following authoritative sources:

Frequently Asked Questions about 3D Vision and Gaussian Splatting

What is the core difference between NeRF and 3D Gaussian Splatting?

NeRF represents scenes implicitly inside a continuous neural network (MLP) and queries scene points using slow ray marching. 3D Gaussian Splatting represents scenes explicitly using millions of parameterized 3D Gaussian point primitives that are rasterized onto image planes at high speed using GPU tile sorting.

Why is Structure-from-Motion (SfM) necessary before training NeRF or 3DGS models?

Both NeRF and 3DGS require precise intrinsic camera calibration parameters (K) and extrinsic camera poses (R, t) for every training photograph. SfM algorithms (such as COLMAP) analyze sparse feature correspondences across images to calculate camera poses and construct initial sparse point clouds.

How do Spherical Harmonics (SH) enable view-dependent color rendering in 3DGS?

Spherical Harmonics are orthogonal basis functions defined on the surface of a sphere. By storing SH coefficients inside each 3D Gaussian, the primitive’s evaluated output color varies smoothly as a function of the camera viewing angle, capturing specular highlights and glossy reflections.

What causes floating artifact “floaters” in NeRF and 3D Gaussian Splatting reconstructions?

Floaters occur when optimization algorithms place high-density particles or semi-transparent Gaussians in empty space close to training camera origins to satisfy color matching constraints from limited viewing angles. Adding density regularizers or multi-view consistency constraints helps eliminate floaters.

How does 3D Gaussian Splatting achieve real-time rendering speeds over 100 FPS?

3DGS replaces slow per-pixel neural network evaluations with a hardware-accelerated tile rasterizer. Screen pixels are partitioned into 16×16 tiles, Gaussians are sorted by depth using fast GPU Radix Sort, and alpha blending accumulates colors in parallel across GPU threads.

Can 3D Gaussian Splatting models be exported directly into standard 3D engines like Unreal Engine or Unity?

Standard game engines traditionally render triangular polygon meshes rather than Gaussian point clouds. However, custom WebGL, Unity, and Unreal plugins using specialized compute shaders now enable native real-time 3DGS rasterization inside interactive game viewports.

What is the memory size of a typical 3D Gaussian Splatting scene file?

A uncompressed 3DGS scene containing 1 to 3 million Gaussians occupies between 500 MB and 1.5 GB of storage space, as each primitive stores coordinates, scale, quaternion rotation, opacity, and multiple Spherical Harmonic color coefficients. Vector quantization compression can reduce file sizes by 70% to 90%.

How does positional encoding prevent blurring in NeRF scene reconstructions?

Standard neural network layers favor learning low-frequency smooth functions. Positional encoding projects 3D spatial coordinates into high-frequency sinusoidal Fourier dimensions (\(\sin(2^k \pi p)\)), forcing the network to represent sharp high-frequency spatial details such as sharp corners and surface textures.

End-of-Page Exercises: Review, Thought-Provoking & Engineering Problems

Review Questions and Answers

1. Write the 3×3 intrinsic camera calibration matrix K.
Answer: K = [[fx, 0, cx], [0, fy, cy], [0, 0, 1]].
2. What 5D input vector parameters are fed into a standard NeRF MLP?
Answer: 3D spatial position coordinates (x, y, z) and 2D viewing direction unit vector (θ, φ).
3. What 4D output values are produced by a NeRF radiance field evaluation?
Answer: Emitted RGB color tuple c = (r, g, b) and volume density scalar σ.
4. List the five trainable parameter groups carried by a single 3D Gaussian primitive.
Answer: Center mean position μ, 3D covariance matrix Σ (quaternion rotation q and scale s), opacity α, and Spherical Harmonics (SH) color coefficients.
5. How is a 3D covariance matrix Σ factorized to guarantee positive semi-definiteness?
Answer: Σ = R S ST RT, where R is a rotation matrix derived from a unit quaternion and S is a diagonal scaling matrix.
6. What tile size is used in standard 3D Gaussian Splatting GPU rasterization pipelines?
Answer: 16×16 pixel screen tiles.
7. What three adaptive density operations modify Gaussian primitive counts during 3DGS training?
Answer: Cloning (duplicating small primitives), Splitting (dividing large primitives), and Pruning (deleting low-opacity primitives).
8. What open-source Structure-from-Motion tool is widely used to calibrate camera poses for 3DGS?
Answer: COLMAP.
9. What formula projects 3D spatial covariance Σ into 2D image-space covariance Σ’?
Answer: Σ’ = J W Σ WT JT, where W is the viewing transform and J is the projection Jacobian.
10. Name four primary real-world application domains for 3D vision and differentiable rendering.
Answer: Architectural digital twins, AR/VR spatial computing, autonomous vehicle perception, and robotics simulation asset creation.

Thought-Provoking Questions and Answers

1. Why does continuous volume rendering in NeRF fail when scenes contain reflective mirrors or shiny metallic surfaces?
Answer: Volume rendering assumes that emitted color c and density σ originate from static 3D spatial points. On reflective surfaces, virtual light appears to originate from behind the mirror surface depending on camera position. This breaks multi-view consistency, causing NeRF optimization to place incorrect semi-transparent density structures behind the reflective plane.
2. How does the choice of positional encoding frequency bounds L affect detail sharpness versus aliasing in NeRF outputs?
Answer: Higher maximum frequency bounds L allow the MLP to represent sharp high-frequency surface details. However, setting L too high causes the model to fit high-frequency image noise and creates high-frequency ringing artifacts (aliasing) when rendering novel views from unobserved camera angles.
3. Why is tile-based sorting using GPU Radix Sort critical for achieving real-time 100+ FPS rendering in 3D Gaussian Splatting?
Answer: Alpha blending requires evaluating Gaussians in exact front-to-back depth order for every pixel. Global sorting of millions of primitives on a per-pixel basis is computationally intractable. Partitioning the screen into 16×16 tiles and executing GPU Radix Sort in parallel limits depth sorting to local tile keys, enabling microsecond alpha accumulation across GPU threads.
4. How do explicit 3D Gaussians allow intuitive scene editing compared to implicit neural network weights?
Answer: Implicit NeRF weights store spatial geometry globally across MLP weight matrices; modifying a weight alters geometry unpredictably across the whole scene. 3D Gaussians are localized explicit primitives. Engineers can select a 3D spatial subset of Gaussians, translate, scale, or delete them directly, or blend two distinct Gaussian scenes together without retraining network weights.
5. Why do unconstrained outdoor captures create spatial divergence issues for standard 3D Gaussian Splatting optimization?
Answer: Outdoor captures contain background sky and distant mountains located effectively at infinite depth. Standard 3DGS attempts to model distant backgrounds by expanding large Gaussians near camera boundaries. These huge Gaussians cause rendering artifacts when the camera moves, requiring background sky dome models or depth-regularized primitive placement.
6. In what way does vector quantization reduce the multi-gigabyte storage footprint of 3D Gaussian Splatting scenes?
Answer: Storing raw 32-bit floating-point numbers for scale, rotation, color, and opacity across 3 million Gaussians generates large file sizes (~1 GB). Vector quantization clusters similar Gaussian parameters into shared codebook dictionaries, storing compact 8-bit dictionary indices per primitive instead of raw floats. This reduces scene file sizes by 80% to 90% with negligible loss in rendering fidelity.
7. Why is gradient computation with respect to camera extrinsics important in joint bundle-adjusting NeRF frameworks?
Answer: Initial camera poses estimated via SfM often contain calibration errors. In joint optimization frameworks (e.g., BARF), backpropagating photometric rendering losses directly to camera pose parameters (R, t) refines camera poses alongside scene geometry, fixing alignment errors and removing double-vision artifacts.
8. How does dynamic 4D Gaussian Splatting extend 3D representations to capture deformable moving objects?
Answer: 4D Gaussian Splatting incorporates a time dimension t. Each primitive center position μ(t) and rotation R(t) are modulated by a learned spatial deformation network or polynomial trajectory function over time. This enables the model to render deforming dynamic humans and moving vehicles while preserving real-time rasterization speeds.
9. What causes shadow inconsistency when placing synthetic virtual objects into a NeRF digital twin scene?
Answer: Standard NeRFs bake lighting directly into emitted colors c without decomposing scene illumination from surface material BRDFs. Inserting a synthetic virtual object does not cast realistic dynamic shadows onto surrounding NeRF surfaces unless the neural field explicitly factorizes environment lighting, albedo, and surface normals (Relightable NeRFs).
10. How do 3D occupancy networks assist autonomous vehicles when navigating unclassified road debris?
Answer: Traditional object detectors classify specific trained bounding-box categories (cars, pedestrians, cyclists) and fail on unusual obstacles like fallen cargo or tree branches. 3D occupancy networks predict physical volumetric occupancy state (occupied vs free space) across a 3D grid regardless of object class, enabling autonomous vehicles to avoid unclassified obstacles safely.

Numerical Engineering Problems & Solutions

Problem 1: Stereo Triangulation Depth Calculation
A rectified stereo vision system has baseline distance b = 0.20 meters and camera focal length f = 1200 pixels. An obstacle point exhibits left image coordinate uL = 850 pixels and right image coordinate uR = 610 pixels. Calculate disparity d and evaluate spatial depth Z in meters.
Solution:
  1. Disparity calculation: d = 850 – 610 = 240 pixels.
  2. Depth calculation: Z = (f ċ b) / d = (1200 × 0.20) / 240 = 240 / 240 = 1.00 meter.
Answer: Disparity is 240 pixels and obstacle depth is 1.00 meter.
Problem 2: Pinhole Perspective Projection Mapping
A camera with focal length fx = 800 pixels, fy = 800 pixels, and principal point (cx, cy) = (640, 480) observes a 3D point with camera coordinates P = [0.50, -0.25, 2.00]T meters. Calculate projected 2D image plane coordinates u and v in pixels using perspective projection formulas u = fx (X / Z) + cx and v = fy (Y / Z) + cy.
Solution:
  1. Normalized image coordinates: x’ = 0.50 / 2.00 = 0.25; y’ = -0.25 / 2.00 = -0.125.
  2. Calculate u: u = (800 × 0.25) + 640 = 200 + 640 = 840 pixels.
  3. Calculate v: v = (800 × -0.125) + 480 = -100 + 480 = 380 pixels.
Answer: Projected pixel coordinates are u = 840 pixels, v = 380 pixels.
Problem 3: Discrete NeRF Ray Marching Transmittance Evaluation
Along a ray r(t), two discrete sample points have evaluated opacity values α1 = 0.40 and α2 = 0.50 with sample colors c1 = [1.0, 0.0, 0.0] and c2 = [0.0, 1.0, 0.0]. Calculate accumulated transmittance values T1 and T2, evaluate rendered color weights w1 = T1 α1 and w2 = T2 α2, and compute total accumulated RGB color Ĉ(r) = w1 c1 + w2 c2.
Solution:
  1. Transmittance at first point: T1 = 1.0. Weight w1 = 1.0 × 0.40 = 0.40.
  2. Transmittance at second point: T2 = (1 – α1) = 1.0 – 0.40 = 0.60. Weight w2 = 0.60 × 0.50 = 0.30.
  3. Accumulated RGB color: Ĉ = 0.40 × [1, 0, 0] + 0.30 × [0, 1, 0] = [0.40, 0.30, 0.00].
Answer: Rendered ray color tuple is [0.40, 0.30, 0.00] (Red = 0.40, Green = 0.30, Blue = 0.00).
Problem 4: Positional Encoding Output Dimension Expansion
A scalar spatial coordinate x is transformed using Fourier positional encoding γ(x) with L = 10 frequency bands. Calculate the total scalar dimension count of encoded vector γ(x). For a 3D coordinate point x = (x, y, z), calculate total output dimensionality after applying positional encoding across all three axes.
Solution:
  1. Single scalar dimension: Each frequency band produces 1 sine and 1 cosine term (2 values per band). For L = 10, total dimensions = 2 × 10 = 20 scalars.
  2. 3D coordinate point (x, y, z): 3 coordinates × 20 scalars/coordinate = 60 dimensions. (If original raw coordinates are appended, dimension count equals 63).
Answer: Single coordinate expands to 20 dimensions; 3D coordinate point expands to 60 encoded dimensions.
Problem 5: 3D Gaussian Covariance Matrix Construction
An unrotated anisotropic 3D Gaussian primitive has diagonal spatial scaling parameters s = [sx, sy, sz] = [0.10, 0.20, 0.50] meters. Assuming rotation matrix R equals the 3×3 identity matrix I3, calculate diagonal scaling matrix S and evaluate elements of 3D spatial covariance matrix Σ = S ST.
Solution:
  1. Diagonal scale matrix S = diag(0.10, 0.20, 0.50).
  2. Covariance matrix Σ = S ST = diag((0.10)2, (0.20)2, (0.50)2).
  3. Evaluate variances: σx2 = 0.010, σy2 = 0.040, σz2 = 0.250 m2. Off-diagonal covariances are zero.
Answer: Diagonal elements of covariance matrix Σ are σx2 = 0.010, σy2 = 0.040, σz2 = 0.250.
Problem 6: Memory Footprint Sizing for 3D Gaussian Splatting Scenes
A high-fidelity 3D Gaussian Splatting scene contains N = 2,500,000 primitives. Each Gaussian primitive stores: 3D position (3 floats), scale vector (3 floats), unit quaternion rotation (4 floats), opacity (1 float), and Spherical Harmonics color coefficients (48 floats). Assuming each float is stored as a 32-bit (4-byte) single-precision number, calculate memory footprint per primitive in bytes and compute uncompressed scene storage footprint in Megabytes (MB). (1 MB = 106 bytes).
Solution:
  1. Total float count per primitive: 3 + 3 + 4 + 1 + 48 = 59 floats.
  2. Byte size per primitive: 59 floats × 4 bytes/float = 236 bytes.
  3. Total scene footprint: 2,500,000 primitives × 236 bytes = 590,000,000 bytes = 590.0 MB.
Answer: Each primitive occupies 236 bytes, yielding an uncompressed scene storage size of 590.0 MB.
Problem 7: Tile-Based GPU Sorting Load Calculation
A 3D Gaussian Splatting rasterizer processes a 1920 × 1080 screen frame (2,073,600 pixels) partitioned into non-overlapping 16×16 pixel tiles. Calculate total screen tile count. If an average of 1,200 Gaussians overlap each tile, calculate total tile-primitive instance keys generated for GPU Radix Sorting.
Solution:
  1. Tiles horizontally: 1920 / 16 = 120 tiles.
  2. Tiles vertically: 1080 / 16 = 67.5 → 68 tiles (padded).
  3. Total screen tile count: 120 × 68 = 8,160 tiles.
  4. Total tile-primitive keys: 8,160 tiles × 1,200 Gaussians/tile = 9,792,000 keys (~9.79 million keys).
Answer: Frame consists of 8,160 tiles, generating 9.79 million tile-primitive keys for sorting.
Problem 8: Alpha Blending Transmittance Cutoff Stopping Criterion
During tile rasterization, pixel color is accumulated using front-to-back alpha blending. Rendering stops early when accumulated transmittance drops below threshold Tstop = 0.001 (99.9% opacity reached). If three ordered Gaussians have individual opacities α1 = 0.80, α2 = 0.90, and α3 = 0.95, evaluate accumulated transmittance after each primitive and determine if early stopping is triggered after Primitive 3.
Solution:
  1. Initial transmittance: T0 = 1.0.
  2. After Primitive 1: T1 = T0 (1 – 0.80) = 1.0 × 0.20 = 0.20.
  3. After Primitive 2: T2 = T1 (1 – 0.90) = 0.20 × 0.10 = 0.020.
  4. After Primitive 3: T3 = T2 (1 – 0.95) = 0.020 × 0.05 = 0.0010.
  5. Check stopping criterion: T3 = 0.0010 ≤ 0.001 (Stopping criterion satisfied).
Answer: Accumulated transmittance drops to 0.0010, triggering early ray termination after Primitive 3.
Problem 9: Epipolar Line Search Space Reduction Factor
In an unrectified stereo camera pair, matching a pixel coordinate requires searching across a 2D image region of size 800 × 600 pixels (480,000 candidate pixels). After stereo image rectification, the epipolar geometry aligns horizontally, restricting the correspondence search to a 1D horizontal line of length 800 pixels. Calculate search space reduction ratio.
Solution:
  1. Unrectified search space: 480,000 pixels.
  2. Rectified search space: 800 pixels.
  3. Reduction ratio: 480,000 / 800 = 600.0 (a 600× reduction in computational search complexity).
Answer: Stereo rectification provides a 600× search space reduction factor.
Problem 10: 3D Voxel Grid Memory Scaling Calculation
A robotics spatial occupancy grid models an indoor room volume of dimensions 10 × 10 × 3 meters. If spatial resolution is configured to voxel cubic size d = 0.05 meters (5 cm voxels), calculate total voxel count V. If spatial resolution is refined to d = 0.01 meters (1 cm voxels), calculate new voxel count and determine memory scaling expansion factor.
Solution:
  1. Room volume: 10 × 10 × 3 = 300 m3.
  2. 5 cm voxel volume: (0.05)3 = 0.000125 m3. Voxel count V5cm = 300 / 0.000125 = 2,400,000 voxels (2.4 million).
  3. 1 cm voxel volume: (0.01)3 = 0.000001 m3. Voxel count V1cm = 300 / 0.000001 = 300,000,000 voxels (300 million).
  4. Scaling expansion factor: 300,000,000 / 2,400,000 = 125.0× memory increase (53 = 125).
Answer: 5 cm grid requires 2.4 M voxels; 1 cm grid requires 300 M voxels, representing a 125× memory expansion factor.
Problem 11: Render Speedup Factor for 3DGS vs NeRF
An implicit NeRF model renders a 1080p frame (1920 × 1080 = 2,073,600 rays) in 8.50 seconds per frame using 128 ray samples per ray. An explicit 3D Gaussian Splatting model renders the same scene at 120 Frames Per Second (FPS). Calculate rendering latency per frame for 3DGS in milliseconds (ms) and determine total rendering speedup factor of 3DGS over NeRF.
Solution:
  1. NeRF latency: 8,500 ms/frame.
  2. 3DGS latency at 120 FPS: 1.0 / 120 = 0.00833 seconds = 8.33 ms/frame.
  3. Speedup factor: 8,500 ms / 8.33 ms ≈ 1,020.4× rendering speedup.
Answer: 3DGS frame rendering latency is 8.33 ms, representing a 1,020× rendering speedup over NeRF.
Problem 12: Spherical Harmonics Vector Dimension Sizing
View-dependent color in 3D Gaussian Splatting is parameterized using Spherical Harmonics (SH) basis functions up to degree lmax. Total basis functions per color channel equals (lmax + 1)2. For degree lmax = 3 across 3 RGB color channels, calculate total scalar SH coefficient count required per 3D Gaussian primitive.
Solution:
  1. Basis functions per channel for lmax = 3: (3 + 1)2 = 42 = 16 basis coefficients.
  2. Total coefficients across 3 RGB channels: 16 basis functions × 3 color channels = 48 scalar coefficients.
Answer: Degree-3 Spherical Harmonics require 48 scalar coefficients per 3D Gaussian primitive.

3D Vision and Gaussian Splatting — Summary & Wrap-Up

3D vision algorithms bridge physical spatial environments and computational models. By advancing from classical pinhole projection (s p = K [R | t] P) and epipolar stereo depth triangulation (Z = f b / d) to continuous implicit Neural Radiance Fields (NeRF) and explicit 3D Gaussian Splatting (Σ = R S ST RT), modern computer vision enables real-time, photorealistic novel view synthesis, tile-based GPU rasterization, and spatial computing across robotics, autonomous driving, and virtual environments.

Explore the Modern Computer Vision Sub-Cluster

3D computer vision forms a core pillar of modern visual intelligence. To explore classical spatial filtering, temporal video tracking, and deep vision foundation models, navigate the complete sub-cluster:

3D Vision and Gaussian Splatting

This page details multi-view geometry, pinhole camera projection, NeRF volume rendering, and real-time 3D Gaussian Splatting rasterization.

Digital Image Processing and 2D Analysis

Details classical spatial filtering, mathematical morphology, contrast enhancement, image compression file formats, and 2D biometric pattern recognition.

Video Understanding and Action Recognition

Examines spatiotemporal modeling, optical flow, video transformer architectures (TimeSformer), and real-time multi-object tracking across video streams.

Vision Transformers and Foundation Models

Explores modern deep learning vision architectures, patch tokenization in Vision Transformers (ViTs), DETR object detection, CLIP multimodal embedding, and zero-shot visual transfer.
Last updated: 24 Jul 2026