// the one-minute version
A depth map is not yet a model: it is one camera’s partial, noisy claim about visible surfaces. Reconstruction transforms many such claims into a common world frame, reasons about free space and visibility, fuses repeated evidence, extracts a surface, and gives that surface usable appearance. Point clouds preserve measurements but lack connectivity. Occupancy grids model empty versus occupied space. Signed-distance volumes encode where a surface should cross zero; a truncated signed distance function (TSDF) averages nearby observations and supports mesh extraction. Every representation trades accuracy, memory, topology, and editability. A trustworthy system tracks uncertainty and reports both accuracy and completeness.
Mira drives the robot around a statue. Every depth frame looks convincing, but the combined cloud becomes a furry double shell. The failure is not “bad meshing.” Slight pose errors moved observations into different world locations. Reconstruction begins with a question: what does each measurement actually assert?
01 Lift pixels into a shared world
For depth Z at pixel (u,v), calibrated intrinsics back-project the point: X=(u−cx)Z/fx, Y=(v−cy)Z/fy. The camera pose then transforms it into world coordinates. A calibration or pose error produces systematic surface thickness, so inspect geometry before inventing a stronger smoother.
Fusion combines evidence only after calibration, synchronization, and pose alignment.
02 Point clouds are samples, not surfaces
A point cloud stores positions and perhaps color, normals, timestamps, and confidence. It is excellent for measurement and nearest-neighbor operations, but it does not say which points connect, which side is inside, or what lies between samples. Downsampling should preserve edges and density metadata, not merely make a pretty picture.
03 Visibility tells us about free space
A depth ray says the space before the measured hit was probably free, the hit was occupied, and space behind it is unknown—not occupied. Occupancy mapping accumulates log-odds evidence per cell. This distinction prevents unseen interiors from being incorrectly declared solid.
04 Signed distance turns a surface into a field
A signed distance field stores distance to the nearest surface, with sign separating inside from outside. Its zero level set is the surface. A TSDF keeps only a band around the surface, saving work and limiting the influence of outliers. Weighted averaging lets many noisy depth samples reinforce a stable crossing.
05 Fusion needs weights, not blind averaging
Weight measurements using range noise, viewing angle, sensor validity, motion, and pose confidence. Integrating dynamic people into a static volume creates ghosts. Robust fusion caps influence or rejects inconsistent observations; semantic or motion masks can separate static background from moving objects.
06 Extract topology from the zero crossing
Marching Cubes examines each voxel cube, classifies corner signs, interpolates edge crossings, and emits triangles. Resolution limits the smallest recoverable feature. Ambiguous cases, thin structures, holes, and noisy normals can create broken or non-manifold meshes; cleanup must not silently erase real geometry.
07 Alternative surface methods encode different priors
Delaunay methods reason over tetrahedra and visibility. Poisson reconstruction finds an implicit function whose gradient fits oriented normals, producing smooth watertight surfaces but sometimes closing genuine openings. Ball pivoting follows local samples. Choose by sampling pattern, desired topology, and whether watertightness is truth or convenience.
08 Appearance requires another inference problem
Texture mapping projects images onto mesh triangles. View selection balances resolution, angle, sharpness, exposure, and occlusion. Seams arise when cameras disagree; blending and color correction help, but lighting baked into texture will look wrong under relighting. Geometry and reflectance are not the same variable.
09 Neural fields are continuous representations
Coordinate networks can represent occupancy, signed distance, density, color, or radiance continuously. They compress detail and support differentiable rendering, but training cost, pose dependence, editability, and uncertain geometry matter. A beautiful novel view is not proof of an accurate metric surface.
10 Evaluate both correctness and coverage
Accuracy measures reconstructed-to-reference distance; completeness measures reference-to-reconstruction distance. Chamfer distance combines both but can hide tails. Report percentiles, normal consistency, topology, holes, thin-object recall, and performance by range/material. Align models only with transformations the protocol permits.
11 Summary and lab
- Depth measurements become 3D only through calibration and pose.
- Visibility separates free, occupied, and unknown space.
- TSDF fusion stabilizes a zero-level surface.
- Meshing and texturing introduce new assumptions.
- Accuracy without completeness is an incomplete result.
12 Source trail
Original explanatory notes following the official book page and Springer’s chapter record.