Szeliski Chapter 12 — Depth Estimation Explained

// the one-minute version

Stereo recovers depth by matching the same scene point in two cameras. Rectification makes epipolar lines horizontal, so correspondence becomes a disparity search along each row. For parallel cameras, Z=fB/d: disparity d is inversely proportional to depth, so one-pixel error hurts far points most. Local descriptors build a cost volume; winner-take-all is noisy, so dynamic programming, semi-global matching, MRFs, or learned 3D aggregation impose spatial structure. Occlusion, repeated texture, reflections, and weak texture create genuine ambiguity. Multi-view stereo adds views; structured light and time-of-flight add active measurements; monocular depth learns scene priors and usually lacks metric scale.

Mira’s stereo system reports a smooth wall, but its estimated depth explodes at the far end of the corridor. The disparity error is only a fraction of a pixel. Because depth is inverse disparity, “small image error” is not small 3D error.

01 Rectification turns geometry into search

Calibrated stereo images are warped so corresponding points share a row. Disparity d=x_L−x_R. For focal length f and baseline B, similar triangles yield Z=fB/d. Larger baseline improves depth precision but increases occlusion and matching difficulty.

Uniform disparity precision produces nonuniform depth precision.

02 Matching cost measures local compatibility

Absolute or squared intensity difference assumes photometric consistency. Normalized correlation handles gain/offset; Census compares local ordering and is robust to exposure. Learned descriptors adapt to data. Window size trades noise resistance against boundary bleeding.

03 A cost volume stores competing hypotheses

C(x,y,d) records evidence for every pixel/disparity. Selecting argmin independently creates speckle in weak texture. Confidence can use cost margin, entropy, or left–right consistency. Preserve the distribution when downstream risk needs uncertainty.

04 Regularization propagates without erasing boundaries

Smoothness priors prefer neighboring disparities while allowing discontinuities at likely edges. Dynamic programming optimizes scanlines; semi-global matching aggregates many path costs; graph methods optimize 2D energies. Deep stereo networks learn cost aggregation and context but inherit dataset bias.

05 Occlusion is one-sided evidence

A point visible in the left camera may be hidden in the right. Left–right consistency detects many occlusions. Filling should respect foreground ownership; copying background disparity across a foreground boundary creates flying surfaces.

06 Multi-view stereo adds constraints and visibility

Plane sweeping projects each image onto candidate depth planes and measures agreement. Patch-based methods grow reliable matches. Volumetric methods combine visibility across views. More views reduce ambiguity but introduce varying illumination, scale, and occlusion.

07 Active sensors change the evidence

Structured light projects known texture and triangulates it; time-of-flight measures phase or travel time; lidar scans range directly. Sunlight, multipath, shiny/transparent surfaces, minimum range, and interference cause characteristic holes and bias. Fuse sensors only after calibration and timestamp alignment.

08 Monocular depth learns priors

One image cannot determine scale from projection alone. Networks learn cues such as perspective, object size, texture gradient, and semantics. Supervised metric labels, self-supervised view synthesis, or ordinal depth provide training signals. Predictions can be sharp and plausible yet metrically wrong under new cameras or scenes.

09 Evaluate in the variable the application uses

Disparity endpoint error differs from metric depth error. Absolute relative, RMSE, log error, threshold accuracy, completeness, and boundary error reveal different behavior. Slice by distance, texture, occlusion, material, weather, and sensor saturation.

gotchasNever compare depth after silently fitting scale unless protocol says so. Rectification errors masquerade as matcher errors. Smooth maps can score well while missing thin obstacles. Confidence must be calibrated against actual depth error.

10 Questions

Why is distant stereo depth noisy?

d=fB/Z becomes tiny; differentiating gives depth error that grows roughly with Z² for fixed disparity error.

Why rectify?

It maps epipolar lines to horizontal scanlines, reducing 2D correspondence search to 1D disparity.

Why not use a huge matching window?

It stabilizes textureless regions but crosses depth boundaries and assumes one disparity for mixed surfaces.

Can monocular depth be metric?

With camera-aware metric supervision and appropriate distribution it can predict scale statistically, but monocular geometry alone cannot guarantee it.

What should a robot consume?

Depth plus calibrated uncertainty, validity/occlusion masks, timestamps, and sensor-frame calibration—not a colorized map alone.

11 Summary and lab

Stereo converts correspondence into inverse-depth geometry.
Cost volumes retain alternatives.
Regularization resolves ambiguity but risks boundary bias.
Active and monocular depth have distinct failure physics.
Metrics must reflect range and safety.

// study labchapter 12

derivePropagate subpixel disparity error through Z=fB/d by distance.

auditCompare stereo and monocular depth on glass, textureless walls, thin objects, and far range.

12 Source trail

Original notes following the official book page and Springer’s chapter record.

Chapter 12 — Depth Estimation, explained.