// the one-minute version
Traditional rendering starts with geometry, materials, lighting, and a camera, then predicts pixels. Image-based rendering starts closer to the answer: photographs already contain the scene’s appearance, so it reuses, interpolates, or reprojects them to synthesize a new view. With known depth, each source pixel can be lifted to 3D and projected into the target camera. The hard parts are visibility, disocclusion, sampling, and view-dependent appearance. Light fields trade geometry for dense angular sampling. Layered representations preserve multiple surfaces along one ray. Neural radiance fields learn a continuous view-dependent scene function, but plausible images and correct geometry are separate claims.
Mira gives the operator a virtual camera around the robot. The floor looks stable, but chair legs split and blank crescents appear behind foreground objects. The source images never observed those newly exposed regions. Rendering cannot copy information that was never captured; it must find another view, model the missing scene, or admit uncertainty.
01 A novel view is a sampling problem
A camera image samples rays through space. Moving the camera asks for different rays. If nearby captured rays see the same surface with similar appearance, interpolation works. Large viewpoint changes, thin geometry, specularities, and occlusion break that assumption. Camera placement therefore controls what views can be synthesized reliably.
Multiple sources fill one another’s occlusions; depth determines where each sample lands.
02 View interpolation blends nearby evidence
Image morphing establishes correspondence, warps both images toward an intermediate viewpoint, then blends colors. Without geometry, correspondences encode an approximate scene model. Linear pixel blending alone creates double images because objects do not stay at the same coordinates.
03 Depth-image warping makes geometry explicit
Back-project each source pixel using its depth, transform the 3D point into the target camera, and project it. Forward warping causes holes and collisions; a z-buffer keeps the nearest sample. Splatting gives each projected sample an area to reduce cracks, but excessive splats bleed across boundaries.
04 Disocclusion is missing information
When the target camera sees behind a foreground object, no source pixel may cover the region. Use additional cameras, background layers, or cautious inpainting. A generative fill can be visually plausible but is not observed evidence; scientific and robotic interfaces should label that distinction.
05 Layers preserve more than the nearest hit
A layered depth image stores several color-depth samples along a pixel ray, so moving the camera can reveal background that a single depth map discarded. Billboards and sprites approximate objects with textured planes; multi-plane images discretize depth into fronto-parallel layers. They render quickly but fail under large rotations or incorrect layer assignment.
06 Light fields sample position and direction
In free space, radiance can be parameterized by where a ray intersects two planes. A dense camera array samples this 4D function; a novel view selects and interpolates appropriate rays. Refocusing becomes a shift-and-sum operation. Sparse angular sampling causes ghosting unless geometry-aware interpolation compensates.
07 Appearance changes with viewpoint
Lambertian surfaces approximately keep color across views; glossy surfaces do not. Reflectance fields and environment matting model how light direction, view direction, and transparency affect appearance. Choosing the nearest source camera sometimes preserves view-dependent effects better than averaging incompatible colors.
08 Neural rendering learns the scene function
A neural radiance field maps 3D position and viewing direction to density and color; volume rendering integrates them along each camera ray. Training minimizes rendered-image error across calibrated views. Extensions improve speed, unbounded scenes, relighting, or editable geometry, but results remain sensitive to poses, exposure, motion, and coverage.
09 Good pixels do not guarantee a true scene
A model may interpolate held-out views while encoding wrong geometry, especially with sparse baselines. Evaluate PSNR and SSIM for fidelity, perceptual metrics for appearance, and geometry separately when required. Test extrapolated viewpoints, disocclusions, temporal flicker, and failure under changed lighting.
10 Design from the intended camera path
For a small head-motion box, dense images and layered depth may suffice. For room-scale navigation, explicit geometry and broad coverage matter. For telepresence, latency and temporal consistency dominate. Select capture spacing, representation, and renderer together; they are one sampling system.
11 Summary and lab
- Novel views require rays not present in one photograph.
- Depth warping relocates samples; visibility decides which survive.
- Layers and multiple cameras address disocclusion.
- Light fields exchange geometry for dense sampling.
- Neural rendering does not remove capture or evaluation constraints.
12 Source trail
Original explanatory notes following the official book page and Springer’s chapter record.