Szeliski Chapter 8 — Image Alignment and Stitching Explained

// the one-minute version

Alignment estimates a warp that maps one image into another. Translation, Euclidean, similarity, affine, and projective models encode increasingly flexible geometry. Feature-based alignment matches landmarks and robustly fits parameters; direct alignment minimizes photometric error over pixels and needs good initialization plus brightness assumptions. A homography aligns a plane or images taken by a rotating camera, not arbitrary 3D translation. Stitching adds global registration, choice of cylindrical or spherical coordinates, exposure compensation, seam selection, and multi-band blending. Ghosting is usually a geometry or scene-motion problem, not a blending problem.

Mira stitches two hallway frames. The far wall aligns, but the nearby railing appears twice. Increasing feathering makes a prettier double railing. The failure is conceptual: one homography cannot align two depths when the camera translates.

01 A warp is a geometric hypothesis

Translation has two parameters; rigid 2D motion adds rotation; similarity adds uniform scale; affine permits shear and nonuniform scale; a homography has eight effective parameters and maps lines to lines. Choose the least flexible family that matches image formation—extra freedom fits noise and distorts content.

02 Feature alignment solves correspondences first

Matched points p′∼Hp provide linear constraints on H. Normalize coordinates before the direct linear transform to improve conditioning, estimate from at least four non-collinear pairs, use RANSAC for outliers, then refine on inliers by geometric reprojection error.

03 Direct alignment optimizes appearance

photometric objectiveE(θ)=Σ_x[I₂(W(x;θ))−I₁(x)]².

Linearize the warped image with spatial gradients and the warp Jacobian. Lucas–Kanade iterations solve a small least-squares update. Inverse compositional alignment precomputes terms when the template is fixed. Robust loss and illumination parameters handle limited violations.

Registration solves where content belongs; blending only resolves how overlapping measurements are combined.

04 Coarse-to-fine enlarges the convergence basin

Direct methods fail when motion exceeds the local linear approximation. Image pyramids estimate broad motion at coarse resolution and refine. But coarse levels can erase small structures or lock onto repeated texture, so monitor residuals at every level.

05 Global alignment prevents chain drift

Pairwise transforms accumulated around a panorama drift. Represent cameras in a common frame and jointly minimize all feature reprojection errors—bundle adjustment. Loop constraints distribute error rather than forcing one visible seam. Gauge freedom is fixed by anchoring one camera.

06 Projection surface determines panorama distortion

Planar projection suits limited field of view. Cylindrical projection handles wide horizontal panoramas while keeping vertical lines straight. Spherical projection handles full viewing directions. Rotational panoramas are cleanest when the camera rotates about its optical center.

07 Photometric calibration makes overlap compatible

Exposure, vignetting, white balance, and response curves differ across images. Estimate gain or low-frequency correction from overlap in linear light. Avoid “fixing” genuine illumination boundaries or saturated pixels.

08 Seams and multi-band blending hide small disagreement

Feathering averages by distance to boundaries. Optimal seams choose low-disagreement paths around objects. Multi-band blending combines low frequencies over wide transitions and high frequencies locally, hiding exposure seams without blurring texture. Moving objects still require content selection or deghosting.

09 Parallax is a model failure

Translation changes relative positions by depth, so no single H aligns foreground and background. Solutions include rotating around the optical center, local warps, multiple planes, seam placement, or full 3D reconstruction. Cropping and straightening are final composition choices, not geometric evidence.

gotchasDo not fit H to a nonplanar translating scene and blame the seam. Optimize geometric, not algebraic, error. Keep interpolation and canvas coordinates consistent. Direct alignment requires linearized brightness. Report coverage and distortion as well as overlap error.

10 Questions

When is one homography physically valid?

For one plane, pure camera rotation, or approximately distant scenes where translation parallax is negligible.

Feature or direct alignment?

Features tolerate large displacement and partial overlap; direct methods exploit all texture for precision but need initialization and photometric consistency.

Why normalize DLT coordinates?

Centering and scaling reduce numerical imbalance among polynomial terms and improve conditioning.

What does multi-band blending solve?

Small residual misalignment and low-frequency exposure differences; it does not solve large parallax or moving-object duplication.

How should stitching be evaluated?

Reprojection residual, loop closure, visible seams, ghosting, line distortion, retained field of view, and runtime on diverse scenes.

11 Summary and lab

Warp families encode scene/camera assumptions.
Features propose; robust geometry and refinement align.
Direct methods optimize pixels locally.
Global adjustment prevents accumulated drift.
Projection, exposure, seams, and blending finish the panorama.

// study labchapter 8

captureCompare rotation-only and translated panoramas at two scene depths.

ablateShow raw warp, exposure correction, seam, feather, and multi-band outputs separately.

12 Source trail

Original notes following the official book page and Springer’s chapter record.

Chapter 8 — Image Alignment and Stitching, explained.