// the one-minute version
Vision repeatedly chooses parameters θ that make predictions agree with measurements. Least squares minimizes Σr² and equals maximum likelihood under independent Gaussian noise. Priors add regularization and produce MAP estimation. Outliers break squared loss, motivating robust penalties and RANSAC. Nonlinear residuals require iterative methods: gradient descent uses slope, Gauss–Newton linearizes residuals, and Levenberg–Marquardt damps unsafe steps. Constraints, initialization, scale, and parameterization determine whether optimization finds a useful solution. Uncertainty comes from residual noise and local curvature; one low final loss is not proof of a correct model.
Mira matches curb points between two frames and fits camera motion. Ten mismatches drag ordinary least squares toward an impossible pose. The equations are working exactly as written—the objective is rewarding compromise with corrupted evidence.
01 A model turns a vision claim into residuals
Let f(xi;θ) predict measurement yi. Residual ri(θ)=yi−f(xi;θ) states how the model fails. Optimization chooses θ minimizing an energy E(θ). Model, residual units, and loss are separate design choices.
02 Least squares is a noise assumption
For linear f=Xθ, normal equations XᵀXθ=Xᵀy yield a solution, but QR or SVD is numerically safer. Least squares is maximum likelihood when residuals are independent zero-mean Gaussians with equal variance. Weighted least squares handles known unequal variance.
03 Priors become regularization
MAP estimation minimizes negative log likelihood plus negative log prior. An L2 prior gives ridge regularization; L1 encourages sparsity; smoothness priors couple neighboring estimates. Regularization resolves underdetermination by declaring which solutions are plausible, so its weight belongs in evaluation.
04 Robust losses limit outlier influence
Squared loss gives influence proportional to residual magnitude, letting one gross mismatch dominate. Huber is quadratic near zero and linear in the tails; Cauchy and Tukey suppress tails further. Iteratively reweighted least squares converts the current robust influence into weights, solves, and repeats.
Robust fitting changes the question from “average every error” to “explain the coherent structure.”
05 RANSAC searches for a clean hypothesis
RANSAC samples a minimal subset, fits a hypothesis, counts points within an inlier threshold, and retains the best consensus. Required iterations rise as inlier fraction falls and minimal sample size grows. Refit on all inliers after selection.
Thresholds must match measurement noise; degeneracy checks reject samples that cannot constrain the model. RANSAC can miss multiple structures or favor the largest wrong one.
06 Nonlinear least squares works by local approximation
Linearize r(θ+Δ)≈r+JΔ. Gauss–Newton solves (JᵀJ)Δ=−Jᵀr. Levenberg–Marquardt adds damping λI, behaving like cautious gradient descent far from a solution and Gauss–Newton nearby. Sparse Jacobians make bundle adjustment feasible.
07 Parameterization shapes the landscape
Angles wrap, rotations must remain orthogonal, probabilities must remain positive, and scales can differ by orders of magnitude. Optimize rotations on an appropriate local representation, normalize data, and use transformations for constrained parameters. A mathematically equivalent parameterization can be far easier numerically.
08 Discrete optimization labels pixels and parts
Segmentation and stereo often minimize unary data costs plus pairwise smoothness. Dynamic programming solves chains; graph cuts solve important binary and metric-label energies; belief propagation passes local messages. Relaxations trade exact combinatorics for tractable continuous problems.
09 Diagnostics matter more than “converged”
Inspect residual maps, inlier locations, Jacobian conditioning, step acceptance, sensitivity to initialization, and held-out prediction. Estimate covariance locally from noise and (JᵀJ)−1, remembering that multimodal or biased problems invalidate a Gaussian summary.
10 Questions
Why not invert XᵀX directly?
Forming normal equations squares the condition number. QR and SVD preserve more numerical accuracy and reveal rank deficiency.
Robust loss or RANSAC?
Robust loss is efficient for moderate contamination around one basin; RANSAC can survive gross outliers by proposing from clean subsets. They are often combined.
Why does initialization matter?
Nonlinear objectives can have multiple basins and local approximations are trustworthy only nearby. Coarse models or minimal solvers provide starts.
What is a gauge freedom?
A transformation of parameters that leaves all predictions unchanged, such as global scale in monocular reconstruction. It makes the Hessian singular until fixed.
How do I validate a fit?
Use held-out residuals, parameter stability, uncertainty, spatial diagnostics, synthetic recovery, and downstream accuracy—not training energy alone.
11 Summary and lab
- Residuals operationalize a model claim.
- Losses encode noise and outlier assumptions.
- RANSAC separates hypothesis generation from consensus.
- Gauss–Newton and LM exploit local residual structure.
- Conditioning, constraints, and gauge freedoms are part of the problem.
12 Source trail
Original notes following Szeliski’s official page and Springer’s chapter record.