// the one-minute version
Optical flow estimates apparent image motion, not automatically 3D object velocity. Brightness constancy I(x+u,y+v,t+1)=I(x,y,t) linearizes to Ixu+Iyv+It=0—one equation for two components. Lucas–Kanade assumes constant motion in a textured neighborhood; Horn–Schunck adds global smoothness. Robust penalties preserve boundaries. Pyramids handle large displacement but can lose small fast objects. Feature tracking, dense flow, parametric motion, and layered models answer different questions. Occlusion, illumination change, blur, rolling shutter, and nonrigid motion violate assumptions and must be detected or modeled.
A cyclist crosses Mira’s camera. Flow on the bicycle points right, flow from robot rotation sweeps the background left, and newly revealed pavement has no correspondence in the previous frame. A single “motion vector” cannot describe the image.
01 Image motion mixes camera, depth, and objects
Projection converts 3D velocity and camera motion into 2D displacement dependent on depth. Nearby static points move faster under translation than distant ones; pure rotation is depth independent. Optical flow is the field consistent with image change, which can differ from true projected motion at illumination boundaries or occlusions.
02 Brightness constancy yields the flow constraint
A first-order Taylor expansion assumes small displacement and constant appearance. One equation cannot recover u,v. Along an edge, only normal motion is visible: the aperture problem.
Texture supplies independent equations; smoothness supplies a prior where texture does not.
03 Lucas–Kanade solves motion in a patch
Assume u,v constant in a weighted window and solve stacked gradient equations by least squares. The same second-moment matrix used for corners reveals whether the patch constrains both components. Iterative warping refines motion; pyramids initialize large displacement coarsely.
04 Dense flow needs regularization
Horn–Schunck minimizes brightness error plus spatial variation in u,v. Smoothness fills textureless regions but crosses object boundaries. Robust norms, edge-aware weights, total variation, and learned priors preserve discontinuities better while making optimization harder.
05 Feature tracking follows selected landmarks
Kanade–Lucas–Tomasi tracking chooses well-conditioned corners, estimates local motion, and rejects tracks with high residual or forward–backward inconsistency. Re-detection replenishes lost features. Tracking-by-detection instead associates semantic detections across frames using appearance and motion models.
06 Parametric and layered motion simplify structure
A global affine or projective model captures stabilization or dominant camera motion. Layers partition the image into regions with separate motion and ownership, explaining boundaries and occlusion. Scene flow extends motion into 3D using depth or multiple cameras.
07 Occlusion breaks correspondence
Forward–backward consistency and warped residuals reveal pixels with no valid match. Motion boundaries indicate likely depth or object boundaries. Treating occluded pixels as ordinary data smears flow and biases motion models.
08 Modern learned flow still encodes classical structure
Networks build feature pyramids, cost volumes, warping, and recurrent refinements—learned versions of correspondence and iterative optimization. Synthetic ground truth helps, but domain shift in texture, blur, and motion requires real tests. Uncertainty should rise in occlusion and repeated texture.
09 Evaluation must separate motion regimes
Endpoint error measures vector distance; angular error emphasizes direction; outlier rates capture catastrophic mistakes. Slice by speed, boundary distance, occlusion, texture, illumination, and object size. For downstream navigation, measure trajectory or collision impact too.
10 Questions
Why is flow underdetermined locally?
Brightness constancy gives one scalar equation for horizontal and vertical motion.
What makes a good KLT feature?
A patch whose gradient matrix has two sufficiently large eigenvalues, so motion is constrained in both directions.
Why use a pyramid?
Large full-resolution displacement becomes smaller at coarse scale, entering the linearization basin.
How is flow different from tracking?
Flow estimates dense or semi-dense displacement; tracking maintains identities or selected trajectories across many frames.
How do I test occlusion handling?
Evaluate occluded/non-occluded pixels separately and inspect forward–backward consistency near motion boundaries.
11 Summary and lab
- Image flow mixes scene and camera motion.
- The aperture problem requires neighborhoods or priors.
- Local and global methods trade detail for support.
- Occlusion is missing correspondence, not noise.
- Learned flow retains pyramids, costs, and iterative refinement.
12 Source trail
Original notes following the official book page and Springer’s chapter record.