// the one-minute version
Computer vision turns images and video into estimates about the world: shape, motion, identity, layout, and events. The hard part is that image formation is many-to-one. A dark pixel might be black paint, shadow, weak illumination, camera exposure, or sensor error. Vision is therefore an inverse problem: start from measurements and infer hidden causes using geometry, physics, statistical priors, and learned data. The field ranges from low-level filtering through mid-level correspondence and segmentation to high-level recognition and 3D reasoning. No method is “good” without a task, data distribution, metric, baseline, uncertainty analysis, and failure study.
Mira points her phone at a campus courtyard. She instantly sees a cyclist behind a tree, understands that the pavement continues under the bicycle, recognizes glass from reflections, and predicts that the cyclist will emerge on the right. Her delivery robot sees a rectangular tensor of noisy red, green, and blue numbers. Chapter 1 begins with that gap: how can a machine recover useful facts about a three-dimensional, changing world from incomplete two-dimensional measurements?
01 Vision is not “looking at pixels”
An image is a sampled measurement of light arriving at a sensor. The useful variables usually live elsewhere: which surfaces exist, how far away they are, which object produced each region, how objects move, what material they are made of, and what will happen next. Computer vision constructs estimates of those latent variables.
This makes vision different from image processing, although the fields overlap. Image processing often maps an image to another image—denoise, sharpen, resize, recolor. Vision maps images to descriptions, decisions, geometry, or actions. Computational photography uses scene and camera knowledge to produce better images. Modern systems combine all three.
02 The forward problem is easy; the inverse problem is ambiguous
The forward problem asks: given scene geometry, materials, illumination, camera optics, exposure, and sensor response, what image will be formed? Graphics and image formation model this direction. Vision asks the inverse: given the resulting image, what scene and camera most plausibly caused it?
The inverse is ill-posed because different worlds can produce nearly identical pixels. A small nearby object and a large distant object may have the same projection. A gray patch under bright light can match a white patch in shadow. A moving edge does not reveal full two-dimensional motion. Occlusion deletes evidence entirely.
Forward rendering selects one image from a known world. Inverse vision must select plausible worlds from one incomplete image.
03 Priors are not cheating—they make inference possible
To choose among explanations, a vision system uses prior structure. Parallel lines in the world often remain geometrically related in the image. Neighboring pixels often belong to the same surface. Natural images contain more low-frequency than high-frequency energy. Objects occupy characteristic shapes. Lighting varies more smoothly than reflectance. Learned systems encode statistical regularities from data instead of writing each prior by hand.
A Bayesian statement makes the logic explicit:
The likelihood says how a proposed scene would generate the observed image; the prior says how plausible that scene was before seeing the image. Maximum likelihood ignores the prior. Maximum a posteriori estimation chooses the most probable explanation after combining both. Deep networks often hide this decomposition inside learned parameters, but they do not escape it—the training distribution becomes a powerful prior.
04 The field is organized by what must be recovered
Low-level methods operate near the measurement: color correction, filtering, edge detection, pyramids, and local features. Mid-level methods organize pixels into correspondence, regions, contours, motion, and depth. High-level methods classify scenes, detect and segment objects, estimate pose, answer questions, and support action.
Another organization follows geometry. Single-view methods infer from one image. Multi-view methods exploit repeated observations and camera motion. Video adds temporal coherence and dynamic objects. Recognition uses appearance and semantics; reconstruction uses calibrated geometric constraints. Real applications rarely respect chapter boundaries.
Measurement
How did light, optics, exposure, sampling, and noise create these numbers?
Correspondence
Which pixels or features across images refer to the same world point?
Geometry
What camera, depth, motion, and surface arrangement explain the projections?
Semantics
Which objects, regions, relations, and events matter for the task?
05 A vision pipeline is a chain of representations
A classical system might calibrate a camera, normalize intensity, detect corners, compute descriptors, match them, robustly estimate a transformation, triangulate points, and optimize all camera poses. A learned system might map pixels directly to detections, but still contains implicit transformations from local texture to shape, context, and decisions.
Each representation discards information. An edge map discards region intensity; a descriptor trades precise appearance for invariance; a class label discards pose and instance detail. Good design preserves what the downstream task needs while removing nuisance variation.
06 Recognition and reconstruction ask complementary questions
Recognition asks what is present: category, instance, part, action, or scene. Reconstruction asks where and in what shape: camera pose, depth, surface, and motion. Semantics can help geometry—a road is probably locally smooth; a person is nonrigid. Geometry can help recognition—size and support relationships rule out implausible labels.
The second edition reflects this convergence. Deep learning is a central tool, but image formation, features, matching, alignment, motion, and multi-view geometry remain essential because data and networks do not remove physical constraints.
07 Applications define different meanings of “correct”
In photo organization, a false duplicate is annoying. In medical screening, false negatives may be dangerous. An autonomous vehicle needs calibrated uncertainty and real-time latency, not only average accuracy. Augmented reality demands stable low-latency camera pose; image search needs retrieval ranking; computational photography may optimize perceptual quality.
The same image can support many outputs. Therefore a dataset without a deployment question is incomplete. Mira’s robot cares about traversability and collision risk, not perfect names for every flower. A model that wins a generic benchmark can fail the actual decision.
08 Evaluation is part of the algorithm
Define the unit of prediction, ground truth, metric, split, and operating point. Classification uses accuracy or calibrated probabilities; retrieval uses precision–recall and ranking measures; detection uses intersection-over-union and average precision; depth uses absolute, relative, or scale-invariant error; pose uses translation and rotation error.
A single mean hides tails. Break results down by illumination, distance, object size, motion, weather, camera, and demographic or geographic group when relevant. Prevent near-duplicate frames or scenes from leaking across train and test. Compare against a simple baseline and report uncertainty across runs or samples.
09 Why apparently small changes break vision
Vision systems are sensitive to distribution shift because appearance combines many causes. Change camera response, focal length, compression, weather, geographic context, clothing, or annotation policy and the pixel statistics move. Correlation learned from one domain may not survive another.
Adversarial examples dramatize sensitivity, but ordinary shifts are more common. An image may also be intrinsically ambiguous, so calibrated uncertainty is not optional. A system should know when evidence does not support one confident answer.
10 Questions a master’s student should be able to answer
Why is vision called an inverse problem?
We observe an image produced by hidden scene, lighting, optics, and sensor variables and try to infer those causes. The forward mapping loses information, so multiple explanations can fit the same pixels.
Are hand-designed geometry and deep learning competing paradigms?
No. Geometry encodes projection constraints and interpretability; learning captures complex appearance statistics. Modern systems often learn features, correspondences, priors, or update rules inside geometrically structured pipelines.
What is the difference between segmentation and detection?
Detection localizes objects, commonly with boxes and scores. Semantic segmentation labels each pixel by class. Instance segmentation separates individual objects while assigning pixel masks. Panoptic segmentation combines instance and “stuff” regions.
Why can high validation accuracy still fail in deployment?
The validation distribution may not match deployment; leakage or duplicate scenes may inflate results; the metric may ignore rare costly errors; thresholds, latency, and uncertainty may not meet system requirements.
What should I implement before a large neural model?
Visualize raw data and annotations, build a deterministic preprocessing pipeline, create a simple baseline, define splits by independent scene or subject, choose error breakdowns, and manually inspect false positives and negatives.
11 Chapter summary and master’s study sheet
- Vision estimates hidden world variables from images; image processing alone need not interpret them.
- Image formation is many-to-one, so inverse inference needs physical models and statistical priors.
- Representations form a pipeline from measurements through correspondence and geometry to semantics and decisions.
- Recognition and reconstruction cooperate; learning and geometry are complementary.
- Applications define correctness through task loss, risk, latency, and operating conditions.
- Evaluation is a scientific contract involving distributions, annotations, metrics, baselines, uncertainty, and failure slices.
derive
design
practice
12 Source trail and scope
This is independent companion material, not a replacement for the textbook. The chapter sequence follows the author’s official second-edition page and Springer’s official table of contents. Explanations, Mira’s case study, diagrams, questions, and study exercises are original.