// the one-minute version
Recognition assigns semantic structure to visual evidence. Classification predicts labels for an image; retrieval ranks similar instances; detection localizes objects; semantic segmentation labels pixels; instance segmentation separates objects; pose estimates landmarks or 3D configuration. Classical systems combined features, vocabularies, classifiers, and spatial verification; modern networks learn most stages jointly. The difficult parts remain: long-tailed categories, scale, occlusion, context, open-set inputs, annotation ambiguity, and dataset shift. Metrics are task-specific—accuracy, precision–recall, AP, IoU, calibration—and thresholds embody costs.
Mira’s model says “bicycle present,” but the robot needs to know where it is, which pixels belong to it, whether a rider is attached, and where the wheels will move. “Recognition” is not one output; it is a hierarchy of increasingly structured claims.
01 Categories and instances answer different questions
Category recognition groups varied appearances under a semantic label. Instance recognition identifies one particular building or product. Fine-grained recognition separates visually close subclasses. Scene recognition uses global layout and object context. A closed-set classifier assumes every input belongs to a known class; open-set recognition must reject unknowns.
02 Classical recognition built invariant summaries
Local descriptors were quantized into visual words; images became histograms or spatial pyramids; linear SVMs separated categories. Instance retrieval matched local features and verified geometry. These pipelines reveal the enduring structure: representation, pooling, scoring, and verification.
03 Classification estimates evidence, not location
A softmax model yields normalized class probabilities under its training assumptions. Multi-label tasks instead use independent sigmoid scores. Top-k accuracy hides probability quality; confusion matrices expose systematic swaps. Context can help—a boat is likely on water—but can become a shortcut.
04 Detection combines classification with geometry
Two-stage detectors propose regions then classify/refine them; one-stage detectors predict dense candidates directly. Anchors or center-based representations cover scale and aspect. Non-maximum suppression removes duplicate boxes, while newer set-prediction systems learn one-to-one matching.
Each richer output requires stronger annotation and a metric that respects structure.
05 Segmentation assigns meaning to pixels
Semantic segmentation predicts a class per pixel; instance segmentation adds object identity; panoptic segmentation unifies countable “things” with amorphous “stuff.” Encoder–decoder networks combine context with skip-connected detail. Boundary quality, small objects, and class imbalance need explicit attention.
06 Pose adds parts and configuration
Keypoint systems predict heatmaps or coordinates, then impose skeletal or geometric consistency. 6D object pose estimates rotation and translation, often combining correspondences with PnP. Symmetric objects make several poses visually equivalent, so evaluation must respect symmetry.
07 Data defines the taxonomy
Labels reflect annotator instructions and cultural categories. Bounding boxes include background; masks disagree at fuzzy boundaries; missing objects become false negatives during training. Long tails leave rare classes underrepresented. Synthetic and weak labels reduce cost but introduce bias.
08 Metrics turn predictions into decisions
Precision asks how many reported objects were correct; recall asks how many true objects were found. Average precision integrates the precision–recall curve. Intersection-over-union measures overlap, and detection AP averages over IoU criteria. Pixel accuracy can hide rare-class failure; mean IoU weights classes more evenly.
A deployment threshold trades errors according to cost. Calibration and selective prediction allow abstention when evidence is weak.
09 Context, shift, and fairness expose shortcuts
Models can recognize snow rather than wolves, hospital scanners rather than disease, or road regions rather than pedestrians. Evaluate counterfactual backgrounds, geography, camera, weather, object size, occlusion, and relevant human groups. Open-world monitoring must detect unfamiliar inputs and label drift.
10 Questions
Why is class accuracy insufficient for detection?
Detection must also localize an unknown number of objects and handle duplicates, missed objects, confidence ordering, and overlap quality.
Semantic versus instance segmentation?
Semantic labels all same-class pixels alike; instance segmentation separates each countable object.
Why is AP threshold-independent but not cost-free?
It summarizes ranking across thresholds, but deployment still selects a threshold whose precision–recall trade matches real costs.
Can context be removed?
Not completely; context is often genuine evidence. Test whether performance survives interventions that break spurious context.
How do I test open-set behavior?
Mix known and semantically novel inputs, measure rejection and calibration, and avoid tuning the unknown detector on the final unknown set.
11 Summary and lab
- Recognition tasks differ by semantic and spatial output.
- Modern networks inherit classical representation–scoring–verification logic.
- Annotations and taxonomies define learnable truth.
- Metrics must match structure and operating cost.
- Context and dataset shift demand counterfactual testing.
12 Source trail
Original notes following the official book page and Springer’s chapter record.