Szeliski Chapter 2 — Image Formation.

Treat this page as the “camera reality check” for the Computer Vision course. Before filtering, detecting edges, extracting SIFT descriptors, or training a network, we need to know what assumptions the image itself carries. Chapter 2 gives those assumptions in a compact but deep form.

01 Why image formation is the first real computer vision model

Szeliski starts computer vision with image formation because every later algorithm quietly assumes some story about how pixels came to exist. A pixel is not a tiny object label. It is the end of a physical chain: light leaves a source, interacts with surfaces, passes through optics, hits a sensor, is sampled on a grid, is quantised into numbers, and is often processed again before you ever load the image in Python. If you forget that chain, you will over-trust images. If you understand it, you can reason about why shadows fool segmentation, why wide-angle cameras bend straight lines, why distant objects shrink, why low light creates noise, and why a red shirt can look different under sunlight and tube light.

For AIMLCZG525, this chapter is the bridge between CV1 foundations and the first practical image operations. The course may later talk about filtering, edges, corners, feature matching, stereo, calibration, recognition, and deep networks, but all of those are downstream from the camera model. A convolutional network still receives samples produced by optics and electronics. A feature detector still operates on projected texture. A stereo system still relies on two images that obey projective geometry. Image formation is therefore not a side topic; it is the physical contract behind the algorithms.

The inverse problem is hard because many different worlds can create the same image. A small nearby object and a large faraway object can occupy the same number of pixels. A white surface in shadow and a grey surface in bright light may have the same intensity. A line in the world may be curved by lens distortion. This many-to-one nature is why vision needs assumptions, priors, calibration, geometry, and statistics. Chapter 2 gives you the vocabulary for those assumptions.

02 Geometric primitives: points, vectors, lines, planes, and homogeneous coordinates

Before a camera can project the world, we need a way to describe geometry. The basic primitive is a point. In an image, a point may be written as \((x,y)\). In 3D, it is \((X,Y,Z)\). A vector represents a displacement or direction. A line in 2D can be written as \(ax+by+c=0\). A plane in 3D can be written as \(aX+bY+cZ+d=0\). These equations are simple, but they become much cleaner when written with homogeneous coordinates.

Homogeneous coordinates add one extra coordinate. A 2D image point becomes \(\mathbf{x}=(x,y,1)^T\), or more generally \((\tilde{x},\tilde{y},\tilde{w})^T\), where the visible coordinate is recovered by division: \(x=\tilde{x}/\tilde{w}\), \(y=\tilde{y}/\tilde{w}\). A 3D point becomes \(\mathbf{X}=(X,Y,Z,1)^T\). The trick looks strange at first, but it lets translation, rotation, scaling, affine transforms, and perspective projection all be written as matrix multiplication.

This matters because cameras are projective devices. They do not preserve Euclidean distances in the image. Parallel 3D lines can meet at a vanishing point. Ratios along a line may change. Homogeneous coordinates handle points at infinity naturally, which is exactly what vanishing points are. In practice, when you see a camera matrix \(P\), a homography \(H\), or an affine transform \(A\), you are seeing homogeneous-coordinate thinking.

A useful exam habit is to separate representation from meaning. The vector \((2,4,2)^T\) and \((1,2,1)^T\) represent the same 2D point because both dehomogenise to \((1,2)\). The scale is arbitrary. That is why projective equations are often defined “up to scale,” written with \(\sim\): \(\mathbf{x} \sim P\mathbf{X}\).

03 2D transformations: translation, rotation, similarity, affine, and projective warps

Images are full of transformations. A patch may shift because the camera moved. It may rotate because the object rotated. It may scale because the object moved closer. It may shear because a planar surface is viewed obliquely. Szeliski’s geometry chapter builds the family tree of these transformations because each family preserves different properties.

A translation moves every point by the same offset: \(x'=x+t_x\), \(y'=y+t_y\). A Euclidean transform adds rotation while preserving distances and angles. A similarity transform adds uniform scale, so shape is preserved up to size. An affine transform allows independent scaling and shear; it preserves straight lines and parallelism, but not angles. A projective transform, or homography, is the most general 2D planar warp used often in vision; it preserves straight lines but not parallelism. A homography is written as

The hierarchy is important because weaker models are more constrained and easier to estimate. If the only change is a small shift, a translation model is robust. If you use a full homography unnecessarily, you may overfit noise. But if a planar poster is seen from a slanted angle, a translation model will fail and a homography becomes natural. This is why image stitching, document rectification, panorama creation, and planar tracking rely heavily on projective transforms.

04 3D rigid motion and coordinate frames

A camera does not observe points in some universal coordinate system. It observes points relative to its own coordinate frame. Therefore we need coordinate transforms. A 3D rigid transform maps a point from world coordinates to camera coordinates using rotation and translation:

The rotation matrix preserves lengths and angles, so \(R^TR=I\) and \(\det(R)=1\). In real systems it may be represented by Euler angles, axis-angle vectors, quaternions, or directly as a matrix. For introductory computer vision, the key idea is simpler: before projection, a world point must be expressed in the camera’s own coordinate system. Only then does the pinhole model make sense.

This is also where extrinsic camera parameters enter. The extrinsics describe where the camera is and how it is oriented relative to the world. If the intrinsics are the camera’s internal geometry, the extrinsics are the camera’s pose. Calibration problems usually estimate both: internal parameters such as focal length and principal point, and external pose for each image of a calibration pattern.

In the CV lectures, this viewpoint explains why camera pose estimation, stereo, structure from motion, and augmented reality all feel related. They all ask some version of: what 3D-to-camera transform makes the observed 2D points plausible?

05 The pinhole camera: projection with depth in the denominator

The pinhole camera is the cleanest mathematical model of image formation. Imagine a tiny hole in a box. Light from a 3D point passes through the hole and hits the image plane. Similar triangles give the central projection equation. If a point in camera coordinates is \((X,Y,Z)\), then the ideal image coordinates are

The focal length \(f\) controls the field of view and magnification. Larger \(f\) behaves like zoom: the same object occupies more pixels. Smaller \(f\) gives a wider view. Depth \(Z\) appears in the denominator, which is the source of perspective: farther objects shrink, and parallel lines in 3D may converge in the image.

With pixel coordinates, we usually include focal lengths in pixel units and the principal point. A common intrinsic matrix is

Here \(f_x\) and \(f_y\) include focal length and pixel scale, \(s\) is skew, and \((c_x,c_y)\) is the principal point. Many modern cameras have nearly zero skew, but the full matrix is useful. The complete projection is often written as \(\mathbf{x}\sim K[R|\mathbf{t}]\mathbf{X}\).

For learners, the most important mental model is this: projection loses depth. The 2D point tells you the ray direction from the camera, not the unique 3D point. Every possible 3D point along that ray can map to the same pixel. This is why a single image cannot recover absolute depth without additional assumptions or learned priors.

06 Intrinsics, extrinsics, calibration, and why checkerboards matter

Camera calibration estimates the numerical parameters that connect world geometry to pixel measurements. Intrinsics describe the camera itself: focal length, principal point, skew, aspect ratio, and distortion parameters. Extrinsics describe camera pose for a particular view: rotation and translation. Calibration is not glamorous, but it is the difference between image processing as pretty pictures and computer vision as metric measurement.

A checkerboard or dot grid is commonly used because its geometry is known. The algorithm observes where known 3D or planar points appear in the image. It then solves for camera parameters that minimise reprojection error: the distance between observed image points and projected model points. The basic objective is

Calibration also teaches a general lesson: models are never perfect. A pinhole camera with no distortion is elegant, but real lenses bend rays. Sensors may not be perfectly aligned. Detected corner locations have noise. Therefore calibration is an estimation problem, not a lookup. You fit a model that is useful enough for downstream tasks.

07 Lenses, aperture, focus, depth of field, and distortion

The pinhole model is sharp but physically inefficient. A tiny hole admits very little light, so real cameras use lenses to gather more light while focusing rays onto the sensor. The thin lens equation is often written as

Focus matters because only a certain depth range maps sharply to the sensor. Points outside the focus plane become blur disks called circles of confusion. A small aperture increases depth of field but lets in less light. A large aperture gathers more light but creates shallower focus. In photography this is aesthetic; in computer vision it is information. Defocus blur can hide texture, smear edges, or even be used as a depth cue.

Real lenses also distort geometry. Radial distortion bends points inward or outward relative to the optical centre. Barrel distortion makes straight lines bulge outward; pincushion distortion pulls them inward. Tangential distortion occurs when lens elements and sensor planes are not perfectly aligned. A common radial model modifies normalised coordinates by a radius-dependent factor such as \(1+k_1r^2+k_2r^4+\cdots\).

08 Photometric image formation: radiance, irradiance, shading, and BRDF

Geometry tells us where a scene point lands in the image. Photometry tells us how bright and what colour it appears. This part is often harder because intensity depends on illumination, surface material, orientation, camera response, and exposure. A matte white wall in shadow can be darker than a grey wall in sunlight. Therefore brightness is not a direct label for reflectance.

A simplified image irradiance equation can be described as: sensor measurement depends on incoming radiance integrated through the lens, aperture, exposure time, spectral sensitivities, and sensor response. For a Lambertian surface, a common shading model is

Lambertian reflectance means the surface looks equally bright from all viewing directions once illumination geometry is fixed. Many rough surfaces approximate this, but shiny materials do not. General reflectance is described by a BRDF, the bidirectional reflectance distribution function, which tells how much light arriving from one direction is reflected toward another direction. Specular highlights, glossy paint, metal, glass, and wet roads require richer models than Lambertian shading.

This is why “same object, same pixel value” is false. Change the light, exposure, view angle, surface normal, or camera pipeline, and the pixel changes. Good vision algorithms either model these factors, normalise them, or learn robustness to them.

09 The digital camera pipeline: sensor, exposure, noise, sampling, and quantisation

A digital sensor converts photons into electrical charge and then into numbers. Each photosite collects light during the exposure time. Longer exposure collects more photons but may create motion blur. Higher gain or ISO amplifies the signal but also amplifies noise. Low light is difficult because photon arrival itself is noisy; this is often called shot noise. Electronics add read noise, dark current, fixed-pattern noise, and compression artefacts.

After light is converted to charge, the signal is sampled on a discrete grid and quantised into finite intensity levels. Sampling chooses where measurements are taken. Quantisation chooses how many numeric levels represent the measurement. If the image has more spatial detail than the sampling grid can represent, aliasing occurs. Fine stripes may turn into fake lower-frequency patterns. This is why cameras use optical low-pass filtering and why image resizing should blur before downsampling.

The raw sensor image is not usually the final JPEG. The camera may subtract black level, correct defective pixels, demosaic colour filter data, white-balance, denoise, sharpen, tone-map, apply gamma, and compress. A computer vision model trained on JPEGs is therefore learning from a processed image, not untouched photons.

10 Colour: spectra, Bayer mosaics, white balance, gamma, and colour spaces

Colour begins as spectrum: light energy at different wavelengths. Human colour perception compresses spectra into three cone responses, and cameras mimic this with red, green, and blue filters. Most consumer sensors do not measure RGB at every pixel. They use a colour filter array, usually the Bayer pattern, where each photosite measures only one colour channel. A common pattern has twice as many green samples as red or blue because human luminance perception is more sensitive to green.

Demosaicing reconstructs a full RGB value at every pixel from neighbouring colour samples. This is an interpolation problem, and it can create artefacts near edges or fine patterns. White balance tries to compensate for illumination colour. A sheet of white paper should look white under daylight and indoor light, but the raw sensor values differ. Gamma correction maps linear sensor values to a perceptually useful nonlinear scale. The common idea is

Colour spaces organise colour for different tasks. RGB is convenient for displays and sensors. HSV separates hue, saturation, and value in a way that can be intuitive for thresholding, though it is not perceptually uniform. CIE spaces are designed around human perception. YCbCr separates luminance from chrominance and is common in compression. For vision, the right colour space depends on the invariance you need.

11 How Chapter 2 connects to CV1-CV7 lectures and later algorithms

For the mid-semester CV1-CV7 sequence, Chapter 2 is the foundation layer. When you study image processing in Chapter 3, the sampling and noise model explains why smoothing helps and why aliasing appears. When you study features in Chapter 4, projection explains why features change scale and orientation; photometry explains why descriptors need some illumination robustness. When you later study stereo, epipolar geometry, calibration, and structure from motion, the pinhole model and camera matrix become the central language.

The chapter also builds intuition for deep learning-based vision. Neural networks may not explicitly contain a camera matrix, but their input distribution is still shaped by cameras. Data augmentation with crops, rotations, brightness changes, blur, and colour jitter is basically a practical acknowledgement of image formation variability. If the training set does not cover the formation conditions seen at test time, performance drops.

A clean revision strategy is to ask three questions for every image problem: what geometry maps the world to the sensor, what photometry maps light to intensities, and what digital processing maps sensor measurements to the image file? Most “mysterious” failures become easier to explain when viewed through those three lenses.

12 Worked revision example: from a real scene to a stored pixel

Imagine photographing a printed calibration checkerboard kept on a desk under a warm LED bulb. The black and white squares are planar 3D surfaces. Their corner coordinates can be written in the checkerboard coordinate frame, usually with \(Z=0\). The board is placed in front of the camera, so a rigid transform \([R|\mathbf{t}]\) maps those board points into the camera frame. The pinhole part of the model projects each corner by dividing by depth. The intrinsic matrix \(K\) converts normalised image coordinates into pixel coordinates. If the lens has radial distortion, the ideal projected point is then pushed slightly outward or inward depending on its distance from the principal point.

Now think photometrically. A white square is not stored as “white.” The LED has a spectrum. The paper has a reflectance spectrum. The surface normal changes the amount of light reaching the camera according to shading. The aperture and exposure time decide how many photons are collected. The sensor converts those photons to charge, adds noise, clips if saturated, and records one colour component at each Bayer photosite. The image signal processor demosaics, white-balances, denoises, sharpens, tone-maps, applies gamma, and possibly compresses to JPEG. By the time OpenCV reads the file, the pixel is the output of a long chain, not a primitive fact about the world.

This example is exactly why calibration images should be taken carefully. The board should cover different parts of the image so distortion is observable. It should be tilted in several ways so pose and focal length are well constrained. Corners should be sharp, not motion-blurred. Exposure should avoid saturation because saturated corners are hard to localise. A calibration dataset with only fronto-parallel images near the centre may look clean but still estimate poor parameters because it has not excited the geometry of the model.

For exams, this worked example lets you connect many definitions in one story. The checkerboard points are 3D primitives. The board-to-camera pose is extrinsic geometry. The focal lengths and principal point are intrinsics. The pinhole equation explains perspective. Distortion explains curved straight lines near the image edge. The LED, paper, surface normal, and sensor explain intensity. Bayer and demosaicing explain colour sampling. Gamma explains why stored numbers are not linear light. If you can narrate one photograph from scene to pixel in this way, you have understood the chapter at a useful level.

For projects, the same thinking becomes a debugging checklist. If a homography estimate is unstable, ask whether the scene is really planar or whether parallax is present. If feature matching fails across two cameras, ask whether focal lengths, distortion, exposure, and white balance differ. If a model trained indoors fails outdoors, ask whether the photometric distribution changed. If a small texture pattern creates strange colours, suspect demosaicing or aliasing. Chapter 2 is not just theory; it is the first diagnostic tool you use when a vision system behaves strangely.

13 References and extra reads

The primary reference is Richard Szeliski, Computer Vision: Algorithms and Applications, 2nd edition, Chapter 2, available from the official book site at szeliski.org/Book. For complementary intuition, look at classic camera calibration notes from computer vision courses, OpenCV camera calibration documentation, and photography resources on aperture, exposure, ISO, depth of field, and lens distortion. The photography material may seem less mathematical, but it builds excellent physical intuition.

Extra topics worth exploring after this chapter are Zhang camera calibration, radial and tangential distortion correction, radiometric calibration, colour constancy, RAW image processing, high dynamic range imaging, and rolling shutter. These are not all needed for the mid-semester exam, but they explain many real-world surprises.

FAQ Quick questions students usually ask

Why start computer vision with cameras instead of algorithms?

Because algorithms operate on pixels, and pixels are produced by a physical imaging chain. Understanding that chain explains common failures such as distortion, shadows, blur, noise, and aliasing.

What is the difference between intrinsics and extrinsics?

Intrinsics describe the internal camera model, such as focal length and principal point. Extrinsics describe camera pose relative to the world, namely rotation and translation.

Why does one image not give exact depth?

A pixel corresponds to a ray through the camera centre. Many 3D points along that ray project to the same pixel unless extra information is added.

Is lens distortion just perspective?

No. Perspective is ideal central projection. Distortion is the real lens deviating from that ideal model.

Why is Bayer sampling important?

Because most cameras do not measure full RGB at every pixel. Demosaicing reconstructs missing colour channels and can create artefacts.

!! Exam traps, implementation gotchas, and debugging smells

++ Takeaways and chapter cheatsheet

• Image formation has geometry, photometry, optics, and digital sampling components.
• The pinhole equation places depth in the denominator, which creates perspective shrinkage and loss of depth.
• Camera calibration estimates intrinsics, extrinsics, and distortion so image measurements can be trusted.
• Brightness depends on illumination, reflectance, surface orientation, exposure, and sensor response, not just object identity.
• Digital images are sampled, quantised, demosaiced, white-balanced, and often gamma-compressed before algorithms see them.