Szeliski Chapter 3 — Image Processing.

This page turns Chapter 3 into a practical revision map for CV lectures. Read it as the bridge between camera physics and feature extraction: most later detectors are built from these operations.

01 What image processing is doing between formation and recognition

Chapter 3 is the toolbox chapter. Chapter 2 gave us pixels as physical measurements; Chapter 3 asks how to clean, transform, enhance, resample, and represent those pixels before higher-level reasoning. In a classic computer vision pipeline, image processing comes before feature detection, segmentation, recognition, stereo, or tracking. In modern deep learning systems, some preprocessing may be learned, but the same ideas still appear: smoothing, contrast normalisation, resampling, pyramids, and frequency control.

Image processing is not merely “making an image look nicer.” It is controlled manipulation of signals. A filter may reduce sensor noise, but it may also blur edges. Histogram equalisation may reveal details, but it may amplify noise. A geometric warp may align images, but interpolation can soften texture. Every operation trades one property for another. Szeliski’s treatment is valuable because it keeps the signal-processing viewpoint visible instead of presenting operations as magic buttons.

For AIMLCZG525, Chapter 3 directly supports the CV lectures on filtering, edge detection preparation, pyramids, and transformations. If you understand this chapter, the later feature detectors make more sense because Harris, SIFT, Canny, HoG, and many descriptors are built from gradients, smoothing, scale-space, and local histograms.

02 Image representation, dynamic range, and why data types matter

Before applying any operation, ask what the image values mean. A grayscale image may store 8-bit integers from 0 to 255, floating-point values from 0 to 1, signed gradients, HDR radiance, or labels. A colour image may be RGB, BGR, YCbCr, HSV, or linear sensor values. The same numeric operation can mean different things depending on representation.

Dynamic range is the span between the smallest and largest useful intensity. An 8-bit image has 256 discrete levels per channel. A 12-bit RAW file has 4096 levels. Floating point can represent much more. If you compute gradients or convolution using unsigned 8-bit arithmetic, negative values can clip or wrap, producing nonsense. That is why many libraries internally convert to float for filtering and derivative operations.

Boundary handling is another representation issue. A convolution kernel near the image edge asks for neighbours outside the image. Libraries may zero-pad, replicate border pixels, reflect the image, or wrap around. Each choice changes output near borders. In exams and implementation, state the boundary assumption when it matters.

03 Point operators: brightness, contrast, thresholding, gamma, and lookup tables

A point operator changes each pixel independently. It does not look at neighbours. Examples include adding brightness, multiplying contrast, thresholding, clamping, inversion, gamma correction, and piecewise tone curves. Mathematically, a point operator is \(g(x,y)=T(f(x,y))\), where \(T\) is a scalar function applied to the pixel value.

Linear contrast stretching maps an input interval to a wider output interval. If an image uses only values from 80 to 150, stretching can use the full display range. Thresholding turns grayscale into binary by choosing a cutoff. Gamma correction changes midtone brightness without changing black and white endpoints. A typical gamma mapping is

Point operators are fast because they can be implemented as lookup tables for 8-bit images. But they cannot remove spatial noise, sharpen edges, or detect structure, because those require neighbourhood information. Their main role is intensity remapping.

04 Histograms, equalisation, and contrast-limited enhancement

A histogram counts how many pixels fall into each intensity bin. It ignores spatial arrangement. Two images can have the same histogram and completely different content. Still, histograms are powerful for contrast analysis because they show whether values are squeezed into a narrow range, clipped, or spread out.

Histogram equalisation tries to remap intensities so the output histogram is more uniform. The mapping is based on the cumulative distribution function:

The intuition is simple: frequently occurring intensity ranges are spread apart, while rare ranges are compressed. This can reveal detail in dark or low-contrast images. But it can also exaggerate noise and create unnatural contrast. Adaptive histogram equalisation works locally, and CLAHE limits contrast amplification to reduce noise blow-up.

05 Convolution and correlation: the core neighbourhood operation

Linear filtering replaces each pixel by a weighted sum of neighbouring pixels. With convolution, the kernel is flipped before sliding; with correlation, it is not. Many practical image-processing libraries use correlation-like implementations but still call the operation filtering. For symmetric kernels like Gaussian blur, the flip makes no difference. For derivative kernels, sign can change.

For a discrete image \(f\) and kernel \(h\), convolution is

The kernel encodes the operation. A box kernel averages. A Gaussian kernel smooths with centre-weighted neighbourhoods. A derivative kernel measures change. A Laplacian kernel measures second derivative-like structure. Sharpening can be built by subtracting a blurred image from the original, a method known as unsharp masking.

Linearity gives two useful properties. Superposition means filtering a sum equals the sum of filtered parts. Shift invariance means the same kernel is used everywhere. Those assumptions are powerful and efficient, but also limited: a linear filter treats an edge like noise if both are high frequency. That motivates nonlinear and edge-preserving filters later.

06 Smoothing filters: box, Gaussian, median, and the noise-detail trade-off

Smoothing reduces rapid intensity variation. If the variation is noise, smoothing helps. If the variation is an edge or texture, smoothing removes useful information. The entire art of denoising is managing this trade-off. A box filter replaces each pixel by the average of a rectangular neighbourhood. It is simple and can be fast using integral images, but it creates blocky frequency behaviour and can blur edges harshly.

The Gaussian filter is usually preferred because it is smooth, isotropic, and has excellent scale-space properties. Its continuous 2D form is

The parameter \(\sigma\) controls scale. Small \(\sigma\) removes fine noise. Large \(\sigma\) removes more detail and keeps only broad structure. Gaussian smoothing is central to scale-space, Canny edges, SIFT, pyramids, and many feature detectors.

The median filter is nonlinear. It replaces a pixel by the median value in its neighbourhood. It is excellent for salt-and-pepper noise because isolated extreme values disappear without averaging them into neighbours. However, it can distort thin structures and is less directly described by Fourier analysis.

07 Separable filters and fast implementation

A 2D filter is separable if it can be written as the outer product of a vertical 1D filter and a horizontal 1D filter. The Gaussian is separable: \(G(x,y)=G(x)G(y)\). This matters because a direct \(k\times k\) convolution costs \(O(k^2)\) operations per pixel, while two 1D convolutions cost \(O(2k)\). For large kernels, the speedup is huge.

Separable filtering also clarifies implementation. To blur an image with a Gaussian, filter rows with a 1D Gaussian, then filter columns with the same 1D Gaussian. The result is equivalent to a full 2D Gaussian if boundary handling and numerical precision are consistent. Derivative-of-Gaussian filters can also be separated: smooth in one direction and differentiate in the other.

Other acceleration ideas include integral images for box filters, FFT-based convolution for very large kernels, GPU shaders, vectorised operations, and recursive approximations. But separability is the first trick to remember because it is mathematically clean and appears everywhere.

08 Nonlinear and edge-preserving filters: bilateral filtering as the main example

Linear smoothing averages across edges because it only sees spatial closeness. The bilateral filter adds a second condition: pixels should influence each other if they are spatially close and similar in intensity or colour. A simplified form is

The spatial Gaussian \(G_{\sigma_s}\) prefers nearby pixels. The range Gaussian \(G_{\sigma_r}\) prefers pixels with similar intensity. Across a strong edge, the intensity difference is large, so the filter refuses to average too much across the boundary. This gives smoothing inside regions while preserving edges.

Bilateral filtering is useful for denoising, detail manipulation, tone mapping, and as a conceptual ancestor of more advanced edge-aware methods. The limitation is that it can be expensive and can create cartoon-like effects if parameters are too strong. It also relies on intensity similarity, which may fail when noise is large or textures are complex.

09 Fourier domain intuition: frequency, filtering, and aliasing

The Fourier transform represents an image as a sum of sinusoids at different spatial frequencies and orientations. Low frequencies describe slow changes such as broad illumination and smooth blobs. High frequencies describe rapid changes such as edges, fine texture, and noise. Filtering in the spatial domain corresponds to multiplication in the frequency domain.

For a continuous signal the Fourier transform is an integral; for images we use the discrete Fourier transform. The key convolution theorem is

A Gaussian blur is a low-pass filter: it suppresses high frequencies. A derivative filter emphasises high frequencies. Sharpening boosts high frequencies. Aliasing happens when high frequencies masquerade as lower frequencies after sampling. This is why downsampling should be preceded by low-pass filtering. Without that blur, fine detail folds into false patterns.

Fourier analysis can feel abstract, but it explains why filters behave as they do. A box filter may look harmless in the image domain, but its frequency response has ripples that can create ringing. A Gaussian is special because it is smooth in both spatial and frequency domains.

10 Pyramids and multi-resolution processing

An image pyramid stores several versions of the same image at different resolutions. A Gaussian pyramid repeatedly blurs and downsamples. A Laplacian pyramid stores band-pass detail between levels. Pyramids are useful because many vision problems have structure at multiple scales. A face, a window, and a road sign should not be processed only at the original pixel size.

The standard downsampling step is not “take every second pixel.” It is blur, then subsample. The blur removes frequencies that the coarser grid cannot represent. The pyramid therefore connects directly to sampling theory from Chapter 2.

Pyramids make algorithms faster and more robust. Optical flow can estimate large motion at coarse levels and refine at fine levels. Template matching can search large scale changes efficiently. SIFT builds scale-space to detect features whose characteristic size is unknown. Image blending uses Laplacian pyramids to combine low-frequency illumination and high-frequency detail smoothly.

11 Geometric transformations, interpolation, and resampling

Geometric transformations move image content. Examples include translation, rotation, scaling, affine warps, perspective warps, lens undistortion, and panorama alignment. The central implementation issue is that transformed coordinates rarely land exactly on integer pixel positions. We need interpolation.

Nearest-neighbour interpolation picks the closest pixel. It is fast but blocky. Bilinear interpolation averages the four nearest pixels and is smoother. Bicubic interpolation uses a larger neighbourhood and often gives better visual quality. Lanczos and other windowed sinc methods preserve detail better but may ring. The right choice depends on task: labels should often use nearest neighbour to avoid creating invalid classes, while natural images usually use bilinear or bicubic.

Resampling also changes frequencies. Enlarging invents samples by interpolation. Shrinking must remove high frequencies first. Rotations and perspective warps can create local minification in some areas and magnification in others, which is why texture mapping systems use mipmaps and anisotropic filtering. Vision code faces the same issue even if the language is different.

12 Morphology and small structural operators

Morphological operations treat images in terms of shapes. They are especially common for binary images, masks, and segmentation cleanup. Erosion shrinks foreground regions by requiring the structuring element to fit inside the foreground. Dilation expands foreground regions if the structuring element touches them. Opening is erosion followed by dilation, useful for removing small bright objects. Closing is dilation followed by erosion, useful for filling small holes.

Although morphology is not always the headline of Szeliski-style early chapters, it is part of the practical image processing toolbox. After thresholding, masks are often noisy. Small speckles, holes, and broken components can be repaired with morphological operators. Unlike linear filters, morphology is nonlinear and shape-driven.

In CV projects, morphology often appears as glue code: clean a thresholded mask, join broken strokes, remove isolated detections, or prepare connected components. It is not glamorous, but it is frequently what makes a demo stable.

13 How Chapter 3 connects to CV lectures and feature extraction

Chapter 3 is the direct preparation for Chapter 4. Harris corners use image gradients and smoothing. SIFT uses Gaussian scale-space, Difference of Gaussians, gradient histograms, and normalisation. Canny edges use derivative-of-Gaussian ideas, non-maximum suppression, and hysteresis. HoG uses local gradient orientation histograms. Even deep CNNs begin with learned filters that often resemble edges, blobs, and colour-opponent patterns.

The practical lesson is to read every feature detector as a composition of image-processing ideas. If a detector is unstable, ask whether the image was smoothed at the right scale, whether contrast was normalised, whether gradients were computed in the right datatype, and whether resampling introduced aliasing.

For revision, organise this chapter by operator support. Point operators have zero spatial support beyond the pixel. Local filters use a finite neighbourhood. Fourier methods describe global frequency content. Pyramids create a stack of representations. Warps change coordinate systems. That taxonomy makes the toolbox easier to remember.

14 Worked revision example: cleaning and resizing a noisy image safely

Suppose you are given a low-light grayscale image of a printed page and you want to prepare it for edge detection and feature extraction. The naive approach is to immediately threshold or run Canny. The Chapter 3 approach is slower but safer: first inspect the representation, then choose operations based on the failure modes. If the image is an 8-bit JPEG, the values are probably gamma-compressed and already denoised or sharpened by the camera. If it is a RAW-derived linear image, brightness operations mean something different. The datatype should be converted to float before computing derivatives so negative gradients and values above 255 do not get clipped.

Next inspect contrast. If the page is underexposed but not saturated, a point operator or contrast stretch may help. If illumination is uneven, global histogram equalisation may over-amplify background noise, while local adaptive methods may reveal text but also exaggerate paper texture. This is the first trade-off: enhancement can make useful structure clearer, but it can also make nuisance variation stronger. A good workflow tries mild operations first and checks intermediate images instead of stacking aggressive filters blindly.

Now consider noise. If the image has Gaussian-like sensor noise, a small Gaussian blur may improve gradients. If it has isolated salt-and-pepper pixels, a median filter is better. If you want to smooth paper grain while keeping printed letter boundaries, a bilateral filter might help because it avoids averaging across strong intensity differences. But the bilateral filter needs parameters: \(\sigma_s\) for spatial range and \(\sigma_r\) for intensity similarity. Too small and nothing changes; too large and the page becomes cartoon-like.

If you must resize the page, the order matters. Downsampling before denoising can alias high-frequency texture. Downsampling without low-pass filtering can create false patterns in text strokes. A Gaussian pyramid step — blur then subsample — is safer. If you rotate or deskew the page, use inverse warping with bilinear or bicubic interpolation for the image. If you are warping a binary mask or label map, use nearest-neighbour interpolation so you do not invent fractional labels. This one distinction prevents many subtle bugs in segmentation pipelines.

Finally, connect this to Fourier intuition. The page contains low frequencies from illumination shading, mid frequencies from broad text strokes, high frequencies from sharp edges, and very high frequencies from noise and paper texture. Gaussian blur suppresses high frequencies. Sharpening boosts them. Downsampling folds them unless they are removed first. Histogram operations change amplitudes but not spatial arrangement. Geometric transforms resample the signal and may blur or ring. Once you can describe each operation in terms of what it does to frequency, locality, and intensity distribution, Chapter 3 stops being a memorised toolbox and becomes a design language.

A practical exam answer can use the same structure. State the operator class, write the core formula, mention the main parameter, and explain the trade-off. For Gaussian filtering: it is a linear local low-pass convolution, controlled by \(\sigma\), separable for efficiency, useful for denoising and scale-space, but it blurs edges. For histogram equalisation: it is a point remapping based on the cumulative histogram, useful for contrast, but it can amplify noise. For pyramids: they are repeated low-pass plus downsample representations, useful for multi-scale search, but they lose detail at coarse levels. This pattern produces answers that sound like understanding rather than a list.

One more useful habit is to separate restoration, enhancement, and measurement. Restoration tries to undo a degradation model, such as blur or noise. Enhancement tries to make an image easier to view or use, even if it is not physically faithful. Measurement tries to preserve quantities needed by later algorithms. These goals can conflict. A sharpening filter may enhance visual crispness but damage measurement by creating overshoot near edges. A denoiser may improve appearance but remove tiny features needed by a detector. A contrast stretch may help a human reader but alter the statistics expected by a trained model. When writing answers or building pipelines, state the goal before choosing the operator.

Colour images add another layer. Many filters can be applied independently to each channel, but that can shift colours if channels behave differently. Smoothing RGB channels separately is usually fine for mild denoising, but histogram equalising each channel separately can create strange colour casts. Sometimes it is better to transform to a luminance-chrominance space and process luminance only. This connects Chapter 3 back to Chapter 2: colour spaces and gamma are not just camera topics; they control whether processing operations are meaningful.

Parameter choice is also part of the method. A \(3\times3\) median filter and a \(9\times9\) median filter are not the same algorithm in practice. A Gaussian with \(\sigma=0.8\) preserves much more detail than one with \(\sigma=5\). A Canny edge detector run after heavy bilateral filtering sees a different image from one run on raw pixels. In project reports, do not only name the operator. Record kernel size, \(\sigma\), threshold, interpolation method, padding mode, colour space, and datatype. Reproducibility in image processing lives in these small details.

A final mental model is to ask whether an operator is invertible. Most useful image-processing operations throw information away. Blurring removes high frequencies. Thresholding discards grey-level detail. Quantisation reduces precision. Downsampling removes samples. Once lost, this information cannot be perfectly recovered by later processing. That is why pipeline order matters so much. Preserve the information needed by the final task until you are sure it is safe to simplify.

15 References and extra reads

The primary reference is Richard Szeliski, Computer Vision: Algorithms and Applications, 2nd edition, Chapter 3. For extra practice, read OpenCV tutorials on filtering, histogram equalisation, image pyramids, geometric transforms, and morphology. For signal-processing depth, revisit convolution, Fourier transforms, sampling theory, and linear systems from a DSP text. For visual intuition, implement each filter on the same noisy image and compare results side by side.

Good follow-up exercises are: write convolution from scratch, compare box and Gaussian blur in the Fourier domain, downsample with and without pre-blur to see aliasing, implement histogram equalisation, implement bilateral filtering on a small image, and build a Gaussian pyramid. These exercises make Chapter 3 feel concrete instead of a list of names.

FAQ Quick questions students usually ask

Why do we blur before edge detection?

Derivative filters amplify high-frequency noise. Blurring suppresses noise so the gradient responds more to meaningful structure.

Is convolution the same as correlation?

They differ by kernel flipping. For symmetric kernels they match; for derivative kernels sign or orientation can differ.

Why is Gaussian blur so common?

It is smooth, separable, isotropic, and forms a well-behaved scale-space representation.

When should I use median instead of Gaussian?

Median filtering is especially useful for impulsive salt-and-pepper noise because it rejects isolated extremes.

Why use inverse warping?

It ensures every output pixel is filled by sampling from the input. Forward warping can leave holes.

!! Exam traps, implementation gotchas, and debugging smells

++ Takeaways and chapter cheatsheet

• Image processing is controlled signal manipulation, not merely beautification.
• Point operators remap intensities without spatial context; filters use neighbourhoods.
• Convolution is the core linear local operation; the kernel defines the behaviour.
• Gaussian smoothing, separability, derivatives, and scale-space reappear throughout computer vision.
• Fourier intuition explains low-pass, high-pass, sharpening, ringing, and aliasing.
• Pyramids and geometric warps make multi-scale and alignment problems manageable.