Teaching a Machine to See — Computer Vision from First Principles

A flat grid of numbers is all a camera ever hands you. From that grid we somehow want to recover shape, identity, motion and meaning. This is the story of how classical computer vision pulls that off — light and lenses turning a scene into pixels, then a careful ladder of ideas (histograms, blurring, derivatives, edges, lines, corners, descriptors) that climbs from raw intensity all the way up to features stable enough to stitch two photos into one panorama. Plain language, real math, and the intuition that glues them together.

Hold up your phone, point it at the room, and tap the screen. In the time it takes the shutter to click, something quietly absurd has happened: a three-dimensional world full of depth, motion and material has been flattened into a rectangle of numbers. That rectangle is the only thing your software ever sees. No depth. No labels. No sense of "that's a chair, that's a face." Just brightness values packed into a grid. The entire field of computer vision is the project of climbing back out of that grid toward the world that produced it.

That is a deeper problem than it sounds, and the deepest, most honest place to start is with why it is hard — because everything else in this article is, in one way or another, a trick for coping with that hardness.

Why seeing is so much harder than it looks

Vision feels effortless to us, which is exactly why it fools engineers. You glance at a cluttered desk and instantly know there's a coffee mug half-hidden behind a laptop, even though you can only see a sliver of it. You'd recognize that same mug in dim light, from above, from the side, photographed against a busy background, or as a cartoon drawing. Your brain does this so fast and so unconsciously that it's tempting to assume a computer should too. It cannot — at least not easily — and understanding why is the foundation of the whole subject.

Here is the core difficulty in one sentence: an image is the result of throwing away information, and we are trying to get that information back. When light from a 3D scene lands on a flat sensor, the third dimension — depth — is collapsed. Infinitely many different 3D scenes could have produced the exact same 2D picture. A small object up close and a large object far away can paint identical patches of pixels. A photograph of a road receding to the horizon, and a clever sidewalk chalk drawing that only "works" from one viewpoint, can be pixel-for-pixel indistinguishable. Recovering the scene from the image is what mathematicians call an inverse problem, and inverse problems are notorious because they are ill-posed: the answer is not unique, and tiny changes in the input can swing the answer wildly.

And ambiguity is only the headline act. Even when we set aside the missing third dimension, the same object can look radically different from one photo to the next. The classic list of villains:

• Viewpoint variation. Move the camera and every pixel changes, even though the object is identical. The number "2" of pixels representing a face from the front shares almost nothing, value by value, with the same face in profile.
• Illumination. Brightness values are the image, and brightness depends entirely on lighting. The same wall photographed at noon and at dusk is two completely different grids of numbers.
• Scale. The object can fill the frame or occupy twelve pixels in a corner. A detector that only knows what a bicycle looks like at one size is nearly useless.
• Occlusion. Objects hide behind other objects. We routinely recognize things from a fraction of their outline, but a naive matcher sees only "most of the pattern is missing."
• Background clutter. The thing you care about sits in a sea of other edges and textures that look, locally, just as interesting.
• Intra-class variation. "Chair" covers office chairs, beanbags, thrones and stools. No single template captures the category.

So why is the situation not hopeless? Because the world, despite all this, is full of regularities — visual cues. Surfaces are mostly smooth. Edges tend to be continuous. Light usually comes from above. Objects are opaque and connected. Texture and shading betray shape. The art of classical vision is to find features and computations that lean on these regularities: things that stay stable when the nuisance factors (viewpoint, light, scale) change, and that vary informatively when the content changes. Hold that goal — invariance to nuisance, sensitivity to content — in your head. It is the compass for the entire journey.

The one abstraction that makes everything tractable

Before we can compute anything, we need a way to write down an image. The move that unlocks all the math is deceptively simple: treat a grayscale image as a function.

This reframing is the hinge of the whole field. The instant an image becomes a function, the entire toolbox of calculus and signal processing swings open. We can talk about how fast intensity changes (derivatives — that's edges). We can average a neighbourhood (integration — that's blurring). We can ask what frequencies are present, where the function is flat, where it has peaks. A picture stops being a static object you look at and becomes a mathematical surface you can probe, slice and transform. Everything that follows is operations on \( f(x,y) \).

How an image is born: light, lenses and the grid

To recover the world from an image, it helps enormously to understand how the image was made in the first place. Image formation is the forward process — scene to pixels — and vision is the attempt to run it backward.

It begins with light. We never actually see objects; we see the light they reflect or emit. A light source pours energy onto a scene, surfaces absorb some wavelengths and bounce back others, and a fraction of those reflected rays happen to travel toward our camera. That already explains a quiet truth from the previous section: change the light and you change the image, because the image is reflected light. We can even split the intensity at a point into two factors — the illumination arriving, \( i(x,y) \), and the reflectance of the surface, \( r(x,y) \), with \( f(x,y) = i(x,y)\,r(x,y) \). Reflectance lives between 0 (total absorption, black) and 1 (total reflection, white). It is a property of the object; illumination is a property of the scene. Many vision algorithms secretly wish they could divide the illumination out and keep only the reflectance, because reflectance is what's intrinsic to the object.

The pinhole: the simplest camera that works

Now those rays need to be organized into a picture. Imagine a sealed box with a single tiny hole in one face. Light from any scene point travels in straight lines; only the one ray aimed exactly at the hole passes through and strikes the back wall. Each scene point maps to one spot on the wall, and you get a sharp image — upside down, because rays from the top of the object cross through the hole and land low, while rays from the bottom land high. This is the pinhole camera, and it is the geometric heart of every camera ever built.

The geometry gives us perspective projection, and it is wonderfully clean. Put the pinhole at the origin and let the image plane sit at distance \( f \) (the focal length) behind it. A scene point at 3D coordinates \( (X, Y, Z) \) — where \( Z \) is depth, the distance straight out from the camera — lands on the image plane at:

Stare at that \( 1/Z \). It is the source of almost everything that feels "perspective-y" about photographs. Double an object's distance and its image shrinks by half. Parallel railway tracks converge as \( Z \) grows. And — crucially — once you've divided by \( Z \), you can never recover it from a single \( (x,y) \): a point twice as far away but twice as large gives the identical image coordinate. This single equation is, mathematically, where depth goes to die. The ambiguity we agonized over in the last section is right here in the algebra.

Why we bolt a lens on the front

If a pinhole already produces a sharp image, why does every real camera carry a lens? Two reasons, and the first is brutal practicality: light. A pinhole lets through only the sliver of light aimed exactly at the hole, so the image is desperately dim and needs long exposures. Make the hole bigger to gather more light and rays from each scene point now spread across many wall positions — the image blurs. A lens resolves the dilemma: it has a wide opening that gathers lots of light, yet bends all the rays from a given scene point back to a single focused spot. More light and sharpness, at once.

The price of that gathered light is that focus is no longer automatic. A thin lens obeys the relation

where \( z_o \) is the distance from the lens to the object and \( z_i \) the distance to the in-focus image. For a fixed lens and sensor position, only objects at one particular depth focus to perfect points; everything nearer or farther blurs into little discs. That blur is not always a flaw — it is exactly the dreamy out-of-focus background of a portrait. The range of depths that look acceptably sharp is the depth of field, and it widens as you shrink the aperture (the effective opening). A small aperture behaves more like a pinhole — deep focus, but less light — while a wide aperture floods the sensor with light at the cost of a shallow focused slab. Every photographer's trade-off lives in that one equation.

From continuous light to a discrete grid: sampling and quantization

So far the image on the sensor is still a continuous thing — intensity defined at every point, varying smoothly. Computers cannot store "every point" or "any real number," so the analog image must be digitized in two independent ways.

Sampling chops up space. We lay a grid of sensor cells over the image plane and record one value per cell. Each cell becomes a pixel. The fineness of this grid is the spatial resolution — how small a detail you can still tell apart. Too coarse and fine textures, thin lines and small text dissolve into blocky mush; this is why a low-resolution photo of a page becomes unreadable.

Quantization chops up intensity. The continuous brightness at each sampled point is rounded to one of a finite set of levels, usually the range \( [0, L-1] \) with \( L = 256 \) for an 8-bit image: 0 is black, 255 is white, everything between is a shade of gray. The number of available levels is the intensity resolution. Use too few levels and smooth gradients — a clear sky, a softly lit cheek — break into visible bands, an artifact called posterization.

After both steps the image is finally what your code receives: a matrix of integers. An \( M \times N \) grid where entry \( (x,y) \) holds an intensity in \( [0, L-1] \). The smooth, infinite, light-drenched scene has become a finite table of numbers — and we are right back at the grid we started with, only now we understand the long chain of physics and approximation that produced it. Every later algorithm operates on this matrix while quietly hoping the sampling was fine enough and the quantization deep enough that the regularities of the real world survived the trip.

Reading and reshaping intensities: histograms and point transforms

Now we have the matrix. Before any fancy detection, there's a whole family of operations that look at intensities alone, ignoring where pixels sit. They're the simplest things you can do to an image, and they teach a habit of thought that pays off everywhere: summarize, then transform.

The summary tool is the histogram. Count how many pixels have each intensity value and plot those counts. A dark image piles its histogram up against the left (low values); a washed-out image crowds the right; a flat, low-contrast image bunches everything into a narrow middle spike. The shape of the histogram is a fingerprint of the image's overall appearance, and reading it is the first diagnostic any image scientist reaches for. We usually normalize it — divide each count by the total number of pixels \( MN \) — so the bars become a probability distribution \( p(r_k) \) that sums to 1. Now the histogram literally answers "if I pick a random pixel, how likely is intensity \( r_k \)?"

Point transforms: a lookup table for brightness

A point or intensity transform replaces every pixel value \( r \) with a new value \( s = T(r) \), using the same rule everywhere regardless of position. It's just a function applied through a lookup table. Despite the simplicity, the family is rich:

• Negative: \( s = (L-1) - r \). Flip black and white. This is genuinely useful — radiologists invert mammograms because faint bright details are easier for the eye to catch against a dark surround.
• Log transform: \( s = c\,\log(1 + r) \). It stretches out the dark values and squashes the bright ones, compressing an enormous dynamic range into something a display can show. The textbook case is a Fourier spectrum, where a few enormous values would otherwise drown out everything; the log makes the structure visible.
• Power-law (gamma): \( s = c\,r^{\gamma} \). The exponent \( \gamma \) bends the brightness curve. With \( \gamma < 1 \) you brighten shadows; with \( \gamma > 1 \) you deepen them. This is the math behind gamma correction: display hardware has a built-in power-law response (old CRTs had exponents around 1.8–2.5), so images are pre-warped by the inverse exponent to come out looking right. The same knob rescues detail in too-dark MRI scans or too-bright aerial photos.
• Contrast stretching: a piecewise-linear curve that takes a cramped range of input values and pulls it out to span the full \( [0, L-1] \). A foggy, low-contrast microscope image snaps into clarity. Push the curve to the extreme — a step that sends everything below a cutoff to black and above it to white — and contrast stretching becomes thresholding, turning a grayscale image into a binary mask.

Histogram equalization: letting the image redistribute itself

The transforms above need you to pick parameters. Histogram equalization is cleverer: it derives the ideal brightness curve from the image's own histogram, automatically. The goal is an image that uses the full range of grays evenly — no wasted dark end, no crammed midtones — because an evenly spread histogram generally means high, balanced contrast.

The trick is to transform intensities using the cumulative distribution. Define the CDF as the running total of the normalized histogram: \( \mathrm{cdf}(r) \) is the fraction of pixels with intensity at most \( r \). Then map each old value through

Why on earth does spreading the CDF boost contrast? Here is the intuition, and it's lovely once it clicks. The CDF climbs steeply wherever many pixels are bunched together in intensity, and climbs slowly where pixels are sparse. Because the transform's output follows the CDF's height, intensities that were crowded together get mapped to widely separated output values — the steep part of the CDF pries them apart — while the few pixels in sparsely populated ranges get squeezed closer. In effect, the algorithm spends precious output levels generously on the intensities that actually occur a lot and stingily on the ones that barely appear. The result is a histogram pushed toward uniform, and a picture where previously invisible detail in the dominant tonal range leaps out. The deeper reason it works: passing any random variable through its own CDF produces a (roughly) uniform distribution — equalization is that theorem turned into an image operation.

Smoothing and filtering: averaging with a sliding window

Point transforms ignore a pixel's neighbours. But neighbours carry most of an image's meaning — an edge, a texture, a smooth surface are all defined by how nearby pixels relate. Spatial filtering is the family of operations where each output pixel is computed from a little neighbourhood of input pixels. It is the workhorse that the rest of the article leans on constantly.

The mechanism is convolution: sweep a small grid of weights — the kernel — across the image, and at every position multiply the overlapping pixels by the weights and sum them up. That sum is the new value of the centre pixel. Change the weights and you change what the filter does. A neighbourhood plus a rule: that's all a linear filter is.

The most important job of filtering is smoothing — averaging away noise and fine variation. Three kernels you'll meet constantly:

• Box filter. Every weight is equal — a plain average over the window. Simple and fast, but crude: it treats a pixel 5 cells away as just as relevant as the one right next door, and it produces boxy, directional artifacts. Bigger box, blurrier result.
• Gaussian filter. Weights follow a bell curve, \( G(x,y) = K\,e^{-(x^2+y^2)/2\sigma^2} \): the centre pixel counts most, neighbours count less the farther they are. This matches intuition — closeness should mean influence — and produces smooth, natural blur with no directional bias. The width \( \sigma \) sets how much you smooth; a larger \( \sigma \) (and correspondingly larger kernel) blurs more aggressively. The Gaussian is the default smoother of computer vision, and we'll see it again in Canny and SIFT.
• Median filter. The odd one out — it is non-linear. Instead of a weighted average it takes the median of the neighbourhood. This is the right tool for salt-and-pepper noise (random black and white speckles): a single wild outlier can wreck an average but barely moves a median. Medians also preserve edges better than averaging, since they don't blend across a sharp boundary.

Separability: a speed trick worth knowing

A 2D Gaussian has a beautiful property: it separates into a 1D Gaussian applied across the rows followed by a 1D Gaussian applied down the columns. Instead of an \( n \times n \) kernel costing \( n^2 \) multiplications per pixel, you do two passes of \( n \) each, costing \( 2n \). For a 21×21 kernel that's 42 operations instead of 441 — an order of magnitude saved, for an identical result. Separability, plus the related fact that convolution is associative, is why blurring huge images in real time is even possible.

Edges: where intensity changes fast

We finally start extracting structure. An edge is the most basic structural event in an image: a place where intensity changes abruptly. Edges trace object boundaries, surface creases, shadows and texture borders — the lines an artist would draw. Turning a dense grid of pixels into a sparse set of edge curves is a massive, useful compression of the image's information, and almost every higher-level task (finding lines, shapes, objects) starts here.

If you picture the image as a landscape where height is intensity, edges are the cliffs. And what defines a cliff? A steep slope. Slope is the language of derivatives — which is exactly why turning the image into a function \( f(x,y) \) was such a pivotal move. To find edges, we differentiate.

Since the image is a discrete grid, we approximate derivatives by finite differences — literally subtracting neighbouring pixels. The horizontal derivative \( G_x \) compares left and right neighbours; the vertical derivative \( G_y \) compares top and bottom. Stack them and you get the gradient, a little arrow at every pixel:

The gradient is the single most important object in low-level vision, so let's unpack what it means physically. It points in the direction of steepest increase in brightness — straight uphill on our intensity landscape, which means straight across the edge from the dark side to the light side. Its length tells you how steep that climb is. So from the gradient we read two things an edge detector lives or dies by:

The magnitude is large exactly where the cliff is steep — strong edges score high. The direction points across the edge; the edge line itself runs perpendicular to the gradient. (If the brightness climbs due-east, the boundary you're standing on runs north–south.) That perpendicularity is a small fact you'll use constantly when reasoning about edges.

Sobel and friends: derivatives that resist noise

The crudest derivative just subtracts two adjacent pixels, but as warned, that is painfully noise-sensitive. The fix is to combine differencing with a little smoothing in the same kernel. The Sobel operator is the famous example — a 3×3 kernel that differences in one direction while averaging in the perpendicular direction, giving a derivative estimate that's far steadier on noisy data. Its cousin, the Prewitt operator, does the same with uniform averaging. You run the \( G_x \) kernel and the \( G_y \) kernel over every pixel, then combine them through the magnitude and direction formulas above. Smooth the image first, and the gradient map cleanly outlines real boundaries instead of glittering with noise.

Not all edges look the same

Real edges come in distinct profiles, and knowing them explains why detectors behave the way they do.

• Step edge: intensity jumps instantly from one level to another — an idealized boundary. Its first derivative is a sharp spike.
• Ramp edge: intensity drifts from one level to another over several pixels. This is what real edges actually look like, because lenses and sampling blur every true step into a ramp. Its first derivative is a broad plateau, not a spike — which is why "where exactly is the edge?" becomes a real question.
• Roof edge: intensity rises to a peak and comes back down — a thin bright line, like a wire against the sky. Its derivative goes positive then negative.

The second derivative and zero-crossings

There's a second route to edges. If the first derivative peaks at an edge, the second derivative — the derivative of the derivative — crosses zero there, swinging from positive to negative right at the edge's centre. Detecting these zero-crossings can localize edges very precisely. The classic operator is the Laplacian of Gaussian (LoG): smooth with a Gaussian, then apply the Laplacian (a second-derivative operator), then look for sign changes. The catch, visible in Fig 2: second derivatives amplify noise even more savagely than first derivatives, which is why the Gaussian smoothing in "LoG" isn't optional. Zero-crossings give clean, thin, closed contours but are touchy — a reminder that every gain in localization tends to cost you something in noise robustness.

Canny: doing edge detection properly

Gradients give you a blurry, thick, noisy edge map. Useful, but not edges in the crisp sense we want — single-pixel curves you could trace. In 1986 John Canny asked what an optimal edge detector should even mean, wrote down the goals formally, and derived a pipeline to hit them. The result is still, decades later, the default edge detector and a perfect case study in turning fuzzy desiderata into a concrete algorithm.

Canny named three goals, and they pull against each other:

1. Good detection. Find the real edges and don't get fooled by noise. (Pushes toward more smoothing.)
2. Good localization. Report the edge exactly where it truly is, not shifted. (Pushes toward less smoothing — smoothing smears positions.)
3. Single response. Mark each edge once, as a one-pixel-thick line, not a fat smear of "yes" pixels.

The genius is the pipeline that balances all three. Four stages:

Step 1 — Smooth with a Gaussian

Blur the image to knock down noise, because (as we keep insisting) the next step is differentiation and differentiation hates noise. The choice of \( \sigma \) is the detection-vs-localization dial in physical form: more blur catches fewer false edges but blurs true edge positions. Often this is fused with the next step into a single "derivative of Gaussian" filter, since smoothing then differentiating equals convolving with the derivative of the smoothing kernel — convolution's associativity saving an operation.

Step 2 — Compute gradient magnitude and direction

Run Sobel-style derivatives to get \( G_x \) and \( G_y \) at every pixel, then form the magnitude \( \sqrt{G_x^2+G_y^2} \) and direction \( \operatorname{atan2}(G_y,G_x) \). Now you have a thick map of "edge strength" plus, at each pixel, the direction pointing across the local edge. That direction is the secret ingredient the next step needs.

Step 3 — Non-maximum suppression

This is the step that delivers single response, and it's beautifully simple. The gradient magnitude is fat — an edge produces a ridge of high values several pixels wide. We want only the crest of that ridge. So for each pixel, look along its gradient direction (straight across the edge) at its two neighbours. If the pixel is not the local maximum among those three — if a neighbour across the edge is brighter in gradient — suppress it to zero. Only true ridge-tops survive. A blurry band collapses into a clean one-pixel line sitting right on the edge.

Step 4 — Double threshold and hysteresis

We still have weak responses from faint edges and leftover noise. A single threshold is a trap: set it high and you fragment real edges into dashes; set it low and you readmit noise. Canny's answer is two thresholds and a connectivity rule. Pixels above the high threshold are "strong" — definitely edges. Pixels below the low threshold are discarded. Pixels in between are "weak," kept on probation. Then comes hysteresis: a weak pixel is promoted to a real edge only if it connects, through a chain of other weak pixels, to a strong one. The intuition is that a genuine edge has at least one confidently strong point and extends as a continuous curve, whereas noise produces isolated weak specks with no strong anchor. A high-to-low threshold ratio around 2:1 to 3:1 works well in practice.

From edge pixels to lines: the Hough transform

Canny hands us a set of edge pixels. But "a bunch of edge pixels that happen to lie along a line" is not the same as knowing the line. We want the line's actual parameters — and we want them even when the edge is broken into gaps, buried in clutter, and jittered by noise. Why can't we just fit a line to the points? Because we don't know which points belong to which line, how many lines there are, or which points are junk. Fitting needs that grouping, and the grouping needs the fit — a chicken-and-egg trap.

The Hough transform escapes the trap with one brilliant idea: voting. Instead of asking "which points form a line?", let every edge point vote for all the lines it could possibly lie on, and then see which lines collect the most votes. Noise and clutter vote too, but their votes scatter inconsistently, while points on a real line all pile their votes onto the same answer.

The mechanism rests on a duality between two spaces. In image space, a line is \( y = mx + b \) and a point is a dot. Now flip your perspective: fix a single image point \( (x_0, y_0) \) and ask which lines \( (m, b) \) pass through it. The lines satisfying \( y_0 = m x_0 + b \), i.e. \( b = -x_0 m + y_0 \), form a line in the \( (m,b) \) parameter space. So:

• A point in image space becomes a line in parameter space (all lines through that point).
• A line in image space becomes a point in parameter space (its unique \( m, b \)).

And here's the payoff: if several image points lie on one real line, each draws its own line in parameter space — and all those parameter-space lines cross at a single point, the \( (m,b) \) of the shared line. Detecting a line becomes detecting an intersection, which becomes detecting a peak in a vote tally.

One practical snag: the slope \( m \) blows up to infinity for vertical lines, so the \( (m,b) \) parameterization can't represent them. The fix is the normal form of a line, using a distance and an angle instead of slope and intercept:

Here \( \rho \) is the perpendicular distance from the origin to the line and \( \theta \) is the angle that perpendicular makes with the x-axis. Both stay finite for every possible line, vertical ones included. In this parameterization, each image point traces a sinusoid through \( (\theta, \rho) \) space, and points on a common line produce sinusoids that all intersect at the line's \( (\theta, \rho) \).

The full algorithm, then: build a 2D grid of accumulator bins over \( (\theta, \rho) \); for each edge pixel, sweep \( \theta \) across its range, compute the matching \( \rho \), and add one vote to that bin; finally, scan the accumulator for peaks. Each tall peak is a line, its coordinates handing you \( \theta \) and \( \rho \) directly. Gaps in the edge don't matter — surviving points still vote for the right bin. Clutter doesn't matter much — its votes spread thin. And the very same idea generalizes: to find circles, vote in a 3D parameter space of centre-x, centre-y and radius. Voting is a general strategy for finding structure amid mess, and Hough is its cleanest incarnation.

Corners: the keypoints worth trusting

Lines are great for man-made scenes, but to match two images of the same object — to say "this spot here is that spot there" — we need points, not lines. And not just any points. The question becomes: what makes a good keypoint, a location you can find again reliably in another photo taken from a different angle?

Think about trying to pin down a location by peering through a small window. Over a flat patch of uniform colour, shift the window any direction and nothing changes — you have no idea if you've moved. You cannot localize it. Along an edge, move the window along the edge and the view is unchanged (you could be anywhere on the line), though shifting across it does change things — so an edge pins you in one direction but not the other. Now a corner: nudge the window any direction and the view changes immediately. A corner is the only kind of point that is locally unique in every direction, which is exactly what makes it findable again. That's the whole intuition behind the Harris corner detector.

Turning "changes in every direction" into math

Harris measures how much a window's contents change when shifted by a small amount \( (u,v) \), by summing the squared intensity differences over the window. Through a Taylor approximation, that change-under-shift boils down to a compact quadratic form built entirely from the image gradients we already know how to compute. The whole local behaviour is captured by a 2×2 matrix — the structure tensor — summed over the window:

This little matrix encodes the distribution of gradient directions in the patch. Its two eigenvalues \( \lambda_1, \lambda_2 \) measure how much intensity varies along the two principal directions. The three cases fall right out:

• Both eigenvalues small → barely any gradient in any direction → flat region.
• One large, one small → strong variation across one direction, little along the other → edge.
• Both large → strong variation in every direction → corner. Exactly what we want.

Computing eigenvalues at every pixel is expensive, so Harris uses a clever shortcut that needs only the determinant and trace of \( M \) (both cheap, since \( \det M = \lambda_1\lambda_2 \) and \( \operatorname{trace} M = \lambda_1 + \lambda_2 \)):

Read \( R \) like a traffic light. It is large and positive at a corner (both eigenvalues big, so the product dominates). It is large and negative at an edge (one eigenvalue dwarfs the other). It is small on flat ground. So the algorithm is: compute \( R \) everywhere, keep pixels where \( R \) exceeds a threshold, and — because strong corners light up a little cluster of pixels — take only the local maxima so each corner is reported once. The constant \( k \) is an empirical knob trading off how eagerly it fires.

Corners earn their keep because they are repeatable: the same physical corner tends to be detected across changes in viewpoint, rotation and lighting. That repeatability is the bedrock of image matching, panorama stitching, 3D reconstruction, camera calibration and visual tracking — anywhere you need to say "this point and that point are the same point in the world."

Describing local shape: Histogram of Oriented Gradients

Detecting where an interesting point sits is only half the battle. To match it — or to recognize an object — you also need to describe the neighbourhood around it in a way that survives small changes in lighting and position. That description is called a descriptor, and the Histogram of Oriented Gradients (HOG) is a classic, intuitive one. It's the engine behind the pedestrian detectors that powered a generation of "is there a person here?" systems.

The insight behind HOG is that local shape is captured by the distribution of edge directions, regardless of exactly where each edge sits. The silhouette of a standing person — head, shoulders, legs — produces a characteristic pattern of gradient orientations, and that pattern persists even if the person shifts a few pixels or the lighting dims. So instead of recording raw pixel values, HOG records "which way do the gradients point around here, and how strongly?" The recipe:

1. Gradients everywhere. Compute \( G_x \) and \( G_y \), then magnitude \( \sqrt{G_x^2+G_y^2} \) and orientation \( \operatorname{atan2}(G_y,G_x) \) at every pixel — the familiar machinery.
2. Divide into cells. Carve the image (or detection window) into small cells, typically 8×8 pixels. Cells are the unit of description.
3. Orientation histogram per cell. For each cell, build a histogram — usually 9 bins spanning 0°–180° — where every pixel votes for its orientation bin weighted by its gradient magnitude. Strong edges count more than faint ones, and votes are split smoothly between neighbouring bins so a pixel near a bin boundary doesn't get arbitrarily assigned. The result is a compact summary: "in this cell, the dominant edge directions are these." A great deal of pixel detail is thrown away; the shape signature is kept.
4. Group cells into blocks and normalize. This is the step that buys robustness to lighting. Gather cells into overlapping blocks (say 2×2 cells), concatenate their histograms, and normalize that vector to unit length (L2 norm). Because brightness and contrast scale all the gradients in a neighbourhood together, dividing by the block's overall magnitude cancels that scaling out. A person in shadow and the same person in sunlight produce nearly identical normalized descriptors. The overlap means each cell contributes to several blocks under different normalizations, adding stability.
5. Concatenate. String all the block descriptors together into one long feature vector — for a standard 64×128 detection window that famously works out to thousands of numbers. Feed that vector to a classifier (a linear SVM, classically) and you have an object detector.

HOG is robust precisely because of where it chooses to be precise and where it chooses to be vague. It's vague about exact pixel positions (it pools them into cells) and vague about absolute brightness (it normalizes that away), but precise about the pattern of edge orientations. It keeps the signal that identifies the object and discards the nuisances that don't — that invariance-to-nuisance, sensitivity-to-content compass again, made concrete.

SIFT: features that survive scale and rotation

Harris finds repeatable corners; HOG describes local shape. But both have an Achilles' heel: scale. A corner detected in a close-up photo may not register the same way when the object is small and far. Zoom in on a corner and it stops looking like a corner at all — it becomes a gentle curve. A descriptor computed at one zoom level won't match the same patch at another. To match images taken from genuinely different distances and angles — to stitch a panorama from handheld shots — we need features that are invariant to both scale and rotation. That is what David Lowe's Scale-Invariant Feature Transform (SIFT) delivers, and it is the crown jewel of the classical era.

SIFT's master idea is to stop pretending there's a single right zoom level and instead search across all scales at once. If a feature is going to be found reliably regardless of how big it appears, we must look for it in a stack of progressively blurrier versions of the image — a scale space.

Building the scale space and the Difference of Gaussians

Take the image and blur it with a Gaussian at increasing \( \sigma \), producing a stack where each level is a smoother, "more zoomed-out-looking" version of the one below. Increasing blur is mathematically how an object looks when it shrinks in the frame, so this stack simulates seeing the scene at many scales. Then comes the elegant computational shortcut. To find blobs and corners robustly across scale we'd ideally apply a Laplacian-of-Gaussian at each level — but that's expensive. Lowe's trick: simply subtract adjacent blur levels. The Difference of Gaussians,

closely approximates the scale-normalized Laplacian for free, since you already computed the blurred images. Each DoG image highlights structures at a particular scale — a band-pass filter isolating "features about this big." Stack the DoG images and you have a 3D volume indexed by position and scale.

Finding keypoints and pinning down their orientation

A SIFT keypoint is a point that is a local extremum in this 3D volume — brighter or darker in the DoG than all 26 of its neighbours (8 around it at the same scale, plus 9 above and 9 below in scale). Demanding that a point stand out across scale as well as position is what makes it scale-invariant: whatever zoom you photograph the object at, the feature will pop out at the corresponding level of someone else's scale space. Lowe then refines each candidate's location to sub-pixel, sub-scale precision and throws away unstable ones — low-contrast points that noise could erase, and points strung along edges (which, as we know, aren't well-localized in one direction).

To earn rotation invariance, each keypoint is assigned a dominant orientation. Look at the gradients in the patch around it, build a histogram of their directions weighted by magnitude, and take the peak as "this feature's reference angle." From now on everything is measured relative to that angle — so if the whole image rotates, the reference angle rotates with it and the description stays the same. The feature carries its own little compass.

The 128-dimensional descriptor

Finally, describe the patch. Around the keypoint (rotated to its reference orientation and sized to its scale) take a 16×16 region, split it into a 4×4 grid of subregions, and in each subregion build an 8-bin orientation histogram of the local gradients. That's \( 4 \times 4 \times 8 = 128 \) numbers — the famous 128-D SIFT descriptor. Normalize it for lighting and you have a vector that is, remarkably, stable across scale, rotation, modest viewpoint shifts, illumination changes and noise. It is HOG's core idea — pooled, orientation-binned gradients — but wrapped in scale and rotation invariance.

Matching features and the panorama payoff

Two photos, each reduced to a few hundred SIFT keypoints with 128-D descriptors. To align them, we match: for each descriptor in image A, find the nearest descriptor in image B by Euclidean distance in that 128-D space. Two patches that are truly the same point in the world will have nearly identical descriptors and sit close together; the nearest neighbour is the candidate match.

But raw nearest-neighbour matching is littered with false matches — repeated textures, lookalike corners. Lowe's ratio test is the simple, devastatingly effective filter: for each query descriptor, find the two nearest neighbours in B and compare their distances. If the best match is much closer than the second-best (say the ratio of distances is below ~0.7–0.8), it's distinctive — accept it. If the best and second-best are nearly tied, the feature is ambiguous (it looks like several things), so reject it. Demanding a clear winner throws out most false matches at the cost of a few true ones, and dramatically cleans up the correspondence set.

Even after the ratio test a few wrong matches survive, and they'd wreck any naive fit. The classic cleanup is RANSAC (Random Sample Consensus): repeatedly pick the minimal random subset of matches needed to fit the geometric transform between the images, count how many other matches agree with that transform (the inliers), and keep the model with the most agreement. Good matches all agree on the same transform; bad matches disagree in their own random ways — "all good matches are alike; every bad match is bad in its own way." With the outliers gone, a final least-squares fit over the inliers gives a precise alignment.

And there's the payoff that ties the whole arc together: feed those clean correspondences into an alignment step and you can warp one photo onto the other and blend them — a seamless panorama stitched from handheld shots taken at slightly different angles, zooms and exposures. The same pipeline — detect features, describe them, match, fit a robust model — underpins 3D reconstruction from photos, object recognition, image retrieval, augmented-reality tracking and the visual SLAM that lets a robot or phone map a room while locating itself inside it. Every one of those applications is just a different thing to do with reliable point correspondences.

Why the old pipeline still matters

Step back and look at the staircase we climbed. A scene of light became a grid of numbers through projection, lenses, sampling and quantization. We summarized those numbers with histograms and reshaped them with point transforms. We smoothed to fight noise, then differentiated to find edges, then disciplined those edges into clean curves with Canny, then voted them into lines with Hough. We found corners that are locally unique, described their neighbourhoods with oriented-gradient histograms, and finally built features robust to scale and rotation that let us match across wildly different photographs. Each rung exists to fix a weakness of the rung below, and the whole ladder is held together by one obsession: find what stays stable when the nuisances change.

It is tempting to think deep learning made all of this obsolete. It did not — it absorbed it. When you peer inside a trained convolutional network, the first layers learn filters that look uncannily like Gaussian smoothers and Sobel-style edge detectors. Deeper layers compose those into corner- and texture-sensitive units, then into part detectors, then into object detectors — the same low-to-high progression from edges to features to semantics that we built by hand here. The network rediscovers, through data, the very hierarchy that decades of careful engineering arrived at through reasoning. So understanding the classical pipeline is not nostalgia. It is the clearest available map of what a vision system has to compute and why — the intuition that makes the black box legible. Learn to see the way these algorithms see, and the modern ones stop looking like magic and start looking like a faster, learned version of the same good ideas.

References & extra reads

• Richard Szeliski, Computer Vision: Algorithms and Applications — the standard modern reference, free online.
• Rafael C. Gonzalez & Richard E. Woods, Digital Image Processing — histograms, intensity transforms, spatial filtering, Canny and Hough, worked in detail.
• Milan Sonka, Vaclav Hlavac, Roger Boyle, Image Processing, Analysis, and Machine Vision.
• John Canny, "A Computational Approach to Edge Detection," IEEE PAMI, 1986 — the original optimal-edge-detector derivation.
• Chris Harris & Mike Stephens, "A Combined Corner and Edge Detector," 1988.
• Jianbo Shi & Carlo Tomasi, "Good Features to Track," CVPR 1994.
• Navneet Dalal & Bill Triggs, "Histograms of Oriented Gradients for Human Detection," CVPR 2005.
• David G. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints," International Journal of Computer Vision, 2004.