// the one-minute version
A neural network composes parameterized transformations and learns parameters by differentiating a task loss. Backpropagation is repeated chain rule; stochastic optimizers estimate its gradient from minibatches. Convolution shares local filters across position, encoding translation structure and reducing parameters. Nonlinearities, normalization, pooling, residual connections, attention, and multi-scale paths determine information flow. Generalization depends on data, augmentation, regularization, pretraining, and evaluation—not architecture alone. A network can minimize loss by exploiting background, annotation, or camera shortcuts, so controlled splits, ablations, calibration, and failure slices are mandatory.
Mira replaces hand-built curb features with a large network. Validation accuracy jumps, but a new camera makes predictions collapse. The network learned the laboratory’s color pipeline as faithfully as it learned curb shape. Deep learning automates representation choice; it does not automate scientific judgment.
01 A network is a differentiable program
A layer computes z=W x+b and applies a nonlinearity such as ReLU. Composing layers creates functions whose intermediate activations are learned representations. The last layer maps them to logits, coordinates, masks, depth, or another task output.
02 Loss defines what the network learns
Cross-entropy trains class probabilities; regression may use L1, L2, Huber, or geometric residuals; segmentation sums or reweights pixel losses. The empirical objective adds regularization to average training loss. Class imbalance, missing labels, and noisy annotation alter the optimum.
03 Backpropagation is the chain rule organized efficiently
The forward pass stores values; the backward pass propagates vector–Jacobian products from loss to every parameter. Automatic differentiation handles bookkeeping, but gradient checking on a tiny case still catches incorrect custom operations.
Backprop says how to change parameters locally; it does not promise a global optimum or meaningful features.
04 Convolution encodes image structure
A CNN applies shared kernels at every location. Local receptive fields exploit nearby dependence; sharing gives translation equivariance. Stride and pooling reduce resolution and expand receptive fields but can discard small objects. Padding changes boundary behavior.
Deeper layers combine edges into textures, parts, and context, but this hierarchy is task- and data-dependent, not guaranteed semantics.
05 Optimization is a dynamical system
SGD uses noisy minibatch gradients; momentum averages directions; Adam rescales coordinates adaptively. Learning-rate schedules often matter more than optimizer brand. Initialization controls signal variance. Batch normalization stabilizes activation statistics; layer and group normalization avoid batch dependence.
Vanishing and exploding gradients motivate careful nonlinearities and residual connections, which learn corrections around identity paths.
06 Architectures express resolution and context choices
Classification networks compress spatial maps. Fully convolutional encoder–decoders recover dense outputs; skip connections restore fine location. Feature pyramids combine semantic coarse features with detailed fine ones. Attention permits content-dependent long-range interaction; transformers trade convolution’s locality for learned token relations.
07 Data augmentation teaches invariance
Crops, flips, color changes, blur, and geometric transforms create examples while asserting label preservation. An invalid augmentation teaches contradictions—for example flipping text or altering a clinically meaningful color. Mixup and synthetic data change the training distribution and require validation on real deployment conditions.
08 Transfer learning reuses representation
Pretraining on a large labelled or self-supervised corpus provides features; fine-tuning adapts them. Freeze–then-unfreeze protects small datasets. Domain mismatch can cause negative transfer. Self-supervised objectives learn invariance from views, masking, or prediction without task labels, but their pretext assumptions still matter.
09 Generalization must be attacked experimentally
Use independent scene, subject, or device splits; prevent adjacent frames from crossing splits; compare parameter-matched baselines; report compute and seeds. Calibration asks whether 80% confidence is correct about 80% of the time. Out-of-distribution and corruption tests reveal brittle shortcuts.
10 Questions
Why does weight sharing help?
The same visual pattern can appear anywhere, so sharing reduces parameters and builds translation equivariance.
Why use logits with cross-entropy?
A stable log-softmax combines normalization and log likelihood without explicitly forming tiny probabilities.
What does residual learning change?
It creates identity gradient paths and asks layers to learn corrections, making deep optimization easier.
Does more data always fix shortcuts?
Only if added data breaks the shortcut’s correlation. More samples from the same biased process reinforce it.
How do I prove augmentation helps?
Ablate it under fixed training budget, evaluate clean and shifted conditions, and verify the transform preserves labels.
11 Summary and lab
- Networks learn representations by differentiated task loss.
- Convolution builds local sharing; architectures control scale and context.
- Optimization and normalization shape trainability.
- Augmentation and pretraining encode prior knowledge.
- Splits and stress tests decide whether apparent generalization is real.
12 Source trail
Original companion notes following the official book page and Springer’s chapter record.