Supervised learning starts with labels. Unsupervised deep learning starts with a harder question: if the data comes with no answers, what structure can the model discover by itself? The course arc is one answer in layers. PCA says "find the axes that preserve variance." Autoencoders say "learn a compressed code that can reconstruct the input." Likelihood models say "learn the probability distribution that could have generated the data." Autoregressive models make that distribution tractable by predicting one variable at a time. Flow models keep likelihood and sampling tractable by forcing the map between data and latent noise to be invertible. Same target throughout: useful hidden structure without labels.
The core problem: no labels, still structure
Suppose you are handed a folder of images, time-series, text snippets, or sensor readings. There is no class label, no target value, no teacher saying "this is right." That does not mean the data is random. Faces share pose, lighting and identity factors. Images have edges and textures. Speech has local waveform structure. Documents have topics. The job of unsupervised learning is to recover useful hidden variables from raw observations.
The phrase hidden variable is the important one. We observe \(x\), but we suspect \(x\) was produced by some smaller, cleaner, more meaningful internal state \(z\). Maybe \(z\) is a set of principal components, an autoencoder code, a noise vector passed through a generator, or a latent representation in a flow. Different methods disagree about how \(z\) should be learned, but the mental model is shared:
Fig 1 -- UDL asks for a useful latent description \(z\) of observed data \(x\), often while also learning how \(z\) could generate \(x\).
That explains why the lectures move from dimensionality reduction to generative modeling. Dimensionality reduction asks for a compact representation. Generative modeling asks for a probability distribution that can score and sample data. Deep unsupervised learning tries to do both with neural networks.
PCA and factor analysis: rotate the room
Lectures 1-2 · PCA, randomized PCA, incremental PCA, kernel PCA, ICA, CCA, LLE
PCA begins with a simple geometric annoyance. A dataset can be high-dimensional, noisy, and hard to visualize, but its real variation may lie mostly along a few directions. A cloud of 2D points may stretch mostly along one diagonal line. If you rotate your coordinate system so one axis lies along that diagonal, the first coordinate explains most of the data and the second coordinate becomes mostly small residual noise.
So PCA asks: which orthonormal directions preserve as much variance as possible? For a centered data matrix \(X\), the first principal component is the unit vector \(u_1\) that maximizes projected variance:
The answer is the top eigenvector of the covariance matrix. The next principal component is the next eigenvector, constrained to be orthogonal to the first. Eigenvalues tell how much variance each component explains. Dimensionality reduction happens when we keep the first \(k\) components and drop the rest.
Variants and limits
| Method | Main idea | When it helps |
|---|---|---|
| PCA | Linear orthogonal components ranked by variance. | Fast baseline for compression, visualization, denoising. |
| Randomized PCA | Approximate the top components using random projections. | Large matrices where exact SVD is too expensive. |
| Incremental PCA | Update components in mini-batches using partial fits. | Data too large for memory or streaming batches. |
| Kernel PCA | Map data implicitly into a higher-dimensional feature space, then do PCA. | Nonlinear manifolds where linear axes fail. |
| ICA | Find statistically independent non-Gaussian sources. | Blind source separation, such as separating mixed audio signals. |
| CCA / KCCA | Find correlated projections between two views. | Cross-modal retrieval, such as image/text matching. |
| LLE | Preserve local neighbor reconstruction weights. | Nonlinear manifold learning from local geometry. |
PCA's weakness is also its clarity: it is linear. If data lies on a curved manifold, a straight axis may not reveal the real structure. ICA relaxes a different assumption by searching for independent sources rather than maximum-variance axes. Kernel PCA and LLE move toward nonlinear geometry. Autoencoders take the next step: learn the representation with a neural network.
Autoencoders: compress by reconstructing
Lecture 3 · undercomplete, overcomplete, sparse, denoising, convolutional and deep autoencoders
An autoencoder is a neural network trained to copy its input to its output. That sounds useless until you add the bottleneck. If the hidden code \(h\) is smaller than the input \(x\), the model cannot copy every raw detail directly. To reduce reconstruction loss, it must learn the features that matter most.
The encoder maps \(x\) to a code \(h\). The decoder maps \(h\) back to \(\hat{x}\). Training minimizes the difference between \(x\) and \(\hat{x}\). The learned code can then be used for compression, features, denoising, anomaly detection, or as an initialization for other models.
Why regularization matters
An undercomplete autoencoder has \(\dim(h) < \dim(x)\), so it must compress. A linear undercomplete autoencoder with squared error and normalized inputs learns essentially the same subspace as PCA. That connection is important: deep autoencoders generalize PCA by allowing nonlinear encoders and decoders.
An overcomplete autoencoder has \(\dim(h) \ge \dim(x)\). Without constraints it can learn an identity copy and discover nothing. That is why regularized autoencoders exist:
- Sparse autoencoder: add a penalty so most hidden units stay inactive. The code becomes selective instead of blindly copying.
- Denoising autoencoder: corrupt the input and train the model to reconstruct the clean version. It must learn stable structure, not pixel-level memorization.
- Convolutional autoencoder: use convolution and pooling for image structure; the latent layer stores compact visual features.
- Deep autoencoder: stack multiple nonlinear layers to learn hierarchical features.
Loss and output activation
For real-valued reconstruction, a linear output with MSE is natural. For binary or one-hot outputs, sigmoid/softmax-style outputs with binary or categorical cross-entropy are usually better because outputs represent probabilities. This is why exam autoencoder questions often ask both: "what output activation?" and "what loss?" The right answer depends on the data type, not on habit.
Likelihood models: learn the distribution, not just the code
Lectures 4-5 · likelihood, maximum likelihood, autoregressive models
Reconstruction is useful, but a stronger target is density estimation: learn \(p_\theta(x)\), a probability distribution close to the true data distribution. If the model can compute \(p_\theta(x)\), it can score anomalies, compress data, and compare how plausible samples are. If it can also sample from \(p_\theta\), it becomes a generative model.
The primitive version is a histogram. Count how often each outcome appears, divide by the number of samples, and use those frequencies as probabilities. This works for tiny finite spaces and collapses in high dimensions. A binary 28 by 28 image has \(2^{784}\) possible configurations. No dataset covers that space.
So the course moves to parameterized distributions and maximum likelihood:
Maximum likelihood says: choose model parameters that assign high probability to the observed data. With neural networks, this becomes SGD over log likelihood. The catch is architectural: the network must define a valid probability distribution whose log probability is easy to compute and differentiate.
Autoregressive models: make a hard joint easy one factor at a time
The chain rule of probability gives a legal factorization of any joint distribution:
This is the central trick. Instead of modeling a massive joint distribution directly, model a sequence of conditional distributions. Each pixel, audio sample, or variable is predicted from previous ones. Likelihood becomes tractable because log probability is the sum of conditional log probabilities.
Sampling is simple but slow: sample \(x_1\), then \(x_2\), then \(x_3\), and so on. Training can be more parallel because all true previous values are known in the dataset.
MADE: masks turn a feedforward net into an autoregressive net
MADE, the Masked Autoencoder for Distribution Estimation, uses binary masks on network connections so output \(i\) can depend only on inputs before \(i\). The same neural network computes all conditionals, but masks enforce the ordering. Type A masks prevent a variable from seeing itself; Type B masks allow the output layer to use permitted previous variables while respecting the autoregressive rule.
WaveNet and PixelCNN: autoregression for sequences and images
WaveNet applies autoregression to audio using causal dilated convolutions. Causality prevents future samples from leaking into the current prediction. Dilation grows the receptive field exponentially without huge kernels. Residual and skip connections keep training deep stacks practical.
PixelCNN applies the same idea to images. An image is ordered by raster scan, and masked convolutions make each pixel depend only on previous pixels. Standard PixelCNN has a blind spot in its receptive field. Gated PixelCNN fixes this with vertical and horizontal stacks plus gated residual blocks. PixelCNN++ improves the output distribution by moving away from 256-way softmax toward mixture of logistics and adds architectural improvements such as downsampling. PixelSNAIL combines masked convolution with causal self-attention, so it captures both local texture and long-range context.
| Model | Key mechanism | Main tradeoff |
|---|---|---|
| MADE | Masked dense layers enforce variable ordering. | Fast likelihood training, serial sampling. |
| WaveNet | Causal dilated convolutions for audio/sample sequences. | Strong quality, slow autoregressive inference. |
| PixelCNN | Masked 2D convolutions over raster-ordered pixels. | Tractable likelihood, blind-spot issue. |
| Gated PixelCNN | Vertical/horizontal streams and gated residual blocks. | Better receptive field and expressivity. |
| PixelCNN++ | Mixture of logistics plus improved architecture. | Better continuous-ish pixel modeling. |
| PixelSNAIL | Causal self-attention plus convolution. | Long-range context, heavier computation. |
Normalizing flows: invert the generator
Lectures 6-7 · 1D flows, 2D flows, N-D flows, RealNVP, MAF, IAF, Glow, Flow++, FFJORD, dequantization
Autoregressive models score data well but sample serially and do not naturally give a simple latent representation. Flow models attack this by requiring an invertible differentiable map between data \(x\) and latent noise \(z\):
The latent distribution \(p_Z(z)\) is simple, such as a standard Gaussian or uniform distribution. The data distribution is obtained by change of variables:
This equation explains every design constraint in flows. The map must be invertible so sampling is possible. It must be differentiable so the Jacobian exists. The Jacobian determinant must be cheap enough to compute, otherwise likelihood training is too expensive.
Training, inference and sampling
- Training: map each data point \(x\) to \(z=f(x)\), compute \(\log p_Z(z)+\log|\det J_f(x)|\), maximize over the dataset.
- Inference: evaluate exact log likelihood using the same change-of-variable formula.
- Sampling: draw \(z\sim p_Z\), then compute \(x=f^{-1}(z)\).
In 1D, any monotonic differentiable function can be a flow. In high dimensions, the trick is to build invertible layers with triangular or easy Jacobians. Coupling layers, autoregressive flows, invertible \(1\times1\) convolutions, spline flows and continuous-time flows are different ways to satisfy that engineering constraint.
RealNVP and coupling layers
RealNVP splits variables into two parts. One part passes through unchanged; the other is scaled and shifted using functions of the unchanged part:
The inverse is easy:
The log determinant is just the sum of scale outputs: \(\sum s(x_a)\). Masks decide which variables are held fixed and which are transformed. Checkerboard masks work naturally for images; channel-wise masks split feature channels. Masks are alternated across layers so every variable eventually gets transformed.
MAF vs IAF
Masked Autoregressive Flow (MAF) uses an autoregressive network to parameterize a flow. It is efficient for density evaluation because all conditional parameters can be computed in parallel from \(x\), but sampling is serial. Inverse Autoregressive Flow (IAF) flips the convenience: sampling is fast because the inverse direction is parallel, while density evaluation is slower. That is why Parallel WaveNet uses a student IAF-style model to avoid the slow sample-by-sample generation of the teacher WaveNet.
Why dequantization appears in flow questions
Flows are continuous density models. Images are usually discrete integers from 0 to 255. If a continuous model is trained directly on discrete points, likelihood can become ill-behaved because probability mass is concentrated on isolated values. Dequantization adds small uniform noise, commonly \(u\sim U(0,1)\), turning discrete values into continuous ones before training. Variational dequantization learns a better noise distribution.
What the supplied problems are really testing
Lecture 8 says most midsem questions are quantitative and around autoencoders, autoregressive models, flows, PCA variants, and assignment-style CNN/autoencoder calculations. The supplied solution docs match that:
- Autoencoder output activations, reconstruction losses, backprop updates, sparsity and regularization.
- Transposed convolution and convolutional autoencoder shape/parameter calculations.
- Laplace MLE, where the median estimates location and mean absolute deviation estimates scale.
- PCA complexity and variants; ICA applications.
- MADE masking and sampling order.
- PixelCNN++ / Gated PixelCNN / PixelSNAIL architecture advantages.
- Normalizing-flow change of variables, invertibility checks, RealNVP masks and log determinants.
Sources used
Built from the local udl/ folder: Lecture 1 PDF, Lecture 2 PDF, Lecture 3 PDF, Lecture 4 and 5 PDF, Lecture 6 PDF, Lecture 7 PDF, Lecture 8 PPTX tips deck, and the EC-2 regular/makeup solution documents.