Dense revision for UDL lectures 1-8: PCA/factor methods, autoencoders, likelihood and autoregressive models, PixelCNN family, normalizing flows and previous-year calculation traps.
Core map
| Family | Question answered | Exam hook |
|---|---|---|
| PCA / factor analysis | Which hidden axes explain variation? | covariance, eigenvectors, complexity, ICA/CCA/LLE compare |
| Autoencoder | Can a compressed code reconstruct input? | activation/loss, shapes, parameters, L2/sparsity |
| Autoregressive model | Can a hard joint be written as easy conditionals? | MADE masks, WaveNet, PixelCNN, sampling order |
| Normalizing flow | Can an invertible map turn data into simple noise? | change of variables, RealNVP, Jacobian, dequantization |
PCA and factor methods
- PCA objective choose orthonormal directions with maximum projected variance / minimum reconstruction error.
- Covariance variance is spread along one dimension; covariance is joint variation between dimensions.
- Eigenvalue explained variance; eigenvector principal direction.
- Dim reduction keep top \(k\) eigenvectors, drop low-eigenvalue components.
- Randomized PCA fast approximate top-\(k\); Incremental PCA mini-batch/partial-fit; Kernel PCA nonlinear via kernel matrix.
- ICA independent non-Gaussian sources; use case: blind source separation.
- CCA/KCCA correlated projections between two views; use case: image-text retrieval.
- LLE preserve local neighbor reconstruction weights on a manifold.
\[
\text{standard PCA: }O(Nd^2+d^3),\quad \text{randomized PCA: }O(Ndk),\quad \text{kernel PCA: }O(N^3)
\]
Autoencoders
\[
h=f(W_ex+b),\qquad \hat{x}=g(W_dh+c),\qquad L=\lVert x-\hat{x}\rVert^2
\]
- Undercomplete AE: \(\dim(h)<\dim(x)\), forced compression.
- Linear AE + linear decoder + squared loss + normalized/centered data = PCA-like subspace.
- Overcomplete AE can copy identity; regularize with sparsity, denoising, weight decay or contractive penalties.
- Binary/one-hot reconstruction: sigmoid/softmax output and BCE/CE loss. Real-valued reconstruction: linear output and MSE.
- Convolutional AE: Conv/Pool encoder, Dense or feature-map latent, decoder reconstructs image.
\[
\text{Conv params}=(k_hk_wc_{in}+1)c_{out}
\]
\[
\text{transposed conv output}=(n-1)s+k-2p
\]
Likelihood and autoregressive models
\[
\theta^*=\arg\max_\theta\sum_i\log p_\theta(x^{(i)}),\qquad
p(x_1,\ldots,x_D)=\prod_{i=1}^D p(x_i|x_{<i})
\]
- Histogram fails in high dimensions because bins explode.
- Autoregressive likelihood is exact and tractable; sampling is serial.
- MADE uses masks so output \(i\) sees only inputs \(
- WaveNet uses causal dilated convolutions for sequence/audio generation.
- PixelCNN uses masked conv over raster order; Gated PixelCNN fixes blind spot; PixelCNN++ improves output distribution; PixelSNAIL adds causal self-attention.
Normalizing flows
\[
z=f_\theta(x),\quad x=f_\theta^{-1}(z),\quad
p_X(x)=p_Z(f_\theta(x))\left|\det\frac{\partial f_\theta(x)}{\partial x}\right|
\]
- Train: maximize \(\log p_Z(z)+\log|\det J|\). Sample: draw \(z\), invert to \(x\).
- Requirements: invertible, differentiable, tractable Jacobian determinant.
- RealNVP affine coupling: \(y_a=x_a,\ y_b=x_b\odot e^{s(x_a)}+t(x_a)\); log-det is \(\sum s\).
- Masks: checkerboard and channel-wise; alternate masks so all variables transform.
- MAF: fast density, slow sample. IAF: fast sample, slow density.
- Dequantization: \(x_{\text{deq}}=x+u,\ u\sim U(0,1)\) for discrete pixels.
Exam gotchas
- Continuous random variable: \(P(X=0)=0\), even if density is high.
- Invertibility means one-to-one: \(\exp(-|x|)\) and \(x^2-x\) fail.
- For Laplace MLE, median estimates \(\mu\), mean absolute deviation estimates scale.
- For Keras Conv2D shape, "same" preserves spatial size; pooling halves it if pool size/stride 2.
- For MADE mask questions, do not draw forbidden self/future connections.