Formulas are not decoration. Each one below says what situation it belongs to, what the symbols mean, when to apply it, and the trap that usually costs marks. Read left to right: meaning -> when to apply -> how to compute.
PCA and component analysis
\[
\bar{x}=\frac1N\sum_{i=1}^{N}x_i
\]
| Meaning | The data center. PCA must compare points around their mean, not around zero unless data is already centered. |
|---|---|
| When to apply | Before covariance/PCA. If a problem says "center the data" or gives raw points, compute this first. |
| Symbols | \(x_i\): sample vector, \(N\): number of samples, \(\bar{x}\): sample mean vector. |
| Trap | Running PCA on uncentered data changes the first component toward the offset, not the true spread. |
\[
\Sigma=\frac1N\sum_{i=1}^{N}(x_i-\bar{x})(x_i-\bar{x})^T
\]
| Meaning | Covariance matrix. It stores how each pair of dimensions varies together. |
|---|---|
| When to apply | Use when asked to find PCA directions, eigenvectors, variance explained, or relation between dimensions. |
| Symbols | \(\Sigma\): covariance matrix. Diagonal entries are variances; off-diagonal entries are covariances. |
| Trap | Some books use \(1/(N-1)\) for sample covariance. In exam, follow the convention given; PCA direction usually unaffected by this scalar. |
\[
\Sigma u_j=\lambda_j u_j
\]
| Meaning | PCA directions are eigenvectors of covariance. \(\lambda_j\) says how much variance lies along direction \(u_j\). |
|---|---|
| When to apply | When asked "find principal components", "which PCs to keep", or "how much information is lost". |
| Symbols | \(u_j\): principal component direction, \(\lambda_j\): eigenvalue/variance captured by that component. |
| Trap | Keep largest eigenvalues first. Small eigenvalues usually mean low-information/noise directions. |
\[
h(x)=U_k^T(x-\bar{x}),\qquad \hat{x}=\bar{x}+U_kh(x)
\]
| Meaning | \(h(x)\) is the compressed PCA representation. \(\hat{x}\) reconstructs the original vector from top \(k\) components. |
|---|---|
| When to apply | Use for PCA compression/reconstruction questions. |
| Symbols | \(U_k\): matrix of top \(k\) eigenvectors as columns, \(h(x)\): low-dimensional feature vector. |
| Trap | Subtract mean before projecting; add mean back during reconstruction. |
\[
\text{explained variance ratio of PC }j=\frac{\lambda_j}{\sum_l\lambda_l}
\]
| Meaning | Fraction of total variance explained by one component. |
|---|---|
| When to apply | When deciding how many PCs to keep or interpreting scree plots. |
| Trap | Use cumulative ratio for "keep enough components to explain 95%" questions. |
Autoencoders
\[
h=f(W_ex+b),\qquad \hat{x}=g(W_dh+c)
\]
| Meaning | Encoder compresses input \(x\) into code \(h\); decoder reconstructs \(\hat{x}\). |
|---|---|
| When to apply | Any autoencoder architecture question: identify encoder, latent code, decoder, output. |
| Symbols | \(W_e,b\): encoder weights/bias. \(W_d,c\): decoder weights/bias. \(f,g\): activations. |
| Trap | Output activation \(g\) depends on data type: sigmoid/softmax for binary/one-hot; linear for real values. |
\[
L_{\text{MSE}}=\lVert x-\hat{x}\rVert^2
\]
| Meaning | Squared reconstruction error. Penalizes numeric distance between input and output. |
|---|---|
| When to apply | Real-valued inputs/outputs, linear output activation, regression-style reconstruction. |
| Trap | For binary outputs, BCE usually matches probability interpretation better than MSE. |
\[
L_{\text{BCE}}=-\sum_j[x_j\log\hat{x}_j+(1-x_j)\log(1-\hat{x}_j)]
\]
| Meaning | Binary cross-entropy. Treats each reconstructed output as a Bernoulli probability. |
|---|---|
| When to apply | Binary vector reconstruction, one-hot/binary inputs, sigmoid output nodes. |
| Symbols | \(x_j\in\{0,1\}\): target bit; \(\hat{x}_j\): predicted probability that bit is 1. |
| Trap | Include both terms unless the target value makes one term zero. If \(x_j=1\), term is \(-\log\hat{x}_j\). If \(x_j=0\), term is \(-\log(1-\hat{x}_j)\). |
\[
\theta_{t+1}=\theta_t-\eta\nabla_\theta L
\]
| Meaning | Gradient descent update: move each parameter opposite the loss gradient. |
|---|---|
| When to apply | Backprop/one-iteration autoencoder weight-update questions. |
| Symbols | \(\theta\): any weight/bias, \(\eta\): learning rate, \(\nabla_\theta L\): derivative of loss w.r.t. parameter. |
| Trap | Sign matters. If derivative is negative, subtracting it increases the weight. |
\[
J=\mathbb{E}[(X-\alpha\beta X)^2]+\lambda(\alpha^2+\beta^2)
\]
| Meaning | 1D linear autoencoder objective with L2 regularization. Reconstruction wants \(\alpha\beta\approx1\); regularization wants small weights. |
|---|---|
| When to apply | Previous-paper 1D AE problem with scalar encoder weight \(\alpha\), decoder weight \(\beta\), linear activations. |
| Symbols | \(\alpha\beta X\): reconstruction, \(\lambda\): regularization strength. |
| Trap | Use \(E[X^2]\) from the distribution. For \(X\sim U[-2,2]\), \(E[X^2]=4/3\). |
Convolution / transposed convolution
\[
H_{\text{conv}}=\left\lfloor\frac{H+2p-k}{s}\right\rfloor+1
\]
| Meaning | Output height/width after ordinary convolution or pooling. |
|---|---|
| When to apply | Keras Conv2D/MaxPooling shape questions. |
| Symbols | \(H\): input size, \(p\): padding, \(k\): kernel/pool size, \(s\): stride. |
| Trap | "same" padding with stride 1 preserves spatial size. Pooling often has stride equal to pool size. |
\[
H_{\text{tconv}}=(H-1)s-2p+k
\]
| Meaning | Output size of transposed convolution. Think "place kernels on a larger grid and add overlaps." |
|---|---|
| When to apply | Previous-paper transposed convolution question with stride 2 and valid padding. |
| Symbols | \(H\): input size, \(s\): stride, \(p\): padding, \(k\): kernel size. |
| Trap | For valid padding \(p=0\). For \(H=2,s=2,k=3\), output is \(5\). |
\[
\#\text{Conv2D params}=(k_hk_wc_{in}+1)c_{out}
\]
| Meaning | Each filter has \(k_hk_wc_{in}\) weights plus one bias. There are \(c_{out}\) filters. |
|---|---|
| When to apply | Conv layer parameter-count questions, especially Lecture 8 kernel-size change. |
| Trap | Do not forget bias. For first grayscale layer \(c_{in}=1\); for second layer \(c_{in}\) equals previous filter count. |
Likelihood and Laplace MLE
\[
\theta^*=\arg\max_\theta \sum_{i=1}^N\log p_\theta(x^{(i)})
\]
| Meaning | Maximum likelihood: choose parameters that assign high probability/density to observed data. |
|---|---|
| When to apply | Any likelihood-based model question: parameterized distribution, autoregressive model, flow model. |
| Symbols | \(\theta\): model parameters, \(x^{(i)}\): training sample, \(p_\theta\): model distribution. |
| Trap | Use log-likelihood because products of many probabilities underflow and logs turn products into sums. |
\[
p(x|\mu,b)=\frac{1}{2b}\exp\left(-\frac{|x-\mu|}{b}\right)
\]
| Meaning | Laplace distribution. It has a sharp peak at \(\mu\) and heavier tails than Gaussian. |
|---|---|
| When to apply | Previous-year MLE question with data sampled from Laplace distribution. |
| Symbols | \(\mu\): location, \(b\) or \(\sigma\): scale/spread. |
\[
\hat{\mu}=\operatorname{median}(x_i),\qquad
\hat{b}=\frac1N\sum_i|x_i-\hat{\mu}|
\]
| Meaning | Laplace MLE: best location is median; best scale is mean absolute deviation from median. |
|---|---|
| When to apply | Data list like \(\{-2,-1,0,2,3\}\) or \(\{2,1,0,-1,-3\}\). |
| Trap | Do not use mean for \(\mu\) in Laplace MLE. Mean is Gaussian instinct; median is Laplace answer. |
Autoregressive models
\[
p(x_1,\ldots,x_D)=\prod_{i=1}^D p(x_i|x_1,\ldots,x_{i-1})
\]
| Meaning | Chain rule factorization. A hard joint distribution becomes a sequence of conditional predictions. |
|---|---|
| When to apply | MADE, WaveNet, PixelCNN, any "generate one variable at a time" question. |
| Symbols | \(x_i\): variable/pixel/sample; \(x_{<i}\): all earlier variables in chosen ordering. |
| Trap | Ordering matters. For images, PixelCNN usually uses raster order; MADE uses chosen variable order. |
\[
\log p(x)=\sum_i\log p(x_i|x_{<i})
\]
| Meaning | Log-likelihood of an autoregressive model is the sum of conditional log-likelihoods. |
|---|---|
| When to apply | When explaining why training is tractable. |
| Trap | Training can be parallel because full data is known; sampling is serial because generated previous values are needed. |
\[
x'_1\sim p(x_1),\quad x'_2\sim p(x_2|x'_1),\quad x'_3\sim p(x_3|x'_1,x'_2)
\]
| Meaning | Sampling from a trained MADE for three binary variables. |
|---|---|
| When to apply | Previous-paper MADE sampling question. |
| Trap | Feed sampled earlier variables back into the network before sampling the next one. |
Flows
\[
z=f_\theta(x),\qquad x=f_\theta^{-1}(z)
\]
| Meaning | A flow is a two-way map between complicated data \(x\) and simple latent noise \(z\). |
|---|---|
| When to apply | Any normalizing-flow question: training, likelihood, sampling, invertibility. |
| Trap | If \(f\) is not invertible, it is not a valid flow layer for exact density. |
\[
p_X(x)=p_Z(f_\theta(x))\left|\det J_f(x)\right|
\]
| Meaning | Change-of-variables formula. Density changes when space is stretched or squeezed by the transform. |
|---|---|
| When to apply | Flow density derivations, transformed distribution questions, RealNVP likelihood. |
| Symbols | \(p_X\): density in data space, \(p_Z\): base density, \(J_f\): Jacobian matrix \(\partial f/\partial x\). |
| Trap | Use absolute determinant. Negative determinant still gives positive density. |
\[
\log p_X(x)=\log p_Z(z)+\log|\det J_f(x)|
\]
| Meaning | Flow training objective for one sample. Base log-density plus volume-change correction. |
|---|---|
| When to apply | When asked "how do flows train by maximum likelihood?" |
| Trap | Do not drop the log-determinant term; it is what makes transformed density normalized. |
\[
y_a=x_a,\qquad y_b=x_b\odot \exp(s(x_a))+t(x_a)
\]
| Meaning | RealNVP affine coupling. One part stays fixed; the other part is scaled and shifted using the fixed part. |
|---|---|
| When to apply | RealNVP numerical questions with 2D input \(x=[2,3]^T\) or masking questions. |
| Symbols | \(s(\cdot)\): scale network, \(t(\cdot)\): translation network, \(\odot\): elementwise multiply. |
| Trap | Only transformed part contributes to log determinant. The fixed part makes inverse easy. |
\[
\log|\det J|=\sum_j s_j(x_a)
\]
| Meaning | RealNVP Jacobian is triangular, so log determinant is just sum of scale outputs. |
|---|---|
| When to apply | When asked log determinant of RealNVP coupling layer. |
| Trap | If the layer uses \(\exp(s)\), log-det is \(s\), not \(\exp(s)\). |
\[
x=z^2,\quad z\sim U[0,1]\Rightarrow p_X(x)=\frac{1}{2\sqrt{x}},\quad 0<x\le1
\]
| Meaning | Example of density transformation. Squaring compresses/expands intervals unevenly, so density changes. |
|---|---|
| When to apply | Previous-paper flow transform question. |
| How to derive | Invert \(z=\sqrt{x}\), then \(p_X(x)=p_Z(\sqrt{x})|dz/dx|=1/(2\sqrt{x})\). |
| Trap | State support: only \(0<x\le1\). Outside that, density is 0. |
\[
x_{\text{deq}}=x+u,\qquad u\sim U(0,1)
\]
| Meaning | Dequantization turns discrete pixel integers into continuous values by adding small random noise. |
|---|---|
| When to apply | Flow models on images/discrete data. |
| Trap | Reason is not data augmentation. Reason: continuous density models behave badly on purely discrete points. |
Complexity and probability gotchas
| Formula | Meaning | When to apply | Trap |
|---|---|---|---|
| \(O(Nd^2+d^3)\) | Cost of standard PCA when \(N\ge d\): build covariance then eigendecompose. | PCA complexity question. | Some variants use \(O(\min(Nd^2,dN^2))\); mention assumption. |
| \(O(Ndk)\) | Approximate top-\(k\) randomized PCA. | Large data, \(k\ll d\). | Do not compare against full \(d\) PCs. |
| \(O(N^3)\) | Kernel PCA eigendecomposition of \(N\times N\) kernel matrix. | Kernel PCA complexity question. | Memory is \(O(N^2)\). |
| \(P(X=a)=0\) | Continuous variable has zero probability at an exact point. | Gaussian "probability of \(x=0\)" question. | Density value \(p(0)\) is not probability \(P(X=0)\). |