Previous-year EC-2 regular/makeup questions and Lecture 8 drill questions, worked as collapsible answers. Some original Word/PPT diagrams were embedded images; where extracted text lacks weights/node diagrams, the answer gives the exact method and states what must be read from the figure.
Autoencoders and CNN drills
Q1 -- transposed convolution feature map regular
A \(2\times2\) image \(I=\begin{bmatrix}25&60\\10&30\end{bmatrix}\) and kernel \(F=\begin{bmatrix}1&0&-1\\1&0&-1\\1&0&-1\end{bmatrix}\) are used in transposed convolution with valid padding and stride 2. Find output size and values.
Show worked answer
Output size: \((2-1)\cdot2+3=5\), so \(5\times5\).
Method: place each input value times the kernel on the output grid, shifted by stride 2, then add overlaps.
Q2 -- binary autoencoder activation, loss, update regular
Fully connected autoencoder, 2D binary input, sigmoid hidden, zero bias, \(x=(1,0)\). Choose output activation/loss, compute loss from output \((0.62,0.5)\), update \(w_1,w_3\) with \(\eta=0.25\), hidden activation \(h=0.5\).
Show worked answer
Binary reconstruction means sigmoid output and binary cross entropy.
Gradient pattern:
Substitute source values:
Q3 -- linear 1D autoencoder with L2 regular
\(Z=\alpha X\), \(Y=\beta Z\), identity activations, \(X\sim U[-2,2]\). Minimize expected squared error plus \(0.5(\alpha^2+\beta^2)\). Find \(\alpha,\beta\).
Show worked answer
\(\hat{X}=\alpha\beta X\), \(E[X^2]=4/3\).
By symmetry \(\alpha=\beta=s\). Let \(u=s^2\):
So \(u=5/8\), hence \(\alpha=\beta=\pm\sqrt{5/8}=\pm0.7905\).
Q4 -- convolutional autoencoder encoder dimensions lecture 8
Encoder: Conv2D(32,3x3,same,input 28x28x1), MaxPool(2x2), Conv2D(64,3x3,same), MaxPool(2x2), Flatten, Dense(30). Find dimensions and first-conv parameters. Explain latent role and effect of removing pooling.
Show worked answer
| Layer | Output |
|---|---|
| Input | \(28\times28\times1\) |
| Conv 32 same | \(28\times28\times32\) |
| Pool | \(14\times14\times32\) |
| Conv 64 same | \(14\times14\times64\) |
| Pool | \(7\times7\times64\) |
| Flatten | 3136 |
| Dense | 30 |
First conv params: \((3\cdot3\cdot1+1)\cdot32=320\). Latent 30 is bottleneck representation. Removing pooling increases compute and flatten size, weakens compression, may preserve detail but worsens representation compactness.
Q5 -- 3x3 kernels changed to 7x7 lecture 8
Conv2D(32) and Conv2D(64) kernels change from \(3\times3\) to \(7\times7\). Discuss trainable parameters, training time, receptive field, reconstruction accuracy.
Show worked answer
First layer: \(3\times3\): \((9\cdot1+1)32=320\); \(7\times7\): \((49\cdot1+1)32=1600\). Second layer if input channels 32: \(3\times3\): \((9\cdot32+1)64=18496\); \(7\times7\): \((49\cdot32+1)64=100416\).
Training time and memory increase. Receptive field increases. Accuracy may improve if larger context helps, but may overfit, oversmooth, and cost more compute.
PCA, probability and MLE
Q6 -- Laplace MLE regular
For Laplace data \(\{-2,-1,0,2,3\}\), estimate \((\mu,\sigma)\). Repeat for \(\{2,1,0,-1,-3\}\).
Show worked answer
Laplace MLE: \(\hat{\mu}\) is sample median; \(\hat{\sigma}=\frac1n\sum_i |x_i-\hat{\mu}|\).
First set median 0; scale \((2+1+0+2+3)/5=1.6\). Answer \((0,1.6)\).
Second set sorted \(-3,-1,0,1,2\); median 0; scale \((3+1+0+1+2)/5=1.4\). Answer \((0,1.4)\).
Q7 -- PCA complexity, ICA and Gaussian point probability makeup
Give complexity of standard PCA, randomized PCA and kernel PCA for \(N\) instances of dimension \(d\). Name one ICA application. If \(x\) is Gaussian, what is \(P(x=0)\)?
Show answer
| Method | Complexity |
|---|---|
| Standard PCA | \(O(Nd^2+d^3)\) when \(N\ge d\), or \(O(\min(Nd^2,dN^2))\) |
| Randomized PCA | \(O(Ndk)\) for top \(k\), plus lower-order terms |
| Kernel PCA | \(O(N^3)\) time and \(O(N^2)\) memory |
ICA application: blind source separation. For continuous Gaussian, \(P(x=0)=0\); density at a point is not point probability.
Autoregressive models
Q8 -- sample from MADE makeup
A trained MADE computes \(p(x_1)\), \(p(x_2|x_1)\), \(p(x_3|x_1,x_2)\). Binary variables. Show generation of \((x'_1,x'_2,x'_3)\).
Show answer
- Sample \(x'_1\sim\operatorname{Bernoulli}(p(x_1=1))\).
- Feed \([x'_1,0,0]\), sample \(x'_2\sim\operatorname{Bernoulli}(p(x_2=1|x'_1))\).
- Feed \([x'_1,x'_2,0]\), sample \(x'_3\sim\operatorname{Bernoulli}(p(x_3=1|x'_1,x'_2))\).
Output is \([x'_1,x'_2,x'_3]\).
Q9 -- MADE masks regular/makeup
Design MADE masks for \(p(x_1)p(x_2|x_1)p(x_3|x_1,x_2)\). Type B at output, Type A in earlier layers. Source figure contains node numbering.
Show method answer
Exact mask matrix needs node degrees from the figure. Method: assign input degrees \(m(x_i)=i\); assign hidden-node degrees from figure/order in \(\{1,\dots,D-1\}\); allow connections only when degree rule preserves autoregressive dependency; Type A blocks self-dependency in hidden transformations, Type B output allows permitted previous-degree connections by course convention. Draw final edges wherever mask entry is 1. Source answer for output node 1 training parameters: 3 = 2 incoming weights + 1 bias.
Q10 -- PixelCNN family papers
Explain Gated PixelCNN architectural elements and utilities. Compare PixelCNN, Gated PixelCNN, PixelCNN++ and PixelSNAIL.
Show answer
PixelCNN uses masked 2D convolutions but has blind spots. Gated PixelCNN uses vertical and horizontal stacks, gated residual blocks and skip connections to improve receptive field and expressivity. PixelCNN++ uses mixture of logistics, downsampling and architecture improvements. PixelSNAIL combines masked convolution with causal self-attention, giving stronger long-range context. Best receptive field among listed variants: PixelSNAIL.
Q11 -- Parallel WaveNet makeup
How does Parallel WaveNet alleviate original WaveNet's drawback?
Show answer
Original WaveNet samples one audio value at a time. Parallel WaveNet trains an autoregressive teacher, then distills into a parallel IAF-style student using probability density distillation / KL matching. Test-time sampling is parallel and much faster, source notes say about 1000x.
Normalizing flows
Q12 -- \(x=z^2\) flow regular
\(z\sim U[0,1]\), \(x=z^2\). Find \(p_X(x)\).
Show worked answer
Inverse: \(z=\sqrt{x}\), \(x\in[0,1]\). Since \(p_Z(z)=1\):
Q13 -- invertibility regular
Are \(x^3-x^5+2x^7\), \(\exp(-|x|)\), and \(x^2-x\) invertible?
Show answer
\(x^3-x^5+2x^7\) is strictly increasing: derivative \(x^2(3-5x^2+14x^4)\), and \(14u^2-5u+3\) has negative discriminant and positive leading coefficient. Invertible.
\(\exp(-|x|)\) is symmetric, e.g. \(f(1)=f(-1)\). Not invertible.
\(x^2-x\) is a parabola, e.g. \(f(0)=f(1)=0\). Not invertible.
Q14 -- RealNVP and masks makeup
In a 2D RealNVP coupling layer with \(x=[2,3]^T\), find \(y\) and \(\log|\det J|\). Explain masking patterns and purpose. Source figure stores actual \(s,t\) values as image.
Show method answer
For affine coupling with first component fixed:
So \(y_1=2\), \(y_2=3\exp(s(2))+t(2)\), \(\log|\det J|=s(2)\). Substitute figure values if given. RealNVP masks: checkerboard for image pixels and channel-wise for channels. Masks keep part fixed for easy inverse/Jacobian and are alternated so all variables transform.
Q15 -- dequantization makeup
How is dequantization done in flow models, and why?
Show answer
Add uniform noise to discrete data: \(x_{\text{deq}}=x+u,\ u\sim U(0,1)\). Needed because flows model continuous densities but pixels are discrete integers; direct likelihood on discrete points can become pathological. Flow++ improves with variational dequantization.