MLA Explained.

Phase 2 traced the evolution of KV cache compression: MQA reduced head count to 1, GQA settled at G=8 as a quality-memory compromise, and Flash Attention made the computation IO-efficient. But even GQA still stores full-dimensional key and value vectors — it just stores fewer of them. Nobody asked whether the vectors themselves needed to be full-dimensional.

DeepSeek-V2 asked that question. The answer — Multi-Head Latent Attention (MLA) — was published in May 2024 (arXiv:2405.04434). This is Phase 3: the centrepiece architectural innovation that makes DeepSeek-V2 and V3 possible at their scale.

MQA and GQA compressed the KV cache by reducing the number of heads. MLA compresses it by reducing the dimensionality of what's stored per head. Same number of full query heads. Dramatically smaller cached representation. 93.3% KV cache reduction vs standard MHA — with comparable or better quality.

The key insight: low-rank projection

Standard MHA stores, for each token at each layer, the key and value vectors for all H heads:

Cache[token t, layer l] = K_t ∈ R^{H × d_h}  +  V_t ∈ R^{H × d_h}
                         = 2 × H × d_h elements per token

For DeepSeek-V2 (H=128, d_h=128): 2 × 128 × 128 = 32,768 elements per token per layer. Across 60 layers at 128K context: 32,768 × 60 × 128K ≈ 240 billion elements. At BF16, that's ~480 GB — physically impossible on any single GPU.

MLA's insight: K and V are computed by linear projections from the same source — the token's hidden state x_t ∈ R^{d_model}. Because these projections are linear, they define a subspace. You don't need to store the full K and V vectors if you can store a compact representation from which K and V can be reconstructed. That compact representation is the latent.

MLA adds a down-projection that compresses x_t into a low-dimensional latent vector c^KV_t ∈ R^{d_c}, where d_c ≪ H × d_h. At inference, only c^KV_t is cached. When a new token generates a query and needs to attend to previous tokens, K and V are reconstructed from the cached latents on-the-fly via up-projections.

The four matrices of MLA

Standard MHA has two projection matrices per layer: W_K and W_V. MLA replaces them with four:

Matrix	Shape	Role	Used when
W_DKV	d_model × d_c	Down-projection — compress hidden state to KV latent	Every forward pass (input token)
W_UK	d_c × (H × d_h^C)	Up-projection — expand latent to all K heads	Every attention computation
W_UV	d_c × (H × d_h^V)	Up-projection — expand latent to all V heads	Every attention computation
W_KR	d_model × (H × d_h^R)	Separate RoPE key projection (decoupled)	Every forward pass (positional)

The down-projection W_DKV runs once when the token is processed. The up-projections W_UK and W_UV run every time any future token attends to the cached position. W_KR handles rotary positional encoding — that complication is covered in Article 3.5.

Why this works: the rank assumption

This only makes sense if there's a low-rank structure in the key-value subspace. Is there?

Empirically: yes. This isn't a DeepSeek-specific finding — low-rank approximations of weight matrices and activations have been observed broadly in large neural networks (LoRA exploits this for fine-tuning, for example). The key insight is that the actual information content relevant for attention computation occupies a much smaller subspace than the nominal dimensionality suggests.

Theoretically: the down-projection W_DKV has rank at most min(d_model, d_c) = d_c = 512. The up-projections then map this 512-dimensional space to the full K and V spaces. If the original K and V information really fits in 512 dimensions, you lose nothing. If it doesn't, you've introduced an approximation error — which manifests as quality degradation.

The DeepSeek-V2 paper reports that the quality degradation from MLA vs MHA is negligible — in fact, MLA often outperforms comparable MHA models because the compression acts as a regulariser. The latent representation learns to encode the most attention-relevant features of each token, dropping noise.

Forward pass step by step

During training and prefill (processing input tokens), for each token at position t:

Step 1 — Compute KV latent:
c^KV_t = W_DKV @ x_t          # (d_c,) = (d_c, d_model) @ (d_model,)

Step 2 — Compute Q normally:
q_t = W_Q @ x_t               # (H × d_h,) — queries unchanged

Step 3 — Expand K from latent (at attention time):
K_t = W_UK @ c^KV_t           # (H × d_h^C,)

Step 4 — Expand V from latent (at attention time):
V_t = W_UV @ c^KV_t           # (H × d_h^V,)

Step 5 — Standard attention:
scores = Q @ K^T / sqrt(d_h)  # (H, S, S)
weights = softmax(scores)
out = weights @ V             # (H, S, d_h^V)

Step 6 — Cache only the latent:
kv_cache[t] = c^KV_t          # (d_c,) NOT (H × d_h × 2)

During decode (generating new tokens), for each new token:

Step 1 — Compute Q for new token only (one row)
Step 2 — Load entire KV cache: c^KV_[0..t-1], each shape (d_c,)
Step 3 — Expand ALL cached latents to K and V via W_UK, W_UV
Step 4 — Run attention (new Q attends to all cached K, V)
Step 5 — Append new latent c^KV_t to cache
Step 6 — Discard expanded K and V (don't need them again)

The up-projection (Step 3) is extra compute that MHA doesn't have. But it's a simple matrix multiplication of shape (H × d_h^C, d_c) applied to each cached latent — and it runs on contiguous memory that loads cleanly. In practice the overhead is small relative to the memory bandwidth savings from loading a 512-dim cache instead of a 32,768-dim cache.

Memory: the concrete numbers

DeepSeek-V2 architecture (from the paper):

• d_model = 5120
• H = 128 attention heads
• d_h = 128 head dimension
• d_c = 512 KV compression dimension
• 60 transformer layers

KV cache per token per layer:

MHA:  2 × 128 × 128 = 32,768 elements
MLA:  512 elements (latent only)

Compression ratio: 32,768 / 512 = 64×
Cache reduction: (32,768 - 512) / 32,768 = 98.4%

At 128K context, BF16, batch size 1:

MHA:  32,768 × 60 × 128,000 × 2 bytes = 503 GB
MLA:  512   × 60 × 128,000 × 2 bytes = 7.9 GB

MHA at 128K context requires 503 GB of KV cache — six A100s just for the cache. MLA at the same context fits in 7.9 GB. That's what makes long-context DeepSeek-V2 serving physically possible on commodity hardware.

Note: the paper's stated "93.3% reduction" accounts for the decoupled RoPE keys (see Article 3.5) that are also cached alongside the latent. The pure latent compression is higher; the effective cache with RoPE keys is still dramatically smaller than MHA.

Parameter count: does MLA cost more weights?

MLA adds matrices that MHA doesn't have. Is there a weight cost?

Component	MHA params	MLA params
Key projection	d_model × (H × d_h) = 655,360	W_DKV: d_model × d_c = 2,621,440 W_UK: d_c × (H × d_h) = 8,388,608
Value projection	d_model × (H × d_h) = 655,360	W_UV: d_c × (H × d_h) = 8,388,608
Total KV projection	~1.3M params	~19.4M params (15× more)

MLA uses significantly more projection parameters. But this doesn't dominate the total parameter count — the MoE (Mixture of Experts) FFN layers in DeepSeek-V2 account for the vast majority of parameters, and the attention projection overhead is a small fraction of 236B total parameters.

More importantly: at inference, the extra parameters in W_UK and W_UV aren't free — they add compute to the up-projection step. But they're used on the already-compact latent vectors (512-dim inputs), so the matrix multiplications are small. The compute tradeoff is strongly worth the memory savings at long contexts.

Query compression: an optional bonus

MLA also compresses the query side during training (but not for caching, since Q is computed fresh each step). A query down-projection creates a query latent:

c^Q_t = W_DQ @ x_t           # (d_c',) where d_c' = 1536
q_t = W_UQ @ c^Q_t           # (H × d_h,) expanded at attention time

This reduces the parameter count of the query projection (W_DQ + W_UQ is fewer total parameters than full W_Q for large H). It also adds a regularisation effect. Since queries aren't cached, this doesn't affect inference memory — it's a training efficiency feature.

The low-rank structure on both sides (query compression for training efficiency, KV compression for inference memory) is what makes MLA feel like a coherent architectural choice rather than an ad-hoc patch.

Why MHA can't match this

Could you get the same effect in MHA by just using smaller head dimensions? Reduce d_h from 128 to 4, and cache the same 512 elements as MLA (128 heads × 4 dims = 512)?

No. The issue is expressiveness during attention. When computing attention scores Q @ K^T, the inner product quality depends on d_h. With d_h=4, queries and keys only have 4 dimensions to express attention patterns — far too compressed for the nuanced patterns that enable strong language model performance.

MLA avoids this by separating the storage dimension from the computation dimension. The cache stores 512-dim latents. The attention computation operates in the full (H × d_h^C)-dimensional space after up-projection. You get compact storage AND full-dimensional attention. MHA can't decouple these.

The MLA trick: the bottleneck happens at rest (storage), not at work (computation). Store small. Expand to think. Discard after.

Training stability

Adding projection layers between the hidden state and the final K/V vectors raises training stability questions. Do the gradients flow cleanly through down-projection → latent → up-projection?

The DeepSeek team reports stable training with standard AdamW and no special initialisation beyond the usual scaled initialisation for deep networks. The up-projections in particular need careful initialisation — the paper uses small normal initialisation for W_UK and W_UV to avoid exploding the latent-to-full-space expansion at the start of training.

One practical consideration: the latent vector c^KV is the only gradient path for both K and V supervision signals. This shared bottleneck could, in theory, cause the latent to specialise toward either K or V at the expense of the other. The DeepSeek paper doesn't report this as an issue — the shared compression appears to work well, and the quality results back this up.

What comes next in Phase 3

Article 3.1 has given you the conceptual picture: compress to latent on the way in, cache the latent, expand to K and V on demand, discard expanded tensors. The cache is 64× smaller per token. The attention quality is preserved.

The remaining Phase 3 articles build on this:

• 3.2 — Full PyTorch implementation. Projections, KV cache management, inference loop, with line-by-line explanation of each operation.
• 3.3 — MLA vs MQA vs GQA side-by-side. Architecture diagrams, memory comparison tables, quality vs compression tradeoff analysis.
• 3.4 — KV cache memory deep dive. Exact numbers across all variants at production-scale batch sizes and context lengths.
• 3.5 — MLA + RoPE. Why RoPE breaks inside MLA, the decoupled RoPE solution, and how it affects the cache slightly.