Phase 2 covered MQA and GQA as sequential responses to the KV cache problem. Phase 3 introduced MLA as a fundamentally different approach. This article puts them all side-by-side: architecture diagrams, memory numbers, quality tradeoffs, and the concrete reasons each approach exists.
The core architectural difference
All four variants (MHA, MQA, GQA, MLA) compute the same thing: scaled dot-product attention with multiple query heads. They differ only in how they handle the key-value side — specifically, what's stored and what's computed.
Fig 1 — What each variant stores in the KV cache per token (shown for H=8 heads, G=4 groups). Only MLA stores a compressed latent rather than full K,V vectors.
The compression dimension
Each approach compresses along a different dimension:
| Variant | Compresses what | How | Limit |
|---|---|---|---|
| MHA | Nothing | Full K,V cached | Memory baseline |
| MQA | Number of KV heads: H → 1 | All queries share 1 K,V pair | Quality degrades below MHA |
| GQA | Number of KV heads: H → G | Groups of queries share K,V | Still linear in d_h; G=1 = MQA |
| MLA | Dimensionality of KV repr: H×d_h → d_c | Low-rank projection to latent | Quality loss if d_c too small |
MQA and GQA reduce the count of KV pairs stored. MLA reduces the size of each stored representation. Different levers, different tradeoffs.
Memory comparison at scale
Using DeepSeek-V2's architecture: H=128 heads, d_h=128, d_c=512, 60 layers, BF16.
| Variant | Dims/token/layer | 4K ctx | 32K ctx | 128K ctx | vs MHA |
|---|---|---|---|---|---|
| MHA | 32,768 | 15.7 GB | 125 GB | 503 GB | — |
| GQA G=8 | 4,096 | 1.97 GB | 15.7 GB | 62.9 GB | 8× |
| MQA | 256 | 0.12 GB | 0.98 GB | 3.93 GB | 128× |
| MLA | 512 | 0.25 GB | 1.97 GB | 7.86 GB | 64× |
Three things stand out:
- MLA (512 dims) is actually slightly larger than MQA (256 dims for H=128, d_h=128, 1 head = 2×128). MLA's advantage isn't maximum compression — it's quality at high compression.
- GQA at G=8 is 8× less than MHA but still grows to 62.9 GB at 128K context — real hardware strain.
- MLA at 128K context needs just 7.86 GB — comfortably fits on a single A100 80GB alongside other allocations.
Quality comparison
Memory matters, but only if quality holds. Here's what the papers and benchmarks say:
| Variant | vs MHA quality | Where it shows | Source |
|---|---|---|---|
| MQA | −0.1 to −0.5 perplexity | Structured generation, reasoning | Shazeer 2019, GQA paper |
| GQA G=8 | Statistically indistinguishable | Standard NLP benchmarks | Ainslie et al. 2023 |
| MLA | Matches or exceeds MHA | Math, code, long-context | DeepSeek-V2 paper 2024 |
The DeepSeek-V2 results are remarkable: MLA doesn't just match MHA, it often outperforms an equivalent MHA baseline. The likely mechanism: the low-rank compression acts as a regulariser. The latent bottleneck forces the model to learn the most attention-relevant features of each token, discarding noise. This is similar to why bottleneck architectures (autoencoders, LoRA) often produce better generalisation than their full-rank counterparts.
Compute tradeoff
Memory isn't free in MLA — the savings are paid for with compute.
| Operation | MHA | GQA G=8 | MLA |
|---|---|---|---|
| KV forward (per token) | 2 matmuls (W_K, W_V) | 2 matmuls (smaller) | 1 down-project + 2 up-projects |
| Cache write | 32,768 elements | 4,096 elements | 512 elements |
| Cache read (per decode step) | 32,768 × seq_len | 4,096 × seq_len | 512 × seq_len (+ up-project) |
| Decode bottleneck | HBM bandwidth | HBM bandwidth | Up-projection compute |
MHA and GQA are both bandwidth-bound at decode — they load large K/V tensors from HBM. MLA shifts the bottleneck: it loads small latents from HBM but must compute up-projections on them. On modern GPUs (A100/H100), the up-projection is small relative to the bandwidth savings — MLA wins on total wall-clock decode latency at long sequences.
When to use which
Practical decision guidance:
| Scenario | Recommended | Why |
|---|---|---|
| Small model (<7B), short context (<8K) | MHA or GQA | KV cache isn't the bottleneck |
| Mid-size model, standard context | GQA G=8 | Industry standard, well-understood quality |
| Large model (>70B), long context (>32K) | GQA or MLA | Cache size becomes critical |
| Frontier model, max quality + long context | MLA | Best quality per cache byte; complex to implement |
| Inference-only serving, fast iteration | GQA | Ecosystem support (vLLM, TGI) fully mature |
MLA requires careful implementation and has limited off-the-shelf tooling compared to GQA. The DeepSeek team built their own inference engine around MLA's specific patterns (absorbed projections, FP8 latent quantisation). For teams using standard serving stacks, GQA remains the pragmatic choice. MLA is the right answer for teams building frontier models from scratch and willing to invest in custom serving infrastructure.
The key insight that MLA adds
MQA and GQA are both answers to the question "how few KV heads do we need?" They vary the integer G from H down to 1 and accept a quality floor.
MLA asks a different question: "how few dimensions per token do we need to reconstruct full-quality attention?" The answer is d_c — and it can be tuned independently of the number of heads or head dimension. You keep all H query heads for full query expressiveness. You keep full d_h for full attention computation. You only compress the cached representation, not the computation.
GQA trades query head diversity against memory. MLA doesn't — it keeps all query heads while compressing storage. That's why MLA matches MHA quality while GQA's quality converges to MQA's as G decreases.
Deployment reality in 2024-2025
As of mid-2025, the landscape looks like this:
- MHA: older models (GPT-3, early LLaMA), research baselines
- MQA: production specialised models (PaLM, Falcon, code completion), not new frontier models
- GQA: virtually all new open-source models — LLaMA-2/3, Mistral, Mixtral, Qwen, Gemma
- MLA: DeepSeek-V2, V3, V4 — and any frontier model following DeepSeek's architecture
GQA won the mid-tier because ecosystem support arrived quickly. MLA is winning the frontier tier because at 100K+ context with billion-scale parameters, GQA's 8× compression isn't enough — MLA's 64×+ is.