Article 2.2 showed that MQA is a 32× KV cache reduction at the cost of noticeable quality degradation on complex tasks. The ML community needed a middle ground — something more compressed than MHA but more expressive than MQA.
Grouped-Query Attention (GQA) is that middle ground. Published by Ainslie et al. at Google Research in 2023 (arXiv:2305.13245, "GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints"), it introduced a single parameter G — the number of KV groups — that interpolates continuously between MQA and MHA. Set G=1 and you get MQA. Set G=H and you get MHA. Set G=8 and you get LLaMA-2.
GQA is now the de facto standard for production LLMs. LLaMA-2, LLaMA-3, Mistral, Mixtral, Qwen, and most 2023-2025 models use GQA with G=8. Understanding GQA is understanding how 90% of deployed open-source models handle attention.
The G parameter
In GQA, query heads are divided into G groups of H/G heads each. Each group shares a single K head and a single V head. Heads within the same group see the same keys and values; heads in different groups have independent key-value projections.
The weight matrices:
- Query:
W_Q ∈ R^{d_model × (H × d_h)}— H query heads, unchanged - Keys:
W_K ∈ R^{d_model × (G × d_h)}— G key heads instead of H - Values:
W_V ∈ R^{d_model × (G × d_h)}— G value heads instead of H
KV cache reduction: H_kv = G, so the formula from 2.1 becomes:
KV_bytes = 2 × L × G × d_h × S × B × bytes_per_element
Compression ratio vs MHA: H/G. With H=32 and G=8, compression is 32/8 = 4×. That's less dramatic than MQA's 32×, but the quality retention is much better.
Fig 1 — GQA with H=6 query heads, G=3 KV groups. Each group of 2 query heads shares one K,V pair. Compression is H/G = 2×.
The spectrum from MHA to MQA
GQA is a generalisation that subsumes both MHA and MQA as special cases:
| Variant | KV heads | Compression | Quality | Example |
|---|---|---|---|---|
| MHA | G = H (32) | 1× (none) | Full | GPT-3, original Llama |
| GQA G=8 | G = 8 | 4× | Near-MHA | LLaMA-2/3, Mistral |
| GQA G=4 | G = 4 | 8× | Slightly degraded | Some Qwen variants |
| MQA | G = 1 | H× (32×) | Noticeable regression | PaLM, Falcon |
The GQA paper ran systematic ablations across G values on T5-style encoder-decoder models. The result: quality degrades smoothly as G decreases, with a notably larger drop going from G=2 to G=1 (MQA) compared to from G=H to G=H/2. The marginal quality cost of the first compression halving is low; the last halving (to G=1) is disproportionately expensive.
G=8 specifically was chosen by the LLaMA-2 team as the sweet spot: 4× cache reduction, with quality indistinguishable from MHA on standard benchmarks including MMLU, HellaSwag, and WinoGrande.
PyTorch implementation
The implementation extends MQA: instead of broadcasting 1 head to H, you broadcast G heads to H by repeating each group's K,V for H/G heads:
def gqa_forward(x, W_Q, W_K, W_V, W_O, H, G, d_h):
B, S, D = x.shape
heads_per_group = H // G
# Q: (B, H, S, d_h)
Q = W_Q(x).reshape(B, S, H, d_h).transpose(1, 2)
# K, V: (B, G, S, d_h) → expand to (B, H, S, d_h)
K = W_K(x).reshape(B, S, G, d_h).transpose(1, 2)
V = W_V(x).reshape(B, S, G, d_h).transpose(1, 2)
# Each of G KV heads repeated for heads_per_group queries
# (B, G, S, d_h) → (B, G, 1, S, d_h) → (B, G, H/G, S, d_h) → (B, H, S, d_h)
K = K.unsqueeze(2).expand(B, G, heads_per_group, S, d_h)
K = K.reshape(B, H, S, d_h)
V = V.unsqueeze(2).expand(B, G, heads_per_group, S, d_h)
V = V.reshape(B, H, S, d_h)
# Standard attention from here (same as MHA)
scale = d_h ** -0.5
attn = (Q @ K.transpose(-2, -1)) * scale
attn = attn.softmax(dim=-1)
out = (attn @ V).transpose(1, 2).reshape(B, S, H * d_h)
return W_O(out)
The KV cache stores G heads per layer (not H, not 1). During decode, you append 1 row to each of the G cached tensors, then expand to H at attention time.
The LLaMA-2 and Mistral numbers
LLaMA-2-70B uses H=64 query heads and G=8 KV heads. Let's trace the concrete memory savings:
- d_model = 8192, d_h = 128
- 80 transformer layers
- BF16 inference
At 4K context, batch size 1:
MHA: 2 × 80 × 64 × 128 × 4096 × 1 × 2 ≈ 10.7 GB GQA: 2 × 80 × 8 × 128 × 4096 × 1 × 2 ≈ 1.3 GB (8.2× less)
At 32K context (LLaMA-2 max):
MHA: 2 × 80 × 64 × 128 × 32768 × 1 × 2 ≈ 85.9 GB (exceeds A100!) GQA: 2 × 80 × 8 × 128 × 32768 × 1 × 2 ≈ 10.7 GB
Without GQA, LLaMA-2-70B at 32K context would be physically impossible on an A100 80GB — the KV cache alone exceeds the GPU's capacity. GQA makes it work.
Mistral-7B uses H=32 query heads and G=8 KV heads. Slightly more aggressive compression (4× vs LLaMA-2's 8×) on a much smaller model. Mistral also uses sliding window attention (SWA) for even longer contexts, but GQA handles the KV budget for the attention heads that are present.
Training from scratch vs checkpoint conversion
The GQA paper demonstrated two valid approaches:
Training from scratch: Initialize W_K and W_V as G-head projections and train normally. This is what LLaMA-2 and Mistral did. The model learns to use G KV heads optimally from the start — it doesn't need to approximate H-head behaviour, it develops G-head behaviour natively.
Converting from MHA: Take a trained MHA checkpoint, and for each of the G groups, mean-pool the H/G MHA key/value heads that map to that group. Then run a short "uptrained" fine-tuning pass (5% of original training tokens) to recover quality. The GQA paper showed this achieves 99%+ quality recovery compared to training GQA from scratch.
# Conversion: MHA checkpoint (H heads) → GQA (G groups)
def convert_mha_to_gqa(W_K_mha, W_V_mha, H, G, d_h, d_model):
heads_per_group = H // G
# W_K_mha: (H * d_h, d_model)
W_K = W_K_mha.reshape(G, heads_per_group, d_h, d_model)
W_K_gqa = W_K.mean(dim=1) # (G, d_h, d_model)
W_K_gqa = W_K_gqa.reshape(G * d_h, d_model)
W_V = W_V_mha.reshape(G, heads_per_group, d_h, d_model)
W_V_gqa = W_V.mean(dim=1)
W_V_gqa = W_V_gqa.reshape(G * d_h, d_model)
return W_K_gqa, W_V_gqa
The mean-pooling preserves the average representation of the grouped heads — it's a principled way to compress rather than just discarding heads or picking one arbitrarily.
Why G=8 became the standard
G=8 appears in LLaMA-2/3, Mistral, Mixtral, and many others. It's not arbitrary — it corresponds to specific hardware realities.
A100 GPUs have 108 streaming multiprocessors (SMs). The optimal attention kernel tile sizes on CUDA hardware tend to align with powers of 2, and G=8 fits naturally into memory access patterns that maximize L2 cache reuse. G=8 also means that for H=64 query heads, each group contains exactly 8 queries — a tidy SIMD-width-aligned number.
Beyond hardware, G=8 on H=64 gives 8× compression. With LLaMA-3-70B (H=64, G=8), this means:
MHA at 128K ctx: 2 × 80 × 64 × 128 × 131072 × 1 × 2 ≈ 343 GB (impossible) GQA at 128K ctx: 2 × 80 × 8 × 128 × 131072 × 1 × 2 ≈ 43 GB (fits on 2× A100s)
GQA is what makes the 128K context window in LLaMA-3 physically achievable on accessible hardware.
GQA vs MQA: the quality argument
The GQA paper ran a careful quality comparison using the T5 1.1 XXL (11B parameters) model across translation (WMT), question answering (Natural Questions), and summarization (CNN/DailyMail). Key findings:
- GQA G=8 matched MHA quality within statistical noise on all benchmarks
- MQA (G=1) showed consistent 1-2 BLEU degradation on translation tasks
- GQA G=2 provided a partial recovery — better than MQA, not as good as G=8
- The quality loss from MHA → GQA (G=H/8) was indistinguishable in the uptrain setting
This is why the community converged on GQA rather than MQA: the quality retention at 4×-8× compression is genuine, not just a claimed approximation. Models that use G=8 do not visibly regress on downstream tasks compared to their MHA equivalents.
The sliding window interaction
Mistral-7B combines GQA with sliding window attention (SWA). SWA limits how far back each attention head can look — instead of attending to all S previous tokens, each head attends to a window of W tokens. This caps the KV cache size at W regardless of sequence length.
GQA and SWA are complementary. GQA reduces the number of KV head tensors stored per token position. SWA limits the number of token positions kept in cache. Together they let Mistral-7B handle 32K+ sequences on a single GPU that would be impractical with vanilla MHA and full context.
Where GQA falls short
GQA is still linear in sequence length. A 4× compression ratio is real, but the KV cache still grows as O(S). At 1 million context tokens (Google Gemini territory), even G=8 GQA produces a cache that's enormous.
The other limitation: all of MHA, MQA, and GQA represent the KV cache as full-precision vectors at the original embedding dimension. There's no compression of the representation itself — just reduction in the number of heads. The actual content being stored per head is unmodified.
This is the gap that DeepSeek-V2's Multi-Head Latent Attention (MLA) addresses. Instead of reducing the number of KV heads, MLA compresses the representation itself into a low-rank latent space. A 512-dimension latent replaces 128 × 16 = 2048 dimensions of full KV heads. That's the Phase 3 story — and the reason DeepSeek-V2 achieves 93% KV cache reduction while maintaining MHA-level quality that GQA at G=8 can't match for the same compression budget.
GQA chose to reduce head count. MLA chose to compress the representation. The latter is a fundamentally different approach — and it's why understanding GQA is necessary context for understanding why MLA is remarkable.
Summary: the attention compression hierarchy
Phase 2 has built up a clear picture of how the field approached KV cache compression:
- Article 2.1 — KV cache is the bottleneck: grows linearly, dominates long-context memory, bottlenecks decode bandwidth
- Article 2.2 — MQA: compress H KV heads to 1, get H× memory reduction, accept quality regression
- Article 2.3 (this) — GQA: compress H KV heads to G, get H/G× reduction, recover most quality
- Article 2.4 — Why even GQA isn't enough at scale: the O(n²) compute wall and Flash Attention's answer to it
Then Phase 3 introduces MLA — DeepSeek's architectural innovation that compresses not the head count but the head content, achieving far greater reduction ratios without the quality tradeoff that limits GQA.