KV Cache Internals.

Phase 1 covered how attention works: queries, keys, values, the scaled dot-product, multi-head splitting. You can now trace a token through the full attention computation. Phase 2 starts with a problem you didn't have to think about in Phase 1 — because Phase 1 was about understanding, not deploying.

This is article 1 of 4 in Phase 2: KV Cache & Efficient Attention. The focus shifts from "how does attention work" to "how do we run it fast enough to ship."

The KV cache is the #1 memory bottleneck in LLM inference. Understanding it — exactly what's stored, why, and how much it costs — is the prerequisite for everything in Phase 2.

The decode loop problem

When an LLM generates text, it doesn't generate all tokens simultaneously. It generates one token at a time, in a loop. Each iteration takes the entire history of tokens generated so far and produces one more.

In attention terms, this means: at step t, the model computes attention between the new token (query) and all previous tokens (keys and values). At step t+1, it does the same thing, but now with one more token in the history.

The naive approach is to recompute everything from scratch on each step. For a sequence of length n, that means recomputing K and V for all n tokens, every single decode step. The total work becomes O(n²) — worse, it grows with each generated token.

The KV cache solves this with a simple observation: the keys and values for previous tokens don't change. Once computed, they can be stored and reused. Only the new token requires fresh computation.

Why K and V — but not Q?

Three matrices come out of attention: Q (queries), K (keys), V (values). We cache K and V. We never cache Q. The asymmetry is worth understanding.

In the decode loop, the query is always the current token — the one we're generating right now. It represents "what am I looking for?" and it's different for every step. Caching it would give you nothing; you'd replace it on the next step anyway.

Keys and values represent the existing context — all the tokens that came before. At decode step t, K and V encode tokens 1 through t−1. At step t+1, they encode tokens 1 through t. The entries for tokens 1 through t−1 are identical in both cases. Those don't need to be recomputed.

So: Q is fresh every step (compute it). K and V for previous tokens are unchanged (cache them, append the new one).

The memory formula

Knowing what's cached, we can calculate exactly how much memory it consumes. The formula is:

KV_bytes = 2 × L × H_kv × d_h × S × B × bytes_per_element

Where:

• 2 — one tensor for K, one for V
• L — number of transformer layers
• H_kv — number of KV heads (equals H for MHA; less for GQA/MQA)
• d_h — head dimension (typically d_model / H)
• S — sequence length (current position during decode)
• B — batch size
• bytes_per_element — 2 for BF16/FP16, 4 for FP32, 1 for FP8/INT8

Concrete numbers: LLaMA-3-70B

Let's plug in real numbers. LLaMA-3-70B specs:

• 80 transformer layers
• 8 KV heads (uses GQA, not full MHA)
• 128 head dimension
• BF16 inference (2 bytes per element)

At 4,096 token context, batch size 1:

2 × 80 × 8 × 128 × 4,096 × 1 × 2 = 1,342,177,280 bytes ≈ 1.25 GB

At 128K token context (max for LLaMA-3):

2 × 80 × 8 × 128 × 131,072 × 1 × 2 = 42,949,672,960 bytes ≈ 40 GB

At 4K context with batch size 8 (serving 8 users simultaneously):

2 × 80 × 8 × 128 × 4,096 × 8 × 2 = 10,737,418,240 bytes ≈ 10 GB

The 40 GB case is striking. LLaMA-3-70B in BF16 weighs about 140 GB. An A100 80GB can't even fit the model weights alone — you'd need at least 2 GPUs. Add 40 GB of KV cache and you need a third. That's the memory bottleneck in concrete terms.

Prefill vs decode: two very different phases

LLM inference has two distinct phases with very different compute and memory profiles.

Prefill

The model processes the entire input prompt in parallel. All tokens are present simultaneously, so Q, K, and V are computed for all of them at once. This is a large, parallel matrix operation — it's compute-bound. The GPU's tensor cores run close to peak utilization. Memory is not yet the concern; the KV cache is being built, not queried.

Prefill is fast. A 2,000-token prompt might take 200ms. The parallelism across tokens makes it efficient regardless of prompt length (up to hardware limits).

Decode

One token at a time. Each step: compute Q for the new token (a tiny matrix), load the entire KV cache from GPU memory, run attention, predict one token. Then repeat.

This is memory-bandwidth-bound. The GPU is mostly waiting for data to be loaded from HBM (high-bandwidth memory) rather than doing useful compute. Modern A100s have ~2 TB/s of HBM bandwidth — but a 40 GB KV cache still takes ~20ms to fully load. With a 50ms target for 20 tokens/second, that's most of your budget.

The arithmetic intensity (FLOPs per byte loaded) during decode is very low. You load the entire KV cache to do a relatively small amount of compute per token. This is the root cause of slow decode speeds, and it's exactly why reducing KV cache size (MQA, GQA, MLA) has such outsized impact on serving performance.

What's actually stored in memory

The KV cache is a pair of 4-dimensional tensors per layer. For a standard MHA model:

K_cache[layer]: [batch_size, num_heads, max_seq_len, head_dim]
V_cache[layer]: [batch_size, num_heads, max_seq_len, head_dim]

In practice, these are pre-allocated at the maximum sequence length. If your max context is 4,096 tokens and you're generating a 10-token response to a 50-token prompt, the cache still holds 4,096 slots — most of them empty. This avoids reallocation overhead during the decode loop but wastes memory for short sequences.

Production serving engines like vLLM use paged attention: the KV cache is divided into fixed-size pages (like OS virtual memory). Pages are allocated on demand and freed when sequences complete. This dramatically improves GPU memory utilization when serving mixed-length requests.

Data types in practice

The bytes_per_element in the formula makes a large difference:

Format	Bytes	Cache at 128K (70B)	Notes
FP32	4	~80 GB	Full precision — never used for inference
BF16	2	~40 GB	Standard for modern inference
FP16	2	~40 GB	Slightly less stable numerically
FP8	1	~20 GB	Emerging standard for KV cache compression
INT8	1	~20 GB	KV quantization with minor quality loss
INT4	0.5	~10 GB	Aggressive; visible quality degradation

FP8 KV cache quantization is increasingly common in production — halving cache size with minimal quality loss. DeepSeek-V3 uses FP8 quantization both for weights and activations.

The bottleneck hierarchy

Put it all together and you get a clear picture of where inference memory goes:

1. Model weights — fixed cost, loaded once (70B model ≈ 140 GB BF16)
2. KV cache — grows with sequence length and batch size; the variable cost
3. Activations — temporary tensors during the forward pass; much smaller

For short-context single-request inference, weights dominate. For long-context or high-throughput serving (large batch sizes), the KV cache becomes the binding constraint. At 128K context with batch size 8, you need 320 GB just for the KV cache — more than the model itself.

This is why MQA, GQA, and MLA exist. They're not attention variants. They're KV cache compression strategies expressed as changes to the attention architecture.

What comes next

Phase 1 gave you MHA: H query heads, H key heads, H value heads. The KV cache stores all H×2 head tensors per layer.

Article 2.2 introduces Multi-Query Attention (MQA): H query heads, but only 1 key head and 1 value head — shared across all queries. One line change in the architecture, H-times reduction in KV cache size. The obvious question is what this costs in quality. That's what we'll measure.