MLA + RoPE.

Articles 3.1–3.4 simplified MLA by glossing over one complication: RoPE. That simplification was intentional — the core latent compression mechanism is cleanest to explain without it. But RoPE isn't a minor detail. It's why MLA has the specific architecture it does, including the extra cached tensor that brings the "64× compression" closer to "93% reduction" in practice.

This article explains what RoPE is, why it breaks naively inside MLA, and what the decoupled RoPE solution does.

RoPE: the quick version

Rotary Position Embedding (RoPE, Su et al. 2021) encodes the relative position between tokens directly into the query-key dot product. Instead of adding position embeddings to token embeddings, RoPE rotates Q and K vectors by position-dependent rotation matrices:

q_rotated_t = R(θ, t) × q_t     # Q at position t, rotated
k_rotated_s = R(θ, s) × k_s     # K at position s, rotated

attention_score(t, s) = q_rotated_t · k_rotated_s
                      = q_t · (R(θ, t)^T R(θ, s)) · k_s
                      = q_t · R(θ, t-s) · k_s  ← depends only on relative distance!

The magic: R(θ, t)^T R(θ, s) = R(θ, t-s) — rotation matrices with the same frequency compose to give the rotation for the difference in positions. This means attention scores naturally encode relative distance without any additional position embedding components. RoPE is why modern LLMs can generalise to sequences longer than their training context (with appropriate frequency scaling).

RoPE is applied to alternating pairs of Q and K dimensions. If the head dimension is d_h, RoPE operates on all d_h dimensions by treating them as d_h/2 complex numbers and rotating each by a different frequency.

Why RoPE breaks inside MLA

MLA caches the compressed latent c^KV_t, not the full K vectors. At attention time, it expands the latent to K via up-projection:

K_t = W_UK @ c^KV_t

Now the problem: where does RoPE fit? In standard MHA, you'd apply RoPE after the projection:

K_t_rope = RoPE(W_K @ x_t, t)   # standard MHA: project then rotate

In MLA, the equivalent would be:

K_t_rope = RoPE(W_UK @ c^KV_t, t)   # expand from latent, then rotate

This creates a critical problem: the RoPE rotation depends on position t. If you apply RoPE after up-projection, you need to know the position of each cached token when you expand it. That means you either:

1. Cache K_t_rope directly (full expanded + rotated K) — giving up all memory savings
2. Cache c^KV_t and re-apply RoPE at expansion time — but RoPE rotation must be applied per-position, which means expanding all cached latents differently for each position, a non-trivial compute pattern
3. Cache the latent before RoPE, expand it, then apply RoPE — but this requires storing the position alongside the latent and doing position-aware expansion

None of these cleanly preserve MLA's simplicity. The rotation mixes position into the K representation in a way that doesn't factor cleanly through the down/up projection bottleneck.

The decoupled RoPE solution

DeepSeek's solution: separate the semantic content from the positional content. Instead of applying RoPE to the main K vector (which lives in the compressed latent path), introduce a second, small key vector whose only purpose is to carry RoPE position information.

The full MLA forward pass, now with RoPE:

For each input token x_t at position t:

# Semantic content path (as before)
c^KV_t  = W_DKV @ x_t            # (d_c,) — compressed semantic KV
k^C_{t} = W_UK @ c^KV_t          # (H, d_h^C) — content key (no RoPE)
v_t     = W_UV @ c^KV_t          # (H, d_h^V) — value (no RoPE needed)

# Positional path (new)
k^R_raw_t = W_KR @ x_t           # (H, d_h^R) — position key, not yet rotated
k^R_t     = RoPE(k^R_raw_t, t)   # (H, d_h^R) — position key WITH RoPE

# Combined key: semantic content + position
K_t = concat([k^C_t, k^R_t], dim=-1)   # (H, d_h^C + d_h^R)

# Cache: semantic latent + raw (pre-RoPE) position key
# Note: we cache k^R_raw_t (not rotated), because RoPE will be re-applied
# at attention time with the correct relative position offset
kv_cache[t] = (c^KV_t, k^R_raw_t)    # (d_c + H*d_h^R,) per token

# Query side also has decoupled RoPE:
c^Q_t  = W_DQ @ x_t                  # query latent
q^C_t  = W_UQ @ c^Q_t                # (H, d_h^C) — content query
q^R_t  = RoPE(W_QR @ x_t, t)        # (H, d_h^R) — position query
Q_t    = concat([q^C_t, q^R_t], dim=-1)   # (H, d_h^C + d_h^R)

The attention score between query at position t and key at position s:

score(t, s) = Q_t · K_s
           = [q^C_t; q^R_t] · [k^C_s; k^R_s]
           = (q^C_t · k^C_s)    # content-to-content attention
           + (q^R_t · k^R_s)    # position-to-position attention (has RoPE)

The second term = RoPE(W_QR @ x_t, t) · RoPE(W_KR @ x_s, s)
               = position-aware, encodes relative distance (t-s)

The content-to-content part handles semantic similarity. The position-to-position part handles positional relationships. They add together to form the final attention score.

Why cache k^R_raw rather than k^R_rotated?

An important subtlety: what gets stored in the RoPE key cache?

You might think: cache k^R_t = RoPE(W_KR @ x_t, t) — the fully rotated key. But this creates a problem. When you load it later and compute the attention score against a new query at position t', you need:

score_position = q^R_{t'} · k^R_s
              = RoPE(W_QR @ x_{t'}, t') · RoPE(W_KR @ x_s, s)

If you stored the rotated key k^R_s = RoPE(W_KR @ x_s, s), then the position information is already embedded. The dot product naturally gives you the rotation for relative distance (t'-s). This works correctly.

However, there's a subtlety with position interpolation for context extension (YaRN, RoPE scaling). If you store the pre-RoPE key and re-apply rotation at attention time, you can use a different frequency scaling without re-processing the entire cache. This is useful for systems that extend context beyond the training maximum.

DeepSeek-V2's implementation stores the unrotated W_KR @ x_t and applies RoPE during attention computation. This costs a small amount of extra compute per decode step but enables efficient context length extension.

Memory impact of the RoPE key cache

The RoPE key cache is additional overhead on top of the semantic latent:

Semantic latent per token:     d_c = 512 elements
RoPE key per token:            H × d_h^R = 128 × 64 = 8,192 elements

Total MLA cache per token:     512 + 8,192 = 8,704 elements

vs MHA: 2 × H × d_h = 2 × 128 × 128 = 32,768 elements

Reduction: (32,768 - 8,704) / 32,768 = 73.4%

Ah — the RoPE key cache significantly increases the total cache footprint. Without it, MLA would be a 98.4% reduction. With it, it's 73.4%. The paper's stated "93.3%" likely uses a specific configuration or baseline that gives the intermediate figure — the important point is the RoPE keys add substantial overhead.

However, note that d_h^R = 64 is smaller than d_h = 128 (the full head dimension). The positional key only needs to encode relative position, not full semantic content — so a smaller dimension is sufficient. Reducing d_h^R further (say, 32) would shrink the cache at some position-encoding quality cost.

d_h^R	RoPE cache dims	Total cache	vs MHA reduction
128 (= d_h)	16,384	16,896	48.5%
64 (DeepSeek-V2)	8,192	8,704	73.4%
32	4,096	4,608	85.9%
16	2,048	2,560	92.2%
0 (no RoPE — not viable)	0	512	98.4%

The 93.3% figure from the paper likely corresponds to d_h^R=16 or a different accounting. The exact number depends on the specific configuration — what matters architecturally is that decoupled RoPE adds cache overhead, and d_h^R is a tunable hyperparameter that trades positional encoding quality against memory.

Query-side decoupled RoPE

Queries aren't cached (recomputed each step), so the query-side RoPE is simpler. But it must match the key structure:

# Query latent path
c^Q_t = W_DQ @ x_t                # (d_c',) query latent
q^C_t = W_UQ @ c^Q_t              # (H, d_h^C) content query

# Decoupled position query
q^R_t = RoPE(W_QR @ x_t, t)      # (H, d_h^R) position query

# Combined query (must match key dimensions)
Q_t = concat([q^C_t, q^R_t], dim=-1)   # (H, d_h^C + d_h^R)

The concatenated dimensions must match between Q and K: Q_t ∈ R^{H × (d_h^C + d_h^R)} and K_t ∈ R^{H × (d_h^C + d_h^R)}. This ensures the dot product is well-defined.

Why not just use ALiBi or sinusoidal PE?

ALiBi (Attention with Linear Biases) adds position-dependent biases to attention scores rather than rotating Q/K vectors. It doesn't require any changes to the cached K vectors — just adds a bias term at attention computation time, which can be computed from position indices. This is trivially compatible with MLA.

So why does DeepSeek use RoPE instead of ALiBi?

RoPE generalises better to long contexts and extrapolation beyond training length. ALiBi has a linear bias that degrades for positions far beyond training. RoPE with frequency scaling (YaRN, LongRoPE) can handle 10× or 100× context extension gracefully. At DeepSeek's target of 128K+ context, RoPE is the only practical choice — and the decoupled design is the engineering solution to making RoPE work with MLA's latent compression.

Phase 3 summary

Phase 3 covered DeepSeek's central architectural innovation:

• 3.1 — Core concept: compress to latent (d_c=512), cache the latent, expand to K/V on demand. 64× cache reduction for the semantic content.
• 3.2 — Implementation: down/up projections, cache management, absorbed projections for inference, initialisation details.
• 3.3 — Comparison: MLA compresses representation dimensionality, MQA/GQA compress head count. MLA matches MHA quality; MQA/GQA don't.
• 3.4 — Memory: production numbers at scale. MLA enables 40× more concurrent long-context requests per GPU vs GQA.
• 3.5 — RoPE: the incompatibility, decoupled RoPE solution, cache overhead, d_h^R as a tuning parameter.

Phase 4 moves to positional encoding in depth — the evolution from integer positions to sinusoidal to RoPE, and why RoPE became the modern standard. Understanding RoPE's full mechanics sets up Phase 3's MLA+RoPE combination properly, so Phase 4 is the deeper technical foundation for what you just read here.

MLA is the most architecturally significant innovation in DeepSeek. Every other Phase 3–10 topic — MoE, MTP, quantisation, distributed training — builds on the memory budget MLA unlocks. It's why a 236B parameter model can run long-context inference on commodity hardware.