RoPE — A Visual Guide.

Article 4.3 ended on one sentence: shifting position is a rotation. Sinusoidal PE proved it but spent the insight in the wrong place — adding a rotated vector to the input. RoPE (Rotary Position Embedding, Su et al. 2021) takes the rotation literally. Don't add anything. Rotate the query and key vectors themselves by an angle proportional to their position. This article builds RoPE visually from that single move.

We met RoPE operationally in Phase 3.5 (decoupled RoPE inside MLA). This is the from-the-ground-up version: what the rotation does, why the dot product comes out depending only on relative distance, and how it's actually computed.

The core move: rotate, don't add

Take a query vector. Split it into 2D pairs of components — exactly like sinusoidal's dimension pairs. For a token at position m, rotate each pair by angle m·θ_i, where θ_i is a per-pair frequency (same geometric ladder as sinusoidal):

θ_i = 10000^(-2i/d)      # i = pair index, d = head dim

For a 2D pair (x, y) of the query at position m:

[ x' ]   [ cos(m·θ_i)   -sin(m·θ_i) ] [ x ]
[ y' ] = [ sin(m·θ_i)    cos(m·θ_i) ] [ y ]

That's it. The query at position 0 is unrotated. At position 1, every pair turns by θ_i. At position 2, by 2θ_i. Position becomes angle. The vector's length is untouched (rotations preserve magnitude) — only its direction encodes where the token sits.

Why the dot product gives relative distance — exactly

This is the whole point, and it's clean. Query at position m, key at position n. RoPE rotates the query by m·θ and the key by n·θ. The attention score is their dot product. For a single 2D pair, a rotation by angle α is R(α), and rotations have two facts we need:

R(α)ᵀ = R(-α)           # transpose of a rotation = inverse rotation
R(α)·R(β) = R(α+β)      # rotations compose by adding angles

Now expand the score between rotated query q and rotated key k:

score = (R(mθ) q) · (R(nθ) k)
      = (R(mθ) q)ᵀ (R(nθ) k)
      = qᵀ R(mθ)ᵀ R(nθ) k
      = qᵀ R(-mθ) R(nθ) k
      = qᵀ R((n-m)θ) k          ← depends ONLY on (n - m)

The absolute positions m and n cancel. What survives is R((n−m)θ) — a rotation by the relative offset. Two tokens 5 apart produce the same positional contribution whether they sit at (0,5) or (1000,1005). Relative distance isn't approximated or learned — it's structurally what the dot product computes. This is requirement 5 from 4.1, finally met exactly, and it's the property sinusoidal only reached implicitly.

Sinusoidal added a rotated vector and hoped attention would learn to read off the offset. RoPE rotates Q and K so the offset falls out of the dot product algebraically. Same trig identity, applied at the right place — and the difference between "implicit" and "exact" is the whole reason RoPE won.

Per-pair frequencies: fast and slow spins

RoPE doesn't rotate every pair by the same angle. Like sinusoidal, each pair i has its own frequency θ_i = 10000^(−2i/d):

pair 0:  θ = 1.0       → spins fast, full revolutions over short distances
pair 1:  θ = 0.1       → 10× slower
pair 2:  θ = 0.01      → 100× slower
...
pair d/2-1: θ ≈ 1e-4   → barely moves across the whole sequence

Fast pairs resolve fine local distance (is this token 1 or 2 away?); slow pairs resolve coarse global distance (same paragraph or 3000 tokens back?). The model reads relative position at multiple scales simultaneously — the multi-frequency idea from binary (4.2), now expressed as multi-rate rotation. The slowest frequencies are exactly the ones that matter for long-context extension, which is where NTK/YaRN scaling acts (4.5).

How it's actually computed

You don't build block-diagonal rotation matrices. RoPE has an efficient elementwise form. For a vector x of head dimension d at position m, precompute cos and sin tables of the per-element angles, then:

# angles: for each pair i, angle = m · θ_i, applied to dims (2i, 2i+1)
# cos[m], sin[m] are length-d vectors (each angle repeated across its pair)

def rotate_half(x):
    x1 = x[..., 0::2]      # even dims
    x2 = x[..., 1::2]      # odd dims
    return interleave(-x2, x1)   # [-x2_0, x1_0, -x2_1, x1_1, ...]

def apply_rope(x, cos, sin):
    return x * cos + rotate_half(x) * sin

# applied to Q and K right before the attention dot product:
q_rope = apply_rope(q, cos[positions], sin[positions])
k_rope = apply_rope(k, cos[positions], sin[positions])
scores = q_rope @ k_rope.transpose(-1, -2)   # already relative-position aware

Two elementwise multiplies and an add per vector — negligible cost, no parameters. Crucially, RoPE is applied to Q and K inside every attention layer, not added once at the input. Position information is re-injected at every layer and never dilutes through depth. Values (V) are left unrotated — only the score computation needs position.

Complex-number view (optional, elegant)

A 2D rotation is multiplication by a unit complex number. Pack each pair (x, y) as x + iy. RoPE at position m is then:

q_rotated = q · e^(i·m·θ)
k_rotated = k · e^(i·n·θ)

score (real part of conjugate product):
  q_rotated · conj(k_rotated) = q·conj(k) · e^(i·(m-n)·θ)
                              → phase depends only on (m - n)

Same result, one line: rotating by position and taking the dot product leaves a phase that's purely the relative offset. The complex form is why you'll see RoPE described as "applying a complex rotation" in the RoFormer paper and DeepSeek's code.

Connecting back to MLA (Phase 3.5)

Now Phase 3.5 makes full sense. MLA caches a compressed latent for K, but RoPE must rotate K by its absolute position before the dot product — and you can't bake a position-dependent rotation cleanly into a position-independent cached latent. DeepSeek's decoupled RoPE splits K into a content part (from the latent, no rotation) and a small dedicated RoPE part k^R (rotated). The relative-distance property you just derived is exactly what the k^R path preserves; the content path carries semantics. That whole design exists to keep this dot-product-gives-relative-distance behaviour intact under latent compression.

What RoPE delivers against the requirements list

• Bounded ✓ — rotations preserve magnitude; nothing grows with position.
• Unique ✓ — distinct angle per position (within the slowest wavelength).
• Length-independent ✓ — angle is m·θ, a fixed function of absolute position, never of sequence length.
• Smooth ✓ — rotation is continuous in m; gradients flow.
• Relative distance recoverable ✓ (exactly) — the dot product depends only on (n−m), structurally.
• Bonus: applied every layer, parameter-free, extrapolatable — sets up the long-context story in 4.5.

For the first time, every requirement from 4.1 is met — and the hardest one (relative distance) is met exactly, not approximately. Article 4.5 closes Phase 4 by asking the comparative question: given sinusoidal, learned, ALiBi, and RoPE all exist, why did RoPE specifically become the standard — and where it still needs help (NTK scaling, YaRN) for million-token context.

References

• Su et al. (2021), RoFormer: Enhanced Transformer with Rotary Position Embedding — the RoPE paper; rotation formulation and complex-number derivation. arXiv:2104.09864
• DeepSeek-AI (2024), DeepSeek-V2 — decoupled RoPE inside MLA. arXiv:2405.04434
• EleutherAI, Rotary Embeddings: A Relative Revolution — popular visual explainer. blog.eleuther.ai