Sinusoidal Positional Encoding.

This is the encoding the original Transformer shipped with — Section 3.5 of Attention Is All You Need, 2017. It takes binary's multi-frequency idea (4.2) and makes it continuous, fixing the discreteness and unlocking the property binary couldn't reach: relative distance recoverable through a fixed linear map. Every modern scheme, RoPE included, is a descendant of this formula.

The formula

For a token at position pos and an encoding of dimension d_model, sinusoidal PE fills each pair of dimensions (2i, 2i+1) with a sine and a cosine at a frequency set by i:

PE(pos, 2i)   = sin( pos / 10000^(2i / d_model) )
PE(pos, 2i+1) = cos( pos / 10000^(2i / d_model) )

where:
  pos  = token position (0, 1, 2, ...)
  i    = dimension-pair index (0, 1, ..., d_model/2 - 1)
  d_model = embedding dimension (e.g. 512)

Define the per-pair frequency once and the rest reads cleanly:

freq_i = 1 / 10000^(2i / d_model)

PE(pos, 2i)   = sin(pos · freq_i)
PE(pos, 2i+1) = cos(pos · freq_i)

Low i → freq near 1 → fast oscillation (short wavelength), like binary's low bits. High i → freq near 1/10000 → slow oscillation (wavelength up to ~2π·10000), like binary's high bits. Same structure as the binary counter, now with smooth sines instead of square waves. The 10000 base sets the longest wavelength — geometric spacing of frequencies across the dimension index.

A worked vector

Take d_model = 8, so 4 frequency pairs. The per-pair frequencies:

i=0: freq = 1 / 10000^(0/8)  = 1 / 1      = 1.0
i=1: freq = 1 / 10000^(2/8)  = 1 / 10      = 0.1
i=2: freq = 1 / 10000^(4/8)  = 1 / 100     = 0.01
i=3: freq = 1 / 10000^(6/8)  = 1 / 1000    = 0.001

Encoding for position 1:

PE(1) = [ sin(1·1.0),   cos(1·1.0),
          sin(1·0.1),   cos(1·0.1),
          sin(1·0.01),  cos(1·0.01),
          sin(1·0.001), cos(1·0.001) ]
      = [ 0.841,  0.540,
          0.0998, 0.995,
          0.0100, 0.99995,
          0.0010, 0.9999995 ]

Note all values sit in [−1, 1] — bounded, requirement 1, no matter how large pos grows. The fast pair (i=0) moves substantially per step; the slow pairs barely budge. Adjacent positions get nearby vectors — continuous and differentiable, fixing binary's jaggedness (requirement 4).

The payoff: relative position via a linear map

This is the property binary couldn't deliver, and it's why sinusoids were chosen. Claim: for any fixed offset k, the encoding at pos + k is a linear function of the encoding at pos — and the linear map depends only on k, not on pos.

It falls straight out of the angle-addition identities:

sin(a + b) = sin(a)cos(b) + cos(a)sin(b)
cos(a + b) = cos(a)cos(b) - sin(a)sin(b)

For one frequency pair, write θ = pos·freq_i and the shift φ = k·freq_i. Then position pos+k has angle θ + φ:

PE(pos+k, 2i)   = sin(θ + φ) = sin(θ)cos(φ) + cos(θ)sin(φ)
PE(pos+k, 2i+1) = cos(θ + φ) = cos(θ)cos(φ) - sin(θ)sin(φ)

In matrix form, per pair:

[ PE(pos+k, 2i)   ]   [  cos(φ)   sin(φ) ] [ PE(pos, 2i)   ]
[ PE(pos+k, 2i+1) ] = [ -sin(φ)   cos(φ) ] [ PE(pos, 2i+1) ]

where φ = k·freq_i depends only on the offset k.

That 2×2 matrix is a rotation by angle φ. Shifting position by k rotates each frequency pair by an angle proportional to k. The model can implement "look k tokens back" as one fixed linear transform that works at every absolute position. Relative position is now linearly accessible — requirement 5, met.

Hold onto this: shifting position = rotating the encoding. Sinusoidal PE proves relative position is a rotation, but it still adds the encoding to the token embedding, so attention only recovers the rotation indirectly. RoPE (4.4) takes the rotation literally — apply it directly to Q and K — and the relative-distance property becomes exact and automatic. RoPE is sinusoidal's insight, applied at the right place.

How it plugs in

Sinusoidal PE is computed once into a fixed table and added to token embeddings at the input, before layer 1:

x_input[pos] = token_embedding[token_at_pos] + PE[pos]

Zero learned parameters — the table is deterministic. It's injected only at the bottom of the stack; every attention layer then sees position information mixed into the residual stream. This "add at input" placement is exactly what later schemes question: the signal dilutes as it flows up through layers, and it's entangled with token content. RoPE instead injects position inside each attention layer, at Q/K, every layer.

Why the 10000 base

The base controls the longest wavelength — the slowest oscillation. With base 10000 and a few hundred dimensions, the slowest pair has a wavelength of order 2π·10000 ≈ 63000 tokens, so even very distant positions get distinguishable encodings before the slowest wave repeats. The fastest pair (i=0, wavelength 2π ≈ 6.28) resolves immediate neighbours. Together the frequency ladder spans fine-grained local order to coarse long-range order. Pick the base too small and long sequences alias (distant positions collide); too large and you waste resolution. 10000 was an empirical sweet spot — and the same base reappears in RoPE, where tuning it (NTK scaling, YaRN) is the lever for context extension, covered in 4.5.

Strengths and the cracks

What sinusoidal got right, and why it's still worth knowing:

• No parameters — deterministic table, nothing to train, nothing to overfit.
• Bounded and smooth — values in [−1,1], differentiable, gradients flow.
• Relative position linearly accessible — the rotation property above.
• Defined for any position — the formula computes PE(100000) fine even if training stopped at 512.

But the cracks that motivated successors:

• Extrapolation is "defined" not "good." You can compute PE at unseen positions, but the model never trained on those activation patterns, so quality still degrades past training length — softer than integer PE's cliff, but real. Length extrapolation stays unsolved.
• Added, not applied. Because PE is added at the input, position and content share the residual stream and the signal attenuates through depth. Attention recovers relative position only implicitly via learned weights, not structurally.
• Absolute by construction. The encoding names absolute positions; relative distance is a derived convenience, not the native quantity. RoPE flips this — relative distance becomes the thing the dot product computes directly.

Property	Binary	Sinusoidal	RoPE (4.4)
Continuous / smooth	✗	✓	✓
Relative pos as rotation	✗	✓ (implicit, added)	✓ (explicit, applied)
Injected where	input	input	every attn layer, on Q/K
Native quantity	absolute	absolute	relative
Long-context extension	✗	weak	strong (NTK/YaRN)

Where this goes next

Sinusoidal PE discovered that shifting position is rotation — but spent the discovery in the wrong place, adding a rotated vector to the input. Article 4.4 keeps the rotation and moves it: instead of adding a position vector to the embedding, rotate the query and key vectors themselves by a position-dependent angle, inside attention. The dot product of two rotated vectors then depends only on their angular difference — i.e. only on relative distance — exactly, automatically, at every layer. That's RoPE, and you've now seen the math it's built on.

References

• Vaswani et al. (2017), Attention Is All You Need, §3.5 — the original sinusoidal formula and the 10000 base. arXiv:1706.03762
• Su et al. (2021), RoFormer: Enhanced Transformer with Rotary Position Embedding — recasts the rotation insight as applied-to-Q/K. arXiv:2104.09864
• Kazemnejad (2019), The Positional Encoding — clear derivation of the linear-shift property. kazemnejad.com