Phase 3 ended on RoPE — the positional scheme baked into DeepSeek, LLaMA, and Mistral. But RoPE didn't appear from nowhere. It's the last step in a chain of failed-and-fixed ideas, each one patching the previous one's weakness. Phase 4 walks that chain from the start so RoPE feels inevitable rather than arbitrary.
This article is step zero: the most obvious way to tell a Transformer where each word sits — tag each token with its integer position. It's the idea everyone reaches for first. It also fails in instructive ways, and understanding exactly why it fails defines every requirement the later schemes must satisfy.
Why position needs encoding at all
Start with the problem. A Transformer processes all tokens in parallel through self-attention. Attention is a weighted sum over value vectors — and a sum doesn't care about order. Swap two tokens in the input and the attention output for the rest is identical. The mechanism is permutation-equivariant: shuffle the inputs, the outputs shuffle the same way, but no information about the original order survives.
That's fatal for language. "dog bites man" and "man bites dog" contain the same tokens. To the raw attention block they're indistinguishable. RNNs never had this problem — they consume tokens one at a time, so order is implicit in the processing sequence. Transformers traded that sequential bottleneck for parallelism, and the bill came due as a missing signal: the model has no idea what order the words arrived in.
Self-attention is a set operation. Language is a sequence. Positional encoding is the bridge — it's the only place a vanilla Transformer learns that token order matters at all.
So every token embedding needs a second signal injected: a representation of where it sits in the sequence. The question for the rest of Phase 4 is simply — what should that signal look like?
The obvious first idea: just add the index
Each token has a position: 0, 1, 2, 3, … Why not feed that integer straight into the model? Take the token embedding and add the position value to it.
token "the" at position 0 → embedding + 0 token "cat" at position 1 → embedding + 1 token "sat" at position 2 → embedding + 2 token "on" at position 3 → embedding + 3
Clean, intuitive, zero parameters. The model can read the position straight off the input. For a 5-word sentence it even works fine. The trouble shows up the moment sequences get long — which, for an LLM trained on thousands of tokens, is always.
Failure 1: unbounded magnitude wrecks the activations
Neural networks want inputs in a small, stable range — roughly mean 0, variance 1. That's what weight initialisation, normalisation layers, and activation functions all assume. Token embeddings live in that range by design.
Now add raw position. At position 4000 you're adding 4000 to embedding values that sit around ±1. The position signal doesn't complement the token signal — it obliterates it. The actual word becomes rounding error against a giant position number.
embedding values: [ 0.4, -0.9, 0.2, 1.1, ... ] ← carries word meaning + position 4000: [ 4000.4, 3999.1, 4000.2, 4001.1, ... ] word meaning: now indistinguishable noise on top of ~4000
Worse, the magnitude depends on sequence length. Position 10 and position 10000 produce activations that differ by three orders of magnitude. Gradients explode, normalisation layers thrash trying to rescale wildly different inputs, training destabilises. A signal whose scale grows without bound is the enemy of stable optimisation.
Failure 2: normalising the index breaks cross-length consistency
Obvious patch — bound the range. Divide every position by the sequence length so it lands in [0, 1].
position 0 of 5 → 0 / 4 = 0.00 position 1 of 5 → 1 / 4 = 0.25 position 2 of 5 → 2 / 4 = 0.50 position 4 of 5 → 4 / 4 = 1.00 position 2 of 100 → 2 / 99 = 0.0202 position 2 of 1000 → 2 / 999 = 0.0020
Magnitude problem solved — everything sits in [0, 1]. But a new bug appears: the same position means different things in different-length sequences. "Second token" is 0.25 in a 5-token sentence and 0.002 in a 1000-token document. The model can never learn a stable notion of "position 2" because the number representing it shifts with total length.
And the step size between adjacent tokens shrinks as sequences grow. In a 5-token sequence neighbours differ by 0.25; in a 10000-token sequence by 0.0001. At long context, adjacent positions become numerically inseparable — the encoding can't even distinguish "next word" from "word after that."
Failure 3: no generalisation to unseen lengths
This is the one that matters most for LLMs, and it's worth stating sharply.
Suppose training sequences max out at 512 tokens. With normalised integer positions, the model has only ever seen the value 1.0 attached to "the last of 512 tokens." Deploy it on a 2000-token document and position 1000 maps to a normalised value (and an absolute step size) the model never encountered in training. It has no learned behaviour for that regime. Output quality falls off a cliff exactly when the input grows past training length.
This is the length extrapolation problem, and it becomes the central battleground of positional encoding. The whole reason RoPE eventually won (article 4.5) is that it extrapolates gracefully where these naive schemes shatter.
Fig 1 — Both naive integer schemes fail. The right column is the requirements list every later encoding must satisfy.
The requirements list this failure produces
The two integer attempts failed, but productively — each failure pins down a property the encoding must have. Collect them and you get the design spec that the rest of Phase 4 chases:
- Bounded magnitude. The position signal must stay in a fixed range regardless of how long the sequence is, so it never swamps the token embedding or destabilises training.
- Unique per position. Every position needs a distinct encoding, otherwise the model can't tell two tokens apart by location.
- Length-independent. "Position 2" must produce the same encoding in a 5-token and a 5000-token sequence. The meaning of a position cannot depend on total length.
- Deterministic and extrapolatable. Position 5000 should be computable even if training never exceeded 512, with behaviour that degrades gracefully rather than catastrophically.
- Relative distance recoverable. Attention fundamentally cares "how far apart are these two tokens", not "what is their absolute index." The encoding should make relative offset easy for the model to extract.
That last requirement is subtle and turns out to be the deepest one. It's exactly what RoPE delivers natively (the dot product of two RoPE-rotated vectors depends only on their distance) and it's why RoPE eventually beat sinusoidal absolute encoding. We met that property already in Phase 3.5; Phase 4 builds up to why it's the property that matters.
Where this goes next
Integer encoding fails requirements 1, 3, and 4. The fix isn't a smarter scalar — it's a different representation entirely. Article 4.2 takes the first real step: instead of one number per position, use a vector of bits. Binary encoding bounds magnitude (every component is 0 or 1), gives every position a unique pattern, and — crucially — different bits flip at different frequencies. That frequency idea is the seed of everything that follows: sinusoidal encoding (4.3) is the smooth, continuous version of binary, and RoPE (4.4) is the rotational version of sinusoidal.
The whole arc of Phase 4 is one idea getting refined: encode position as a multi-frequency pattern, not a single magnitude. Integer PE is the strawman that proves a single scalar can't work. Everything after is the search for the right multi-frequency vector.
References
- Vaswani et al. (2017), Attention Is All You Need — introduces the permutation problem and the need for positional encoding. arXiv:1706.03762
- Su et al. (2021), RoFormer: Enhanced Transformer with Rotary Position Embedding — the destination of this arc; motivates relative-distance encoding. arXiv:2104.09864
- Press et al. (2021), Train Short, Test Long (ALiBi) — frames length extrapolation as the core PE problem. arXiv:2108.12409