Article 4.1 ended with a requirements list: bounded magnitude, unique per position, length-independent, extrapolatable, relative distance recoverable. A single scalar can't hit all of them. The fix is to stop using one number and start using a vector.
Binary encoding is the simplest vector idea, and it's a genuinely useful stepping stone — not because anyone ships it, but because it introduces the one concept that survives all the way to RoPE: different dimensions oscillate at different frequencies. Sinusoidal encoding (4.3) is literally the continuous version of what you'll see here.
The idea: write the position in binary
Every integer has a binary representation. Instead of feeding position 5 as the scalar 5, feed its bits as a vector [1, 0, 1]. Each bit gets its own dimension.
pos 0 → 0 0 0 0 pos 1 → 0 0 0 1 pos 2 → 0 0 1 0 pos 3 → 0 0 1 1 pos 4 → 0 1 0 0 pos 5 → 0 1 0 1 pos 6 → 0 1 1 0 pos 7 → 0 1 1 1 pos 8 → 1 0 0 0 ... pos 15 → 1 1 1 1
With d bits you address 2d distinct positions. 16 dimensions covers 65,536 positions; 20 dimensions covers over a million. Compact, and every position gets a guaranteed-unique pattern.
Check it against the requirements
Binary encoding clears several bars that integer encoding tripped on:
- Bounded magnitude ✓ — every component is 0 or 1. No matter how long the sequence, no value ever exceeds 1. The signal can never swamp the token embedding. This was integer PE's fatal flaw, and binary kills it outright.
- Unique per position ✓ — binary representations are unique by construction. No two positions share a pattern (within range).
- Length-independent ✓ — position 5 is
0101whether the sequence is 8 tokens or 8000. The encoding of a position never depends on total length. This was the normalised-integer flaw, also killed. - Deterministic ✓ — computable for any position with a fixed rule, no training required, no learned table to overflow.
Four of the five requirements, satisfied by a scheme you could implement in one line. That's real progress over Article 4.1.
The key observation: bits are frequencies
Here's the insight that makes binary worth studying. Look down each column of the table as position increases, and watch how often each bit flips:
position: 0 1 2 3 4 5 6 7 8 9 ...
-------------------
bit 0 → 0 1 0 1 0 1 0 1 0 1 flips every step (fastest)
bit 1 → 0 0 1 1 0 0 1 1 0 0 flips every 2 steps
bit 2 → 0 0 0 0 1 1 1 1 0 0 flips every 4 steps
bit 3 → 0 0 0 0 0 0 0 0 1 1 flips every 8 steps (slowest)
Each bit is a square wave with its own period. The low bit toggles fastest (period 2), each higher bit half as fast (period 4, 8, 16, …). The position is encoded by the combination of phases across these different-frequency channels. This is exactly a binary counter — and it's exactly the structure of sinusoidal encoding, except sinusoidal swaps the abrupt square waves for smooth sine/cosine curves.
This is the load-bearing idea of all of Phase 4: encode position as a stack of oscillations at geometrically spaced frequencies. Fast dimensions capture fine local position; slow dimensions capture coarse global position. Binary is the discrete prototype; sinusoidal and RoPE are the continuous refinements.
Fig 1 — Each binary bit is a square wave at a different frequency. Low bits oscillate fast, high bits slow. Position = the phase pattern across all frequencies. Sinusoidal PE replaces these squares with smooth sines.
Why binary still isn't good enough
Binary clears four requirements but stumbles on the fifth — and the fifth is the one that drives the rest of the field. The problem is that binary values are discrete: each component jumps abruptly between 0 and 1 with nothing in between.
Problem 1: discreteness fights gradient descent
Neural networks learn through smooth gradients. A function that snaps from 0 to 1 with no intermediate values has a derivative that's zero almost everywhere and undefined at the jump. The model can't learn a smooth relationship between nearby positions, because "nearby in position" doesn't map to "nearby in encoding." Positions 7 (0111) and 8 (1000) are adjacent in sequence but differ in every single bit — maximally far apart in encoding space. The representation is jagged exactly where you want it smooth.
Problem 2: relative distance is not recoverable
This is requirement 5, and binary fails it. There's no clean operation on two binary position vectors that yields their distance. Compare 0011 (pos 3) and 0101 (pos 5): the gap is 2, but nothing about the bit patterns exposes "2" in a way attention can use linearly. Hamming distance counts differing bits, not positional distance — positions 7 and 8 have Hamming distance 4 despite being adjacent. The encoding scrambles the very quantity attention needs most.
pos 3: 0 0 1 1 pos 5: 0 1 0 1 true distance = 2 pos 7: 0 1 1 1 pos 8: 1 0 0 0 true distance = 1, but ALL bits differ
Attention scores depend on dot products. For relative position to be recoverable, the dot product of two position encodings should be a smooth function of their distance. Binary's discrete bit patterns give no such function. This is the gap that smooth sinusoids — and later RoPE's rotations — close.
The bridge to sinusoidal
Binary handed us the frequency idea but in a broken, discrete wrapper. The fix writes itself: keep the multi-frequency structure, throw away the discreteness. Replace each square wave with a smooth sine wave at the same frequency.
binary bit (discrete): square wave, values ∈ {0, 1}
sinusoidal dim (smooth): sine wave, values ∈ [-1, 1]
A sine wave is continuous and differentiable everywhere, so adjacent positions get nearby encodings — gradients flow. And — the deep payoff — sinusoids have a trigonometric identity (sin/cos angle-addition) that lets a fixed linear transform shift one position's encoding to another's. That's precisely what makes relative distance recoverable, the requirement binary couldn't meet. Article 4.3 develops exactly this.
| Requirement | Integer | Binary | Sinusoidal (next) |
|---|---|---|---|
| Bounded magnitude | ✗ | ✓ | ✓ |
| Unique per position | ✓ | ✓ | ✓ |
| Length-independent | ✗ | ✓ | ✓ |
| Smooth / differentiable | ✓ | ✗ | ✓ |
| Relative distance recoverable | ✗ | ✗ | ✓ |
Binary's contribution isn't a usable encoding — it's the realisation that position is naturally a multi-frequency signal. Once you see position as "a stack of oscillators," sinusoidal and RoPE stop looking like clever tricks and start looking like the obvious continuous version of counting in binary.
References
- Vaswani et al. (2017), Attention Is All You Need — Section 3.5 motivates sinusoidal encoding as a continuous multi-frequency scheme. arXiv:1706.03762
- Su et al. (2021), RoFormer (RoPE) — extends the frequency idea to rotations for native relative encoding. arXiv:2104.09864
- Amirhossein Kazemnejad (2019), Transformer Architecture: The Positional Encoding — widely cited explainer using the binary-counter analogy. kazemnejad.com