Speculative Speculative Decoding — Guessing the Guess Before It Lands.

Here's a thing that's always slightly absurd about large language models: they generate text one token at a time, in strict order, and the most expensive part — the big model — sits mostly idle while it waits for itself. The bottleneck isn't math. It's the sequence. Each token needs the one before it.

Speculative decoding was the clever fix everyone adopted: let a small, fast model guess several upcoming tokens, then let the big model check them all in a single forward pass. When the guesses are right, you got many tokens for the price of one big-model step. Beautiful.

But this paper — "Speculative Speculative Decoding" by Tanishq Kumar, Tri Dao, and Avner May (arXiv:2603.03251) — notices that speculative decoding has a hidden serial step of its own. And it parallelizes that too. The result, an algorithm called Saguaro, runs ~30% faster than tuned speculative-decoding baselines and up to 5× faster than plain autoregressive decoding.

Recap: how speculative decoding works

Two models. A small draft model (cheap, fast, a bit dumb) and a big target model (expensive, slow, the one whose outputs you actually want).

1. Draft: the small model proposes the next $k$ tokens, fast.
2. Verify: the big model runs once over all $k$ guesses in parallel, scoring each. Tokens that match what the big model would have produced are accepted; the first mismatch and everything after it are thrown away.
3. Repeat from wherever verification stopped.

The win: one big-model pass can confirm several tokens at once. The catch nobody talks about: drafting and verifying still happen one after the other. You draft, then you verify, then — only once you know the result — you draft again. There's a serial dependency baked in: the next draft can't begin until you know where the last verification landed.

Speculative decoding removed the per-token serial dependency of the big model. But it introduced a new serial loop: draft → verify → draft → verify. SSD attacks that loop.

The key move: speculate on the speculation

The insight is almost cheeky. While the big model is busy verifying the current batch of guesses, the draft model is idle — waiting to find out the result so it knows where to draft next. But the number of possible verification outcomes is small and predictable.

A verification of $k$ draft tokens has only $k+1$ possible outcomes: accept 0 tokens, accept 1, accept 2, …, accept all $k$. That's it. So instead of waiting, the draft model pre-emptively drafts continuations for the likely outcomes — in parallel with the ongoing verification.

When verification finishes and reveals "we accepted 3 tokens," you check: did we already prepare a speculation for the accept-3 case? If yes, return it immediately — the drafting overhead for that step vanishes entirely. You've hidden the draft latency behind the verify latency.

optional read — why there are only k+1 outcomes

Speculative decoding accepts a prefix of the draft. If the draft proposes tokens $d_1, \dots, d_k$, the verifier accepts the longest prefix that agrees (under the standard accept/reject sampling rule), then emits one extra "bonus" token. The accepted-length $a \in \{0, 1, \dots, k\}$ fully determines where the next draft must continue from:

$$\text{next context} = x \,\Vert\, d_{1:a} \,\Vert\, t_{\text{bonus}}(a)$$

So the branching factor of "what should we draft next" is exactly $k+1$. Small, enumerable, and — crucially — most of the probability mass concentrates on a few of those outcomes, so you don't even need to prepare all $k+1$ branches to win most of the time.

The three challenges (and the fixes)

Pre-drafting for every outcome sounds free, but it isn't. The paper names three concrete problems and offers a principled fix for each — that engineering is what turns the idea into Saguaro.

Challenge 1 — combinatorial blowup

If you naively branch on every possible outcome at every step, the tree of pre-emptive drafts explodes exponentially. You can't prepare everything.

Fix: spend your draft budget where the probability is. Verification outcomes are not uniform — acceptance lengths cluster (a well-matched draft model usually gets most tokens accepted). Allocate pre-emptive drafts to the high-probability outcomes only, and accept that the rare branches fall back to the normal draft-then-wait path.

Challenge 2 — wasted draft compute

Every pre-emptive draft you prepare for an outcome that doesn't happen is thrown away. If you over-speculate, you burn the small model's compute (and GPU memory bandwidth) on branches that never get used — which can erase the latency win.

Fix: treat it as a budget-allocation problem. Balance the expected latency saved against the extra compute spent, and only pre-draft branches whose expected-payoff justifies the cost. The paper formalizes this trade-off rather than guessing.

Challenge 3 — scheduling on real hardware

Drafting and verifying now want to run concurrently on the same GPU. Naive concurrency means they fight over the same compute units and you get no overlap at all — the verify pass and the pre-emptive drafts just serialize anyway under the hood.

Fix: careful kernel scheduling so the small draft work slots into the gaps of the big verify pass (which is often memory-bound and leaves compute headroom). This is the unglamorous systems work that makes the theoretical overlap actually materialize on an accelerator.

The pattern of this paper: a one-sentence idea ("draft the next step before you know the current result") plus three pages of "here's why that's hard and how we actually made it fast." The idea is the hook; the challenge-fixes are the contribution.

Saguaro — the assembled algorithm

Put the three fixes together and you get Saguaro: an SSD implementation that pre-drafts only the high-probability verification outcomes, budgets its speculative compute against expected payoff, and schedules draft + verify to genuinely overlap on the GPU.

When a verification outcome matches a prepared branch — which it does most of the time, because acceptance lengths are predictable — the next tokens are returned with zero drafting latency. Drafting effectively becomes free, hidden entirely behind verification.

The numbers

• ~30% faster on average than optimized speculative-decoding baselines.
• Up to 5× faster than plain autoregressive decoding.
• Measured with open-source inference engines — not a private toy setup, which makes the result more credible and reproducible.

The 30% is the honest headline. Speculative decoding was already a strong baseline; squeezing another third out of it by removing a serial step most people didn't even notice is a real win for anyone paying a GPU bill per token.

Why this is a satisfying paper

It's recursive in the best way — it applies the core trick of speculative decoding (do work ahead of time, verify later) to speculative decoding itself. The name is a joke that also happens to be a precise description. And it doesn't require retraining anything: it's a pure inference-time, drop-in acceleration that sits on top of models you already have.

The limitation is honest too: the win depends on verification outcomes being predictable. If your draft model is poorly matched to the target (acceptance lengths all over the place), the high-probability branches you pre-draft will miss more often and the advantage shrinks toward ordinary speculative decoding. Good draft–target pairing is still doing the heavy lifting — SSD just stops wasting the idle time when that pairing is good.

A worked example, token by token

Concrete numbers make the idea click. Say the draft model proposes $k = 4$ tokens to continue "The capital of France is":

draft proposes:   [" Paris", " ,", " a", " beautiful"]
target verifies:  [" Paris" ✓, " ," ✓, " a" ✓, " beautiful" ✗ → " city"]

The verifier agrees on the first three tokens and rejects the fourth, substituting its own "bonus" token (" city"). Acceptance length $a = 3$. Under plain speculative decoding, the loop now emits those 4 tokens, then wakes the draft model and asks for 4 fresh guesses starting from "…is Paris , a city". That draft call is pure latency — the GPU was idle waiting for the verify result before it even knew where to draft from.

SSD's move: during the verify pass above, the draft model already produced candidate continuations for each possible outcome $a = 0,1,2,3,4$. The instant verification reports $a = 3$, SSD returns its prepared "continue-from-accept-3" draft. No draft call on the critical path — the drafting latency was folded into time the big model was busy anyway.

The latency anatomy: why there's a gap to fill

To see why this is nearly free, look at where time goes in one speculative step. The verify pass is a single forward pass of the big model over $k+1$ positions — the expensive part, and on modern accelerators it's usually memory-bandwidth bound: you're streaming billions of weights through the cores and the actual matmuls don't saturate the compute units. The draft pass is several forward passes of a small model — cheap in FLOPs but strictly serial relative to verification.

So a big, slow, bandwidth-bound op (verify) runs next to a small, compute-light op (draft) that has nothing to do but wait. Textbook setup for overlap: slot the small thing into the idle compute headroom of the big thing. "Slot into the headroom" is exactly Challenge 3 — and it's harder than it sounds on real hardware.

How much can you save? The probability tree

The expected speedup hinges on one thing: how predictable the acceptance length is. Treat the verification outcome as a random variable $A$ over $\{0,\dots,k\}$. If $A$ is sharply peaked — a well-matched draft model that almost always gets 3 or 4 tokens accepted — then pre-drafting just two branches (accept-3, accept-4) covers most of the probability, and you almost always hit a prepared branch.

If $A$ is spread out — an erratic draft model, acceptance all over the place — no small set of prepared branches covers the likely outcomes, you miss often, and SSD degrades gracefully back toward ordinary speculative decoding (you just ate some wasted draft compute).

optional read — expected drafting overhead eliminated

Let $p_a = \Pr[A = a]$ be the acceptance-length distribution and $S \subseteq \{0,\dots,k\}$ the set of outcomes you pre-draft (chosen by Challenge 1's budget allocation). The fraction of steps where drafting latency is fully hidden is

$$P_{\text{hit}} = \sum_{a \in S} p_a$$

and expected per-step draft latency drops from $t_{\text{draft}}$ to about $(1 - P_{\text{hit}})\, t_{\text{draft}}$, minus the scheduling cost of running the pre-emptive drafts concurrently. Because acceptance distributions for well-tuned draft–target pairs are heavily concentrated, even $|S| = 2$ or $3$ pushes $P_{\text{hit}}$ high — which is why the average ~30% gain appears without preparing all $k+1$ branches.

Where SSD sits in the inference stack

Worth placing against neighbours, because the fast-inference space is crowded:

• Medusa / EAGLE — improve the draft step (extra prediction heads, feature-level drafting) so acceptance lengths rise. SSD is orthogonal: it removes the serial wait regardless of how drafting is done, and a better draft model only sharpens SSD's branch prediction.
• Tree attention / token trees — verify many candidate continuations in one pass by packing them into a tree. SSD's pre-emptive branches are a tree over verification outcomes, not over token continuations — a different axis.
• Plain speculative decoding — the baseline SSD beats by ~30%. SSD is a strict superset: disable pre-emptive drafting and you're back to it.

Mental model: most acceleration work gets more accepted tokens per verify. SSD removes a fixed serial cost that sat between every verify. Those gains stack.

Practical takeaways

• Inference-time change — no retraining, drop-in on an existing draft–target pair.
• The win scales with draft–target alignment. Tune a well-matched draft model first; SSD then converts the predictable acceptance into hidden latency.
• The hard part is systems, not theory — concurrency scheduling (Challenge 3) is what makes the overlap real, implemented in Saguaro against open-source engines.
• Diminishing returns when acceptance is unpredictable, but it never runs slower than speculative decoding by more than the wasted-draft budget you permit.

arXiv:2603.03251 — Speculative Speculative Decoding, Tanishq Kumar, Tri Dao, Avner May. Presented at the YC Paper Club.