Speculative decoding — making LLMs generate faster, for free.

LLM text generation is slow because it produces one token at a time, and each token barely uses the GPU — the chip spends its time waiting on memory, not computing. Speculative decoding fixes this with a cheap trick: let a small, fast draft model guess the next few tokens, then let the big target model check all of those guesses in a single forward pass. Because checking many tokens at once is nearly as cheap as checking one, you get 2–3× faster generation. The clever part: a small piece of math guarantees the output is exactly what the big model would have produced alone — same quality, no approximation. This guide builds the whole idea from scratch.

The problem: why generating text is slow

To understand the fix, you have to feel the problem. A large language model generates text autoregressively — one token at a time, where each new token depends on all the tokens before it. To produce "The cat sat on the mat," the model runs a full forward pass to get "The," then another full pass (now seeing "The") to get "cat," then another to get "sat," and so on. Six tokens means six sequential passes through a model with tens or hundreds of billions of parameters.

Here is the painful part, and it is the key to everything: each of those forward passes barely uses the GPU. To generate one token, the GPU must load the model's entire weight matrix — hundreds of gigabytes — from memory, but then do only a tiny amount of arithmetic with it (one token's worth). The chip finishes the math almost instantly and then sits idle, waiting for the next slab of weights to arrive from memory. Generation is memory-bandwidth bound, not compute bound.

The key asymmetry: generating vs checking

Speculative decoding rests on one beautiful observation about transformers:

Why the difference? When you already have a candidate sequence — say someone hands you "The cat sat on" — a transformer can score all of those positions in a single forward pass, because attention looks at all positions at once. This is exactly what happens during prefill (processing the prompt) and during training: many tokens, one pass, GPU fully fed. It is the same compute-bound, efficient regime that single-token generation can never reach.

So the trick writes itself. What if we could turn the slow, sequential generation problem into the fast, parallel checking problem? We would need someone to produce candidate tokens cheaply, so the big model only ever has to check rather than generate. That "someone" is the draft model.

The core idea: draft, then verify

Speculative decoding uses two models that share the same vocabulary:

• The target model — the big, accurate model whose output you actually want (e.g. a 70B model). Slow to run.
• The draft model — a small, fast model (e.g. a 1B model, or even just a few extra layers/heads). Much cheaper per token, but lower quality.

The loop, repeated until generation is done:

1. Draft. The small model quickly generates a guess of the next γ (gamma) tokens — say 4 of them — autoregressively. This is cheap because the draft model is tiny.
2. Verify. The target model takes the current text plus those 4 drafted tokens and runs one forward pass, scoring all 4 positions in parallel. This costs about the same as the target model generating a single token — but it gives us information about 4.
3. Accept / reject. Walk left to right through the 4 drafted tokens. Keep the ones the target model agrees with; the moment it disagrees, throw away that token and everything after it.
4. Repeat from the new, longer text.

Notice the payoff in Fig 2: in a single expensive target forward pass, we advanced the text by 4 tokens (three accepted drafts + one correction), where standard decoding would have advanced by 1. When the draft model guesses well, you get a big multiplier. When it guesses badly, you fall back toward normal speed — never slower than the target alone by much. Best case it's a big win; worst case it's roughly break-even.

The crucial guarantee: it's lossless

The natural worry: "I'm using a dumb little model to write some of my tokens — doesn't that hurt quality?" The answer, and the reason this technique matters, is no — speculative decoding produces output drawn from exactly the same probability distribution as the target model running alone. It is not an approximation. It is not "good enough." It is mathematically identical in distribution. The draft model only affects speed, never what gets generated.

This is guaranteed by a verification rule called speculative sampling (a form of modified rejection sampling), introduced in the Leviathan et al. paper. For each drafted token, let:

• p(x) = the probability the target model assigns to token x,
• q(x) = the probability the draft model assigned to that same token x when it proposed it.

The acceptance rule for a drafted token x is:

Read it intuitively. If the target likes the token at least as much as the draft did (p ≥ q), the ratio is ≥ 1, so you always accept. If the target likes it less than the draft did (p < q), you accept only with probability p/q — sometimes you reject, precisely often enough to correct for the draft over-proposing that token.

And when a token is rejected, you don't just stop — you resample the replacement token from a carefully adjusted distribution, norm(max(0, p(x) − q(x))), which removes the bias the draft introduced. This combination — accept-with-ratio, then resample-from-the-residual — is exactly what makes the algebra cancel out so the final distribution equals the target's p. The proof is a few lines; the consequence is profound: you can speed up generation with no quality cost whatsoever.

A fully worked example, with real numbers

The accept/resample rule is much clearer with concrete probabilities. Suppose the model is choosing the next token from a tiny vocabulary of five words: {cat, dog, mat, sat, on}. At some position, the two models assign these probabilities:

token	draft q(x)	target p(x)	p/q	what happens
cat	0.50	0.40	0.80	accept w.p. 0.80
dog	0.20	0.10	0.50	accept w.p. 0.50
mat	0.20	0.30	1.50→1	always accept
sat	0.10	0.15	1.50→1	always accept
on	0.00	0.05	—	draft never proposes it

Say the draft model sampled dog (it had 0.20 probability of doing so). We draw a uniform random number r ∈ [0,1) and accept if r < p(dog)/q(dog) = 0.10/0.20 = 0.5. So half the time we keep dog; half the time we reject it. The draft proposed dog too eagerly (0.20 vs the target's 0.10), and the ratio test corrects for exactly that over-eagerness.

Now suppose we rejected. We do not resample from the target distribution p directly — that would double-count. Instead we sample from the residual distribution, the part of p that the draft under-covered:

Computing max(0, p − q) for each token:

token	p − q	max(0, p−q)	normalized residual
cat	0.40 − 0.50 = −0.10	0	0.00
dog	0.10 − 0.20 = −0.10	0	0.00
mat	0.30 − 0.20 = +0.10	0.10	0.40
sat	0.15 − 0.10 = +0.05	0.05	0.20
on	0.05 − 0.00 = +0.05	0.05	0.20

The leftover mass sums to 0.10 + 0.05 + 0.05 = 0.20, so after normalizing we resample from {mat: 0.5, sat: 0.25, on: 0.25}. Notice the residual gives mass to on — a token the draft model would never have proposed (q = 0). This is how speculative decoding can still output tokens the draft can't produce: rejection + residual sampling reaches them. The residual leans toward tokens the target liked more than the draft did, precisely cancelling the bias the accept step introduced.

Why it's lossless — the proof, slowly

The claim is that the probability of finally emitting any token x equals the target's p(x), exactly. There are two disjoint ways token x can be emitted at this position:

1. It was drafted and accepted. Probability = (draft proposes x) × (accept) = q(x) × min(1, p(x)/q(x)) = min(q(x), p(x)).
2. Some other token was drafted, rejected, and x came from the residual. Probability = (a rejection happens) × p_residual(x).

The total probability of any rejection is 1 − Σ min(q, p) = Σ max(0, q − p) (the mass the draft over-proposed). And the residual distribution puts probability max(0, p(x) − q(x)) / Σ max(0, p − q) on x. Multiply the rejection probability by the residual — and because Σ max(0, q−p) = Σ max(0, p−q) (the over- and under-shoot are equal, both distributions summing to 1), the normalizing denominators cancel, leaving exactly max(0, p(x) − q(x)). Adding the two paths:

That identity — min(a,b) + max(0, b−a) = b — is the whole proof in one line. Whatever the draft does, the accept-then-residual machinery reconstructs the target distribution token. The draft can be any distribution, even a terrible one; correctness never depends on draft quality, only speed does. That separation of correctness from quality is what makes speculative decoding safe to turn on by default.

How much faster? The acceptance-rate math

The speedup hinges on one number: the acceptance rate α (alpha) — the probability that a drafted token gets accepted. It measures how well the draft model imitates the target. High α means the draft is a good stand-in and most guesses stick; low α means lots of wasted drafting.

With acceptance rate α and γ drafted tokens per round, the expected number of tokens accepted per target pass is:

You don't need to memorize the formula; you need the intuition it encodes:

Acceptance α	Draft quality	≈ tokens / target pass (γ=4)	Effect
0.9	excellent stand-in	~4.1	~3–4× faster
0.7	good	~2.8	~2–3× faster
0.5	mediocre	~1.9	~1.5–2× faster
0.3	poor	~1.4	marginal; overhead may dominate

But raw acceptance isn't the whole story. The real wall-clock speedup also depends on the cost ratio c — how expensive the draft model is relative to the target. Every round you pay for γ draft steps plus one target step. If the draft model is too big, those draft steps eat your savings. The sweet spot is a draft model that is both a good imitator (high α) and very cheap (low c) — and those two goals pull in opposite directions, which is the central tension of designing a speculative system.

The wall-clock model: why verification is almost free

To turn acceptance rate into real speedup you need the cost side. Let the target's per-step time be T and the draft's per-step time be cT, where c is the cost ratio (e.g. c = 0.05 if the draft is 20× cheaper). One speculative round costs:

and yields E[tokens] = (1 − α^(γ+1))/(1 − α) accepted tokens. So the per-token time becomes T·(1 + γc) / E[tokens], versus T for standard decoding. The speedup is:

Plug in numbers: with α = 0.8, γ = 4, c = 0.05, you get E ≈ 3.36 accepted tokens and a divisor of 1 + 4·0.05 = 1.2, so the speedup ≈ 3.36 / 1.2 ≈ 2.8×. The (1 + γc) term is the tax you pay for the draft. This is why a cheap draft matters so much: if c were 0.3 (draft only ~3× cheaper), the divisor jumps to 2.2 and the same acceptance gives only ~1.5×. Draft cost is a denominator; acceptance is a numerator.

The deeper reason the verify pass is cheap takes us back to the hardware. During verification, the target processes γ+1 tokens at once instead of 1. In the language of the Roofline article, this raises the arithmetic intensity of the forward pass — more compute per byte of weights loaded — sliding the operation rightward from the memory-bound regime toward compute-bound. As long as γ+1 tokens still fit under the compute roof (true for the small γ values used in practice), the extra tokens ride along on the same weight-load for nearly free. That is the same "reuse the expensive load" principle behind tiling and batching; speculative decoding is batching across time (future tokens) instead of across requests.

The original algorithm, step by step

Putting it together, here is the speculative decoding loop from Leviathan et al. (and Chen et al., who proposed it concurrently), in pseudocode:

def speculative_decode(prompt, target, draft, gamma):
    tokens = prompt
    while not done(tokens):
        # 1. DRAFT: small model proposes gamma tokens, autoregressively
        draft_tokens, q = [], []
        ctx = tokens
        for _ in range(gamma):
            probs = draft(ctx)            # draft's distribution
            x = sample(probs)
            draft_tokens.append(x); q.append(probs[x])
            ctx = ctx + [x]

        # 2. VERIFY: target scores all gamma positions in ONE pass
        p = target(tokens + draft_tokens)  # p[i] = target prob at position i

        # 3. ACCEPT / REJECT left to right
        for i, x in enumerate(draft_tokens):
            r = uniform(0, 1)
            if r < min(1, p[i][x] / q[i]):
                tokens.append(x)           # accept
            else:
                # reject: resample from adjusted residual distribution
                tokens.append(sample(normalize(relu(p[i] - q_dist[i]))))
                break
        else:
            # all gamma accepted -> take the free bonus token from target
            tokens.append(sample(p[gamma]))
    return tokens

That's the entire mechanism. Two models, one verification pass, a ratio test, and a residual resample. Everything else in the field is a variation on making the draft cheaper, more accurate, or unnecessary.

The plumbing: KV cache, rollback & tree verification

The pseudocode hides the trickiest engineering: managing the KV cache across speculative rounds. Recall from KV cache internals that each model keeps a cache of key/value tensors for every token it has processed, so it never recomputes the past. Speculation complicates this because some processed tokens get thrown away.

Drafting and the draft cache

While the draft model generates its γ tokens, it builds up KV-cache entries for each one, just like normal decoding. Cheap, because the draft is small.

Verifying appends, then maybe rolls back

The target processes all γ drafted tokens in one pass, appending γ new entries to its KV cache. Then comes the catch: if only the first k of γ tokens are accepted, the cache entries for the rejected tokens (positions k+1 … γ) are now invalid — they were computed conditioned on tokens that won't be in the final sequence. The implementation must roll back the KV cache to length (prefix + k + 1), discarding the speculative tail. In practice this is just moving a length pointer — the stale entries are overwritten next round — but getting the bookkeeping exactly right (for both models, across batched requests of different accept lengths) is where real implementations spend their complexity budget.

From a chain to a tree

So far the draft proposes a single linear guess of γ tokens. But why bet on one continuation? Tree-based speculative decoding (used by Medusa, EAGLE, SpecInfer and others) drafts a small tree of candidate continuations — several alternatives at each position — and verifies the whole tree in a single target pass. If position 1 might be "the" or "a", you draft both and let verification pick the branch the target prefers. This raises the chance that some path through the tree is accepted, lifting the effective acceptance length.

Verifying a tree in one pass needs a special tree attention mask: each candidate token attends only to its own ancestors in the tree, not to sibling branches, so all branches are scored simultaneously without contaminating each other. It's a clever use of the attention mask — the same mechanism that normally enforces causal (left-to-right) attention, repurposed to encode tree structure. The cost is a slightly larger verify pass (more tokens) in exchange for higher acceptance; tuning the tree's width and depth is the modern analogue of tuning γ.

The modern variants: where the field went

The 2022 paper kicked off a wave of refinements. The big practical question — "where does the draft come from, and can we avoid a second model entirely?" — drives most of them.

Two-model (classic) speculative decoding

The original setup: a separate small model from the same family (e.g. Llama-1B drafting for Llama-70B). Simple and effective when a good small sibling exists with the same tokenizer. The main friction is needing — and serving — a second model, plus keeping both in memory.

Medusa — extra heads, no second model

Medusa removes the separate draft model. Instead it bolts several extra "heads" onto the target model itself, each trained to predict a future token (the token 2 ahead, 3 ahead, etc.) in parallel from the same hidden state. Those heads produce the candidate continuations, which the model then verifies. One model, no separate draft to serve — at the cost of training and storing the extra heads, and slightly more complex verification over a tree of candidates.

EAGLE — drafting in feature space

EAGLE (and its successors) is currently among the strongest approaches. Rather than predicting tokens directly, it trains a light auto-regressive head to predict the target model's internal feature vectors one step ahead, which turns out to be a much easier and more accurate prediction problem than guessing raw tokens. Higher acceptance rates, larger speedups, and it's widely supported in production serving stacks.

Self-speculative / layer-skipping

Here the draft is the target model running a subset of its own layers (early exit), or with some layers skipped. No extra parameters at all — the model drafts a cheap version of itself and then verifies with the full stack. Attractive when you can't or don't want to train or serve anything extra.

N-gram / prompt lookup — no model at all

The cheapest draft is no neural network whatsoever. Prompt-lookup decoding notices that LLM outputs often repeat spans from the prompt or earlier output (think summarization, code editing, RAG, or anything quoting its input). So the "draft" is just: find a matching n-gram in the existing text and propose its continuation. Zero draft-compute, surprisingly high acceptance on repetitive/grounded tasks, and trivially easy to deploy. It does nothing on highly creative generation where little repeats.

Lookahead decoding

A draft-model-free method that uses the target model's own parallel capacity to generate and verify multiple n-grams per step via a fixed-point (Jacobi-style) iteration — trading extra compute per step for fewer steps overall. Where speculative decoding guesses with a separate model, lookahead has the target model refine several rough future tokens at once until they stabilize, then verifies them. No draft to train or serve; the cost is extra parallel compute per step.

Multi-Token Prediction (MTP) — baked into the model

A recent trend (notably in DeepSeek-V3) trains the model from the start with extra prediction heads that forecast several future tokens, not just the next one. At training time this is a richer learning signal; at inference time those heads double as a built-in, well-aligned draft, so the model speculates on itself with very high acceptance because the heads were trained jointly with the backbone. MTP blurs the line between "model" and "draft" — the speculation capability is a native property of the weights rather than a bolt-on.

SpecInfer and serving-system integration

SpecInfer popularized tree-based speculation with multiple small draft models and a token-tree verifier, and pushed the ideas into production serving. The lasting contribution of this line of work is less a single algorithm than the system machinery — tree attention, batched verification across many requests, continuous batching that mixes speculative and non-speculative sequences — that lets these methods run efficiently inside vLLM / TensorRT-LLM rather than in a benchmark script.

EAGLE-2 and EAGLE-3: dynamic trees

EAGLE kept improving. EAGLE-2 makes the draft tree dynamic: instead of a fixed-shape tree, it uses the draft head's own confidence to decide where to spend the branching budget — growing the tree along promising continuations and pruning unlikely ones — which raises acceptance for the same verify cost. EAGLE-3 pushes further by drafting from multiple layers' features and dropping some of the original feature-prediction constraints, reporting some of the largest lossless speedups to date. The throughline of the EAGLE family: draft in the model's own feature space, and shape the candidate tree adaptively.

Variant	Where the draft comes from	Needs a 2nd model?	Best for
Classic (Leviathan/Chen)	separate small model	yes	when a good small sibling exists
Medusa	extra heads on the target	no (extra heads)	one-model deployments
EAGLE	feature-level draft head	no (light head)	max speedup, production serving
Self-speculative	target's own early layers	no	zero extra params
Prompt-lookup / n-gram	matching spans in the text	no (no model)	summarization, RAG, code edits
Lookahead	target's parallel n-gram guesses	no	draft-free speedup

Choosing and building a draft model

For the classic two-model setup, the draft model is the design. Get it right and you hit 2–3×; get it wrong and you barely move. The levers:

• Same family, same tokenizer. Non-negotiable for two-model speculation: the accept test compares p(x) and q(x) for the same token id, so both models must share a vocabulary. Llama drafts for Llama, Qwen for Qwen. Mixing families breaks the probability comparison. (Some research relaxes this with vocabulary-alignment tricks, but same-family is the safe default.)
• Roughly 10–20× smaller. The cost-ratio math says the draft must be much cheaper than the target, but small enough hurts acceptance. A 7B drafting for a 70B, or a 1B for a 13B, are typical pairings. The exact ratio is empirical — sweep a couple of sizes and measure end-to-end tokens/sec.
• Align the draft to the target by distillation. The single biggest acceptance booster: distill the draft on the target's outputs (or its distributions), so the draft learns to imitate the target specifically rather than just being a generically good small model. A draft trained to mimic this target accepts far more often than an off-the-shelf small model of the same size. This is why EAGLE/Medusa (trained against the target) beat naive small-model drafting.
• Quantize the draft aggressively. The draft only needs to be a good guesser, not perfectly accurate — every guess is checked. So you can run the draft at INT8/INT4 to cut its cost c with little acceptance loss, improving the speedup denominator. The target stays at full precision to preserve quality.
• Match the domain. Acceptance is highest when the draft has seen the target's domain. A code-tuned draft for a code target, a chat-tuned draft for a chat target. A domain-mismatched draft drags α down even at the right size.

The throughput story, in depth

This deserves its own section because it's where teams get surprised. The headline again: speculative decoding optimizes latency at low concurrency, and its benefit shrinks — sometimes to nothing — as batch size grows. The reason is the surplus-compute argument from the wall-clock section, made concrete.

At batch size 1 (one user, interactive), the target's forward pass is deeply memory-bound: it loads all the weights to serve a single token's worth of compute. There is enormous idle compute. Verifying γ+1 tokens uses that idle compute essentially for free, so the speedup is large.

As the batch grows, the server processes many sequences per weight-load. The forward pass becomes compute-bound — the GPU's arithmetic units fill up with real work from concurrent requests. Now there is little idle compute left to absorb verification, and the extra speculative tokens compete with genuine throughput. Worse, when speculative tokens get rejected, that verification compute was wasted — pure overhead. Past some batch size, speculation can reduce total throughput.

The practical implication: speculative decoding is a latency play for interactive, low-concurrency serving — chat, agents, single-user assistants, anything where time-to-token matters and the batch is small. For maximizing tokens/sec on a fully-loaded inference fleet, plain batching often wins, and some serving stacks dynamically disable speculation when the batch is large enough that it stops paying off. Always benchmark at your real concurrency, not at batch size 1, or you'll over-estimate the gain.

Practical considerations: when it helps (and when it doesn't)

A checklist of what actually matters in practice:

• Shared vocabulary is mandatory (for two-model setups). Draft and target must use the same tokenizer, or the probability comparison p(x)/q(x) is meaningless. Use models from the same family.
• Tune γ (draft length). More drafted tokens means more potential acceptances per round, but also more wasted draft work when an early token is rejected (everything after a reject is discarded). Typical values are 3–7; the optimum depends on your acceptance rate. Some systems tune γ dynamically.
• Acceptance is task-dependent. Predictable text (code, structured output, factual continuations) drafts well; highly creative or high-temperature sampling drafts worse. Prompt-lookup in particular shines on tasks that quote their input and does little elsewhere.
• Temperature interacts with acceptance. Greedy/low-temperature decoding tends to have higher acceptance (the target is more "sure," so the draft's confident guesses match more often). High temperature flattens distributions and lowers acceptance.
• Memory cost. A separate draft model needs its own weights and KV cache resident alongside the target. On tight VRAM that overhead competes with batch size and context length.
• It composes with everything else. Speculative decoding stacks on top of paged KV cache, quantization, and tensor parallelism — it's orthogonal to those and is now a standard feature in vLLM, TensorRT-LLM, and Hugging Face transformers.

Using it in practice

You rarely implement the algorithm yourself — the major serving stacks expose it as a config flag. Conceptually:

# Hugging Face transformers — assisted generation (two-model)
out = target.generate(inputs, assistant_model=draft_model)

# vLLM — speculative decoding via a draft model or n-gram
#   --speculative-model <draft>  --num-speculative-tokens 5
#   or  --speculative-model "[ngram]"  for prompt-lookup

# TensorRT-LLM — supports draft-target, Medusa, EAGLE, lookahead

The honest workflow is empirical: pick a candidate draft (a small sibling model, or n-gram for repetitive tasks), set γ around 4–5, measure tokens/second and acceptance on your prompts and batch size, then adjust. Because the output is provably identical to the target's, you can A/B purely on speed without worrying about quality regressions.

Where it sits among inference optimizations

Speculative decoding is one tool in a larger latency/throughput toolbox, and it's worth knowing how it relates to the others — because they compose, and they attack different bottlenecks.

Technique	What it attacks	Lossless?	Composes with spec-dec?
Speculative decoding	sequential decode latency	yes (exact)	—
Quantization (INT8/FP8/INT4)	weight memory + bandwidth	no (small quality cost)	yes — and shrinks the draft too
Paged KV cache (vLLM)	memory fragmentation	yes	yes
Tensor / pipeline parallelism	model too big for one GPU	yes	yes
Continuous batching	throughput / GPU utilization	yes	yes (with care at high batch)
Distillation to a smaller model	everything (use a small model)	no (quality drops)	n/a (different model)
FlashAttention	attention memory traffic	yes	yes

The key distinction: most speedups either change the model (quantization, distillation — accepting some quality loss) or improve the systems plumbing (paged cache, batching, FlashAttention — lossless but bounded by the same per-step decode cost). Speculative decoding is unusual: it's lossless like the plumbing optimizations, but it attacks the fundamental sequential nature of decoding that none of the others touch. That's why it stacks on top of all of them — you quantize, page the cache, parallelize, batch, and speculate. Each addresses a different axis.

A short history

The idea matured fast:

When	Milestone
Nov 2022	Leviathan, Kalman & Matias (Google) — Fast Inference from Transformers via Speculative Decoding: the speculative-sampling proof and the draft-target loop.
Feb 2023	Chen et al. (DeepMind) — concurrent Accelerating LLM Decoding with Speculative Sampling, demonstrated on Chinchilla.
2023	SpecInfer — token trees + multiple drafts, serving-system integration.
Jan 2024	Medusa — extra decoding heads, no separate draft model; Lookahead decoding — draft-free Jacobi iteration.
2024	EAGLE / EAGLE-2 — feature-level drafting and dynamic trees, state-of-the-art lossless speedups.
2024–25	Multi-Token Prediction baked into pretraining (e.g. DeepSeek-V3); EAGLE-3; broad support in vLLM, TensorRT-LLM, and Hugging Face.

In two years it went from a clever paper to a default feature in every major serving stack — fast adoption, because it's lossless and stacks on everything else.

A tuning playbook & troubleshooting

If you're turning speculative decoding on and the numbers disappoint, walk this list:

• Measure acceptance rate first. Most frameworks report it. If α is low (<0.5), fix the draft (distill it, match the domain, lower temperature) before touching anything else — α is the master dial.
• Sweep γ at your acceptance rate. Higher α supports larger γ; lower α wants smaller γ. Start at 4–5. With tree methods, tune tree width/depth instead.
• Check the cost ratio. Profile draft-step vs target-step time. If the draft is more than ~10–15% of the target's cost, your denominator (1+γc) is eating the gains — shrink or quantize the draft.
• Lower the temperature if quality allows. Greedy/low-temp decoding accepts more (the target is more decisive, the draft matches more often). High temperature inherently caps acceptance.
• Confirm it's actually lossless in your setup. If outputs differ from non-speculative decoding in distribution (beyond expected sampling randomness with a fixed seed), the implementation's accept/resample is buggy — correct speculative decoding is exact. This is a good regression test.
• Watch memory. A separate draft model and its KV cache compete with the target's batch size and context length on the same VRAM. On tight memory, a draft-free method (Medusa/EAGLE/n-gram) avoids the second cache.

FAQ

Does speculative decoding change the model's answers?

No. With the speculative-sampling acceptance rule, the output is drawn from exactly the same distribution as the target model decoding alone — token for token, in distribution. Quality is unchanged; only speed improves. This losslessness is the whole point and is what separates it from lossy approximations like using a smaller model outright.

If the draft is just a small model, why not use the small model directly?

Because the small model alone is lower quality. Speculative decoding gives you the small model's speed on the tokens it gets right, while the big model catches and fixes every token it gets wrong — so you keep the big model's quality. You're borrowing the draft's speed, not its judgment.

What's a typical real-world speedup?

Commonly 2–3× for interactive, low-batch serving with a well-matched draft, and more with strong methods like EAGLE on favorable tasks. Gains shrink at high batch sizes (the GPU is already busy) and on creative, high-temperature generation (lower acceptance). Numbers are very task- and setup-dependent — always benchmark.

Why verify all the drafted tokens in one pass instead of one at a time?

Because that single parallel pass is the entire saving. A transformer scores N given positions in one forward pass for nearly the same memory cost as scoring one — that's the prefill/training regime. Verifying one token at a time would just be normal slow decoding again, with no benefit.

Do I need to train anything?

For classic two-model and prompt-lookup: no — use an existing small model or no model at all. For Medusa and EAGLE: yes, you train the extra heads/draft head (relatively cheap, since the base model is frozen). Self-speculative needs no training but may benefit from light tuning of the early-exit layers.

Can the draft and target run on different GPUs?

Yes, and in some serving setups the small draft runs on cheaper or spare hardware while the target occupies the main accelerators. The coordination (draft → verify → accept) adds some communication, so the benefit depends on the interconnect and how the scheduler overlaps the two stages.

What is the acceptance rate, exactly, and what's a good value?

It's the probability that a drafted token survives the accept test — effectively how often the draft's guess matches what the target would have done. Above ~0.7 is good (gives ~2.5–3× with a cheap draft); below ~0.4 the overhead often isn't worth it. It depends on draft quality, how well the draft is aligned/distilled to the target, the task (predictable text accepts more), and temperature.

Why must I resample from the residual instead of just from the target?

Because the accept step already emitted some tokens with probability min(q,p). If, on rejection, you then sampled from the full target p, those tokens would be double-counted and the final distribution would be skewed. Sampling from the residual max(0, p−q) adds back exactly the mass the accept step missed, so the two paths sum to p. The one-line identity min(a,b)+max(0,b−a)=b is why it works.

Does speculative decoding help the prompt-processing (prefill) phase?

No — prefill already processes the whole prompt in one parallel, compute-bound pass; it's the efficient regime speculation is trying to reach. Speculative decoding only speeds up the sequential generation (decode) phase. If your bottleneck is a huge prompt and a short answer, speculation won't help much; if it's a long generated answer, it will.

How does tree-based speculation differ from the basic linear version?

Linear speculation bets on one continuation of γ tokens; if token k is wrong, everything after it is discarded. Tree speculation drafts several alternative branches and verifies them all in one pass using a tree attention mask, so even if the most-likely branch is rejected, another branch may be accepted — raising the effective accepted length per round. Medusa, EAGLE, and SpecInfer all use trees.

Is it worth it if I'm already serving a quantized model?

Often yes — they're orthogonal. Quantization shrinks weight memory and speeds each step but doesn't change decoding's one-token-at-a-time nature; speculation does. They compose: you can even quantize the draft aggressively to cut its cost. Just benchmark the combination at your real batch size, since both interact with the memory/compute balance.

Takeaways

• LLM generation is slow because it's one token per full weight-load — memory-bound, with the GPU's compute mostly idle.
• Checking many tokens is a single parallel pass; generating them is many sequential passes. Speculative decoding converts generation into checking.
• A cheap draft model proposes γ tokens; the target verifies all of them in one pass; you accept the agreeing prefix and correct the first disagreement.
• The speculative-sampling accept rule (min(1, p/q) + residual resample) makes the output provably identical to the target alone — lossless.
• Speedup is driven by the acceptance rate α and the draft's cost; the best draft is cheap and a good imitator (the central tension).
• Modern variants — Medusa, EAGLE, self-speculative, prompt-lookup, lookahead — mostly remove or cheapen the draft.
• It's a latency win at low batch sizes, less so at high throughput; it composes with quantization, paged KV cache, and tensor parallelism.

References & further reading

• Leviathan, Kalman & Matias — Fast Inference from Transformers via Speculative Decoding (2022) — the foundational paper and the speculative-sampling proof.
• PyTorch Blog — A Hitchhiker's Guide to Speculative Decoding — practical walkthrough and benchmarks in the PyTorch/serving stack.
• NVIDIA Developer Blog — An Introduction to Speculative Decoding — intuition, draft-model selection, and TensorRT-LLM support.
• Chen et al. — Accelerating LLM Decoding with Speculative Sampling (2023) — the concurrent DeepMind formulation.
• Cai et al. — Medusa — multiple decoding heads, no separate draft model.
• Li et al. — EAGLE & EAGLE-2/3 — feature-level drafting and dynamic draft trees.
• Miao et al. — SpecInfer — token-tree speculation and serving-system integration.
• Fu et al. — Lookahead Decoding — draft-model-free Jacobi-style parallel decoding.
• DeepSeek-V3 — Multi-Token Prediction baked into pretraining, doubling as a native draft.