# Why speculative decoding is faster, and what interviews ask

Source: https://www.techinterview.org/post/3233477300/why-speculative-decoding-is-faster/
Updated: 2026-08-10 · techinterview.org

The cheapest way to make a 70B model spit out tokens faster without retraining or quantizing it is to let a small model guess the next few tokens and have the big model check all the guesses at once. That is speculative decoding. If you are interviewing for an inference or serving role in 2026, at a Baseten or a Together or anywhere running vLLM in production, you will be asked to explain why it works, when it stops helping, and whether it changes what the model says.

The last question is the one people fumble, so start there.

## No, it does not change the output

It does not, and knowing why is what the question is really checking. Plain speculative decoding is exact: the tokens you get are drawn from the identical distribution the target model would have produced on its own. It is not an approximation and not a quality-for-speed trade like quantization.

The mechanism is a verification step. The draft proposes tokens; the target runs one forward pass over the whole proposed sequence and produces its own probability for each position. In greedy mode you keep a drafted token only if it matches the target's argmax at that position, and you stop at the first mismatch. In sampling mode you accept a drafted token with probability min(1, p_target / p_draft), and when you reject one, you resample that position from the corrected residual distribution. That correction is what makes the whole thing provably equal to sampling from the target directly. Interviewers like this because a candidate who says "it's roughly the same, close enough" has revealed they memorized the pitch instead of the math.


```
draft = small_model.generate(prefix, k=4)     # 4 guessed tokens
logits = target_model(prefix + draft)         # one parallel pass over all 4
for i, tok in enumerate(draft):
    if accept(tok, logits[i]):                # argmax match, or prob test
        prefix.append(tok)
    else:
        prefix.append(sample(corrected(logits[i])))  # fix the first miss
        break
else:
    prefix.append(sample(logits[-1]))         # bonus token if all accepted
```

Notice the target still runs. You did not skip it. You batched several candidate positions into a single call instead of paying for one call per token.

## Why checking four tokens costs about the same as making one

This is the part that makes the whole idea pay off, and it comes straight from where the time goes. Generating one token at batch size 1 is bound by memory bandwidth, not compute. To produce a single token the GPU has to stream every weight of the model out of HBM through the compute units. For a 70B model in fp16 that is 140GB of reads per token. The matrix multiplies themselves finish long before the weights finish loading, so the tensor cores sit mostly idle waiting on memory.

A forward pass over four candidate tokens reads those same weights once and does four positions' worth of arithmetic while the weights are resident. The extra compute is nearly free because you had spare compute the whole time. So the target's cost per verification pass is close to its cost for a single token, but a good draft lets that one pass advance the sequence by two, three, or four positions. That is the entire trick: turn idle compute into accepted tokens.

Say that out loud in an interview and you have already separated yourself from the candidate who describes speculative decoding as "running a small model instead of the big one." You are not running the small model instead. You are running both, and the small one only earns its keep by keeping the big one's rejection rate low.

## The acceptance-rate math they will make you do

Expect a back-of-envelope question. If the per-token acceptance probability is roughly a and you draft g tokens per pass, the expected number of tokens you commit per verification pass works out to (1 - a^(g+1)) / (1 - a), which already includes the one token the target always produces at the end.

Put numbers in. A standalone draft model with a 0.6 acceptance rate drafting 4 tokens gives (1 - 0.6^5) / 0.4, about 2.3 tokens per pass. So you make roughly 2.3 tokens' worth of progress for one big-model call, before you subtract the draft's own cost. Push acceptance to 0.8, which the stronger methods reach on in-distribution work, and the same formula gives about 3.4 tokens per pass. That gap between 2.3 and 3.4 is exactly why the field spent the last two years chasing acceptance rate, and it maps onto the 3-4x throughput numbers vendors quote for the best methods on an H100 or H200.

The follow-up: why not draft 20 tokens instead of 4? Because acceptance compounds against you. Each extra drafted token is only reached if every token before it was accepted, so the marginal token contributes a^k, which decays fast. Draft too far and you spend draft-model time generating tokens that get thrown away, and you inflate the verification pass. There is a sweet spot, usually somewhere between 3 and 8 depending on the method and the workload.

## The methods, and what each one costs you

By 2026 the question is rarely "what is speculative decoding" and more often "which flavor would you pick here." The right answer depends on whether you have a good small sibling model, how much you can change the serving stack, and what your traffic looks like.

| Method | How it drafts tokens | Extra cost to run it | Typical acceptance rate | Best fit |
| --- | --- | --- | --- | --- |
| Separate draft model | A small aligned model (say 1B) generates ahead autoregressively | A second model in GPU memory, must share the target's tokenizer and family | 40-60% on general traffic | When a well-aligned small sibling already exists |
| Medusa | Extra decoding heads bolted on the target predict positions +1, +2, +3 in parallel | Light finetune of the heads, no second model | Solid at short drafts, drops off as the draft lengthens | A quick add-on when you cannot ship a second model |
| EAGLE-3 | Feature-level autoregression that reuses the target's hidden states, plus a draft tree | Train the drafter, more involved serving with tree attention | 75-88% on coding and instruction-following | Highest speedup for general-purpose LLMs in 2026 |
| N-gram / prompt lookup | Copies candidate spans straight from the prompt or prior context, no model at all | Zero training, negligible memory | Very high on repetitive text, near zero on novel text | Code editing, RAG, summarizing text the model was handed |

The n-gram trick deserves a mention because it surprises people. When the model is editing code or answering from a retrieved document, huge spans of the output are copied verbatim from the input. A dumb string match against the context proposes those spans for free, the target verifies them, and you get most of a speculative speedup with no draft model and no training. It is the first thing to reach for in a RAG or code-assistant setting, and naming it signals you have actually watched where the tokens come from.

EAGLE's edge over Medusa is worth understanding rather than reciting. Medusa's heads each predict a future position on their own and do not see each other, so their joint guesses drift out of agreement as the draft grows, which is why acceptance sags at longer draft lengths. EAGLE-3 drafts in the target's own feature space and reuses its hidden states, so the guesses stay coherent with what the target would actually do, and feature fusion across layers is what buys those extra points of acceptance on Llama and Qwen models.

## Where it quietly stops helping

The sharpest interview question in this space is "you turned on speculative decoding and throughput went down, what happened." The answer is almost always batch size.

Everything above assumed the memory-bound regime, which is where you live at batch size 1 or with a handful of concurrent requests. Once you batch many requests together, the target is already doing enough arithmetic per weight load that the tensor cores are busy. Now you are compute-bound, and the spare compute speculative decoding was feeding on no longer exists. Worse, every rejected draft token was real work the GPU did and threw away, so verification of bad guesses becomes pure overhead. At high batch sizes speculative decoding can leave throughput flat or make it slightly worse while still helping per-request latency. A serving system that runs both regimes has to decide, sometimes at runtime, when speculation is worth it.

Two more failure modes come up. A draft model that is misaligned with the target, wrong tokenizer, different fine-tune, out-of-distribution traffic, watches its acceptance rate collapse, and a low-acceptance drafter is worse than no drafter because you pay for it and get little back. And the draft model competes for the same HBM and the same memory bandwidth as the target's KV cache, so on a memory-tight deployment the draft can push your max batch size down and cost you more than it saves.

## How to actually talk about it

The strongest candidates frame speculative decoding as a bet on predictability. When the next few tokens are easy to guess, boilerplate, closing brackets, copied context, common phrasing, a cheap drafter nails them and the target rubber-stamps a batch of them for one pass. When the text is genuinely hard, acceptance drops and you fall back to close to normal decoding speed, having lost only the draft's small overhead. The technique never hurts output quality and mostly helps latency, with the size of the help swinging on acceptance rate and batch size.

If you can walk from the memory-bandwidth argument, through the acceptance formula with a real number in it, to the batch-size caveat, you have shown the thing the interview is checking for: that you know why a serving optimization works and can predict when it will let you down. That reasoning counts for more with the interviewer than remembering that EAGLE-3 beats Medusa, though it helps to know that too.