Speculative decoding makes a large language model generate text faster by pairing it with a smaller, cheaper model that guesses ahead. The small model drafts several tokens at once. The large model checks that draft in a single pass instead of writing one token at a time. That shift from writing to checking is the entire trick, and it is why some AI products feel instant while others crawl.
Every large language model, left alone, generates text one token at a time. It predicts a token, appends it, feeds the whole sequence back in, and predicts the next one. This is called autoregressive decoding. It is slow by design because each new token depends on every token before it. Speculative decoding does not change this math. It just finds a way to skip the expensive part of it most of the time.
Picture a writer and an editor. The writer drafts five sentences fast and loose. The editor reads all five at once and either approves them or stops at the first mistake and fixes it. The editor never writes from scratch unless the draft goes wrong. That is the whole system.
In technical terms, a small "draft" model generates a handful of candidate tokens, usually four to eight at a time. The large "target" model then runs a single forward pass over all of those candidate tokens simultaneously, something a transformer is very good at. It compares its own predicted probabilities against what the draft model guessed. Tokens that match within an acceptable range get accepted immediately. The first token that does not match gets thrown out, and the target model generates the correct one itself.
Nothing here lowers the quality bar. The target model still makes every real decision. It just makes several decisions in parallel instead of one at a time, because checking a sequence is cheaper than producing it token by token.
The draft model needs to be fast and roughly aligned with the target model's style. It does not need to be smart. A 1-billion-parameter model can draft for a 70-billion-parameter target as long as the two were trained on similar data or the draft model was distilled from the target. Mismatched pairs waste cycles because the draft keeps guessing wrong.
| Role | Draft Model | Target Model |
|---|---|---|
| Size | Small, cheap | Large, expensive |
| Job | Guess several tokens fast | Verify guesses in one pass |
| Runs | Many times per cycle | Once per batch of guesses |
| Determines final output | No | Yes |
Some setups skip the separate draft model entirely and use the target model's own earlier layers as a shortcut predictor, a technique called self-speculative decoding. Google and several open-source labs use variants of this because it avoids the cost of maintaining and syncing two separate models in production.
This is the part people get wrong. Running a model is not cheap because of the math per token. It is cheap or expensive based on memory bandwidth. Every time a model generates one token, it has to pull its entire set of weights, sometimes hundreds of gigabytes, from GPU memory. That transfer is the bottleneck, not the computation itself.
Verifying five tokens at once requires pulling those same weights into memory exactly once, then running the comparison across all five positions in parallel. One memory pull instead of five. The GPU does more math per pull, but GPUs have math to spare. What they do not have to spare is memory bandwidth. Speculative decoding trades an operation GPUs are bad at hiding the cost of, repeated memory access, for one they are great at, parallel computation.
This is why the speedup numbers look almost too good. Depending on the task and the draft model's accuracy, teams report 2x to 3x faster generation with zero change in output, because the target model is still the one making every final call.
The entire gain depends on one number: how often the draft model guesses right. This is called the acceptance rate. If the draft model nails 80% of its guesses, you get a real speedup. If it nails 20%, you barely save anything, because the target model keeps rejecting the draft and falling back to writing tokens itself.
Acceptance rate is not fixed. It moves with the type of text being generated.
Teams tune the draft window length based on this tradeoff. Too short and you barely beat standard decoding. Too long and rejection rates eat the gains. Most production systems land somewhere between four and six speculative tokens per round.
It does not help everywhere, and anyone selling it as a universal fix is wrong. Three cases blunt its advantage.
First, very short outputs. If a user asks a one-word question and gets a one-word answer, there is nothing to speculate ahead on. The setup overhead outweighs the gain.
Second, batch serving at high concurrency. Speculative decoding shines when a GPU has spare compute sitting idle during memory-bound generation. Under heavy load with many simultaneous users, that spare compute is already being used for other requests, so there is less idle capacity to exploit.
Third, tasks with naturally low predictability, like open-ended creative writing or novel mathematical reasoning where the next token genuinely could go several directions. The draft model's guesses land less often, acceptance rate drops, and the whole system edges back toward plain autoregressive speed.
None of this makes the technique weak. It makes it situational, which is true of almost every inference optimization in this field. The mistake is assuming one trick solves every workload.
If you are calling a hosted model through an API, you likely already benefit from speculative decoding without knowing it. Major providers quietly run it behind the scenes on specific model tiers because it lowers their own compute cost per request, and some of that savings shows up as lower latency, sometimes as lower price.
If you are running your own model, the calculation changes. Deploying speculative decoding means maintaining two models in sync, re-validating the draft model whenever you fine-tune the target, and tuning the draft window per workload. For a team serving one steady use case with predictable text patterns, like customer support scripts or code completion, the setup pays for itself fast. For a team running varied, creative workloads, the engineering overhead may not be worth the modest gain.
My take: speculative decoding is the most underrated lever in inference optimization right now, more so than quantization, because it costs nothing in output quality. Quantization trades some precision for speed. Speculative decoding trades nothing. The only price is engineering complexity, and for any team serving millions of requests a day, that price is small next to the GPU bill it cuts.
Speculative decoding is a technique where a small, fast model drafts several candidate tokens ahead, and the large target model checks them all in a single pass. Accepted tokens are kept, and the first wrong guess is replaced by the target model's own prediction. It speeds up text generation without changing the final output.
No. The target model still approves or rejects every token, so the final text is identical to what standard decoding would have produced. The draft model only proposes guesses, it never has the final say, which is what keeps output quality unchanged.
Reported speedups typically range from 2x to 3x, depending on the draft model's accuracy and the type of text being generated. Predictable text like code sees larger gains, while open-ended creative writing sees smaller ones because the draft model guesses wrong more often.
The draft model is small and fast, and its job is to guess several upcoming tokens quickly without needing to be highly accurate. The target model is the large, expensive model that actually determines the final output by verifying or rejecting the draft model's guesses in one parallel pass.
Its speedup depends on acceptance rate, meaning how often the draft model guesses correctly, and that rate drops for unpredictable or highly creative text. It also loses its edge under heavy concurrent load, since it relies on spare GPU compute that busy servers may not have available.