speculative decoding

Speculative decoding is an inference technique that speeds up autoregressive generation by having a fast draft model propose several tokens at once, which the larger target model then verifies in a single forward pass.

Generating one token at a time leaves a large model bound by memory bandwidth rather than compute, because every step reads all of its weights to produce a single token. Scoring a short batch of proposed tokens costs about as much as generating one, so verification is close to free.

The target model compares its own probabilities against the draft’s at each proposed position. It accepts the longest prefix that a modified rejection-sampling rule allows, discards the remaining proposals, and samples a correction token in place of the first rejected one.

That rule preserves the target model’s output distribution exactly, so the technique makes generation faster without changing what it produces. The payoff tracks the acceptance rate: predictable text runs roughly two to three times faster, while a poorly matched draft can end up slower than plain decoding.

The simulation below runs one propose-and-verify round at a time, marking which proposals survive and counting how many tokens each target pass buys.

Interactive diagram — enable JavaScript to view.

Proposals come from several sources:

  • A separate smaller model from the same family, sharing the target’s tokenizer
  • A lightweight draft head trained on the target model’s own hidden states, as in Medusa and EAGLE
  • N-gram or prompt lookup, which copies repeated spans straight out of the prompt

Serving stacks such as vLLM, TensorRT-LLM, and llama.cpp ship more than one of these. The gain is largest at low request volume, where spare compute sits idle. Under heavy batching the hardware is already saturated, and verification competes with real requests for throughput.

How to Run Large Language Models Locally With Ollama

Tutorial

How to Use Ollama to Run Large Language Models Locally

Learn how to use Ollama to run large language models locally. Install it, pull models, and start chatting from your terminal without needing API keys.

intermediate ai tools

For additional information on related topics, take a look at the following resources:

Have a question about this? Mentor AI can show you examples, compare related terms, and point you to tutorials.

  • Ask Mentor AI
  • Pick a question and Mentor AI takes it from here.
By Martin Breuss • Updated Oct. 1, 2026
AI Coding Glossary Share Feedback