What Is Speculative Decoding? How LLMs Accelerate Text Generation

Large language models typically generate text autoregressively: predict one token, add it to the context, and then predict the next. Even though modern GPUs are extremely powerful at parallel computation, this dependency between consecutive tokens creates an inherently sequential part of LLM inference.

Speculative Decoding is an inference optimization technique for large language models in which a faster, less computationally expensive Draft Model proposes several candidate tokens in advance, and the original Target Model verifies those candidates together. If several are accepted, generation can advance by multiple tokens in one Target Model step instead of progressing one token at a time.

For AI researchers, ML practitioners, developers working on LLM inference, and technically curious readers trying to speed up autoregressive text generation, this makes speculative decoding one of the most practical ways to reduce latency without changing the Target Model or sacrificing output quality when the correct acceptance and sampling rules are used. Rather than replacing a large model with a smaller one, it uses the smaller model to cut the number of expensive sequential decoding steps the Target Model must perform.

From there, the article explains how the Draft/Target pattern works, why autoregressive generation is a bottleneck, how speculative decoding compares with techniques such as KV Cache, which factors determine its performance, including acceptance rate, what its variants change, and where it fits in broader LLM inference optimization.

What Is Speculative Decoding?

Speculative Decoding is a method for accelerating autoregressive inference in large language models, where normal autoregressive generation produces tokens one at a time by cheaply predicting several possible future tokens first and then having the Target Model verify those predictions.

Under conventional autoregressive decoding, generating four tokens generally requires a sequence of generation steps:

T1 → T2 → T3 → T4

T2 depends on T1, T3 depends on the preceding sequence, and so on. The Target Model cannot simply determine all four output tokens independently because each choice changes the context for subsequent predictions.

Speculative Decoding changes how this work is executed by separating prediction from verification in language model inference. A faster Draft Model first proposes several candidate tokens, which is the core of the speculative decoding method. The Target Model can then evaluate the candidate sequence in parallel across positions. If multiple candidates satisfy the acceptance criteria, several tokens can be committed at once, which is why applying speculative decoding is especially useful for latency-sensitive large models.

The technique therefore does not primarily change what the model knows. It is a powerful inference optimization technique in large language models that changes how the Decode phase is executed.

Why Is Autoregressive LLM Generation a Speed Bottleneck?

One of the fundamental bottlenecks in LLM text generation comes from Autoregressive Decoding.

Suppose the Prompt is:

AI models can

The model first predicts another token, such as:

generate

Only after that token has been selected can the model use the updated context:

AI models can generate

to predict the next token. It might then generate "text" before repeating the process again.

This creates a dependency chain between output tokens, where each token depends on earlier tokens in the sequence. The model cannot know exactly what token will appear at position 20 before the earlier tokens have been selected because those earlier choices influence the probability distribution at later positions.

Modern GPUs excel at large parallel matrix operations, but sequential token generation limits how effectively this parallel capacity can be used during Decode. This is particularly relevant for low-batch or latency-sensitive inference, where Decode may not utilize the hardware as efficiently as the Prompt Prefill phase.

Speculative Decoding attempts to reduce the number of sequential Target Model Decode steps required to produce the same output sequence in autoregressive models, helping address both the sequential decode bottleneck and the memory-bandwidth bottleneck in autoregressive text generation.

How Does Speculative Decoding Work?

A typical Speculative Decoding system has two main components: separate draft and target models.

The Draft Model quickly generates candidate tokens and quickly guesses a sequence of future tokens, while the Target Model remains responsible for validating the candidates and determining the final output.

Suppose the current input context is:

Artificial intelligence is

As an example, the Draft Model might quickly propose these draft tokens:

changing → the → way → people

Practical systems often generate 3 to 12 candidate tokens in a speculative round.

Instead of having the Target Model generate those four positions one by one, the Target Model evaluates the candidate sequence and computes the relevant token distributions for those positions.

If the first three candidate tokens are accepted but the fourth fails the acceptance rule, the system can retain the accepted prefix:

changing → the → way

At the rejection point, the algorithm determines the appropriate replacement token rather than relying on a separate draft model, and then begins another speculative round from the updated context.

The overall process can be represented as:

Current Context → Draft Model Proposes Tokens → Target Model Verifies Candidates → Accept Valid Prefix → Correct at Rejection Point → Repeat

The main source of acceleration is straightforward: one Target Model verification can potentially advance the output by multiple tokens.

What Do the Draft Model and Target Model Do?

Understanding the roles of the two models is central to understanding Speculative Decoding.

The Target Model is the model whose output distribution the system ultimately wants to preserve. If a user originally intended to run a particular large LLM, that model remains the Target Model.

The Draft Model performs the speculative work. It is usually faster or less computationally expensive and attempts to predict tokens that the Target Model is likely to accept.

This means the smallest possible Draft Model is not necessarily the best choice, though a smaller version used as the Draft Model can sometimes deliver up to 3x speed improvements. A very fast model that frequently proposes candidates the Target Model rejects may provide little benefit.

An effective Draft Model needs to balance two properties: the draft and target models should be closely aligned, and the Draft Model must be significantly cheaper than the Target Model to be effective. Fine-tuning or distilling the Draft Model against the base model can improve acceptance rates by about 15% and, in some cases, improve inference speed by 3x.

How Does the Target Model Verify Multiple Tokens?

This is one of the most easily misunderstood parts of Speculative Decoding.

The Target Model does not simply look at the Draft Model’s proposed sentence and decide whether it is "correct." Instead, it computes token probabilities at the candidate positions and uses a rejection sampling algorithm for token acceptance.

Suppose the Draft Model proposes:

A → B → C → D

Because Transformer inference lets the target model processes the candidate sequence across positions in parallel, it can verify multiple positions in a single forward pass and obtain the relevant distributions for these candidate positions.

The system then evaluates candidates sequentially from the beginning. If A, B, and C are accepted but D is rejected, the accepted prefix can be retained:

A → B → C

It generally cannot simply skip D and accept a later candidate, because changing D also changes the context on which later predictions depend.

In classical speculative sampling, acceptance and rejection are based on the relationship between the Draft and Target probability distributions, with corrective sampling when necessary, so the final accepted sequence matches what the model would sample from its own distribution. The process is more sophisticated than checking whether the two models have the same Top-1 token.

This probability correction is what allows standard speculative sampling algorithms to preserve the Target Model’s intended sampling distribution by adjusting target probabilities after rejection.

Why Can Speculative Decoding Make LLMs Faster?

The potential speedup comes from combining low-cost speculation with parallel verification.

Under ordinary decoding, generating 100 tokens with a Target Model requires many sequential Decode steps. KV Cache can prevent the model from recomputing historical K/V projections, but new output tokens still need to be generated sequentially.

Speculative Decoding lets the Draft Model perform some of this prediction work at lower cost. If the Target Model accepts several candidates during each verification round, the sequence can advance by more than one token, with each accepted additional token reducing the number of expensive sequential Target Model steps required.

A simplified comparison looks like this:

Standard Decoding

Target → 1 Token Target → 1 Token Target → 1 Token Target → 1 Token

Speculative Decoding

Draft → 4 Candidate Tokens Target → Verify Candidates → Accept Multiple Tokens with one forward pass

This does not mean that all four tokens will always be accepted, nor does it imply a fourfold speed increase. Real performance depends on candidate acceptance rate, Draft Model cost, Target Model size, hardware utilization, and verification overhead.

The real question is whether the expensive Target Model work saved is greater than the additional cost of drafting and verification. In practice, speculative decoding offers meaningful latency gains, and benchmark-style results often report speedups around 2.3x to 3.4x, though actual gains still vary.

Why Does Acceptance Rate Matter?

Acceptance Rate describes how often the draft tokens proposed by the Draft Model are accepted by the Target Model, and that acceptance rate directly influences the overall speedup of the process. It is one of the most important factors affecting Speculative Decoding performance.

If the Draft Model closely predicts the Target Model’s behavior, a single verification round may accept several tokens. The sequence can then advance quickly, with higher acceptance translating more directly into overall speedup.

If the two models behave very differently, the Target Model may reject candidates early. Much of the Draft Model’s work then fails to become accepted output, reducing the benefit of speculation.

There is also a trade-off in how many tokens the Draft Model should propose tokens at once. A longer speculative sequence creates the possibility of advancing further during one Target Model verification, but uncertainty generally increases farther into the future, which can reduce the probability that later candidates are accepted.

Practical systems therefore need to balance Draft Speed, Acceptance Rate, and Speculation Length, rather than simply asking the Draft Model to generate as many tokens as possible.

Does Speculative Decoding Reduce LLM Output Quality?

Standard Speculative Decoding and speculative sampling algorithms are designed to preserve the Target Model’s original output distribution when the correct verification and probability-correction procedures are used.

This distinction matters.

Simply allowing a smaller model to generate text and asking a larger model to perform a lightweight check could change the final output distribution. Classical Speculative Decoding instead uses the Target Model’s probability distribution to accept, reject, and, when necessary, resample tokens.

The objective is therefore to change how the output is computed, not to substitute the Draft Model’s generation quality for that of the Target Model.

However, the term "speculative decoding" is also used broadly for a growing family of inference techniques. Not every approximate implementation necessarily provides the same theoretical distribution-preservation guarantees. Some systems may deliberately trade exactness for additional speed.

Whether a specific implementation affects output behavior therefore depends on its verification and sampling algorithm.

Speculative Decoding vs. KV Cache: What Is the Difference?

Speculative Decoding and KV Cache are both important LLM inference optimizations, but they address different bottlenecks.

KV Cache addresses repeated computation over historical tokens. It stores previously calculated Attention Keys and Values so that new tokens can reuse those representations.

Speculative Decoding addresses the sequential nature of autoregressive generation. It attempts to predict several future tokens in advance and validate them together, reducing the number of expensive sequential Target Model steps.

The two techniques are complementary rather than competing. An inference system can use KV Cache while also performing Speculative Decoding.

Speculative Decoding vs. Standard Decoding: What Is the Difference?

Standard autoregressive decoding relies on the Target Model to advance generation step by step. Whether the system uses Greedy Decoding, Temperature, Top-k, or Top-p sampling, the basic process remains sequential generation.

Speculative Decoding adds a speculation stage. This is how speculative decoding works: the Draft Model proposes several future candidates before the Target Model verifies or rejects them.

It usually works best with small batch sizes, where latency matters more than throughput.

Speculative Decoding is therefore best understood as an Inference Optimization technique, rather than a new model-training method, and its latency improvements can be substantial, with favorable deployments dropping latency from 600 ms to 250 ms for latency-sensitive uses like chatbots and coding assistants.

Does Speculative Decoding Always Make LLMs Faster?

No. Its effectiveness depends heavily on the model pair, hardware, and workload.

Speculative Decoding is more likely to provide meaningful gains when the Target Model is expensive, the Draft Model is substantially cheaper, and the Draft Model achieves a high acceptance rate.

If the Draft Model itself requires substantial computation or frequently proposes tokens that are rejected, the additional drafting and verification work may offset the saved Target Model steps.

Hardware utilization also matters. In high-batch, throughput-oriented inference systems, GPUs may already be heavily utilized, so the performance characteristics can differ from low-batch, latency-sensitive workloads, and larger batch sizes can reduce the latency advantage and limit throughput gains in wall clock time.

Model size, memory bandwidth, context length, speculative sequence length, and serving architecture can all influence the result.

There is therefore no universal speedup factor associated with Speculative Decoding. Its benefits need to be evaluated for the specific model, infrastructure, and workload. It is especially useful in latency-sensitive systems such as multi-step AI agents and reasoning chains, where per-request delay matters.

Are There Other Types of Speculative Decoding?

The classical approach uses a separate smaller Draft Model, but the broader idea of speculate first, verify later also includes variants that do not rely on a separate draft component.

One direction is Self-Speculative Decoding, where a separate small model may not be required. Instead, these approaches reuse parts of a single model, using portions of the Target Model, early-exit mechanisms, or another lightweight path to generate candidates, with verification still anchored to the target model’s output.

Another approach involves Multi-Token Prediction or additional prediction heads that attempt to propose several future tokens before the main model verifies them. EAGLE-3 is one example that uses multi-layer fused feature representations to propose tokens efficiently.

Some techniques also construct tree-like candidate structures rather than predicting only one speculative sequence. Multiple possible future branches can then be evaluated, increasing the chance that the Target Model finds an acceptable path.

The algorithms differ considerably, but they share a common objective: reduce the number of expensive sequential Decode operations and move more work into parallel computation.

How Does Speculative Decoding Fit Into LLM Inference Optimization?

Speculative Decoding is one component of a much broader LLM inference optimization stack.

LLM performance can be limited by computation, GPU memory, memory bandwidth, request scheduling, and the sequential nature of autoregressive generation. Modern serving systems therefore combine multiple optimization techniques.

KV Cache reduces repeated K/V computation for historical tokens. Paged Attention improves how KV Cache memory is managed. Continuous Batching helps multiple requests use GPU resources efficiently. Quantization can reduce model or cache memory requirements and computational cost. Speculative Decoding specifically targets the sequential Decode bottleneck.

These technologies solve different problems and can often be used together.

This also explains why real-world LLM response speed is not determined by parameter count alone. Model architecture, Attention design, KV Cache, decoding algorithms, GPU hardware, inference engines, and request scheduling all contribute to final inference performance.

Summary

Speculative Decoding is an LLM inference optimization designed to reduce the bottleneck created by autoregressive token generation. In the classical approach, a lower-cost Draft Model proposes several candidate tokens before the Target Model verifies them, potentially allowing multiple output tokens to be accepted during a single Target Model verification step.

Its effectiveness depends heavily on how accurately the Draft Model predicts candidates that the Target Model will accept. A high Acceptance Rate can reduce the number of expensive sequential Decode steps, while frequent rejections can diminish the benefit by adding unnecessary Draft and verification work.

Unlike KV Cache, which reduces repeated computation over historical Attention states, Speculative Decoding focuses on reducing the sequential bottleneck of generating tokens one at a time. The two techniques can be combined with other inference optimizations such as Continuous Batching, Paged Attention, and Quantization.

Speculative Decoding highlights a broader principle in modern AI infrastructure: making an LLM faster does not always require using a smaller model or sacrificing output quality. Improving the inference and decoding process itself can make existing models generate text more efficiently.

FAQ

Does Speculative Decoding always require two models?

Classical Speculative Decoding typically uses a Draft Model and a Target Model, but variants such as Self-Speculative Decoding and additional prediction-head approaches can generate candidates differently.

Is a smaller Draft Model always better?

No. The Draft Model needs to be fast while remaining sufficiently aligned with the Target Model’s predictions. If it is too weak and produces a low Acceptance Rate, the overall speedup may decline.

Does Speculative Decoding modify the Target Model?

No. It is an inference and decoding optimization and does not require changing the Target Model’s trained parameters.

Can Speculative Decoding and KV Cache be used together?

Yes. KV Cache reduces repeated Attention computation over historical tokens, while Speculative Decoding reduces expensive sequential Decode steps, so the two techniques can complement each other.

Why doesn’t Speculative Decoding provide a fixed speedup?

Actual performance depends on factors including Draft Model cost, Acceptance Rate, speculation length, Target Model size, GPU hardware, Batch Size, and workload characteristics.