Speculative decoding changes how tokens are proposed and checked. Correct verification is essential, and the extra work must be measured against a strong target-only baseline.
The sequential bottleneck
Ordinary autoregressive generation produces one token, adds it to the prefix, and runs the next decoding step. Speculative decoding introduces a cheaper proposal mechanism: draft several tokens, then let the target model check that block. The foundational work shows how an acceptance-and-correction procedure can preserve the target sampling distribution. [S1]
The smaller model is doing tentative work. Its output is not automatically approved because it sounds plausible. Verification determines which proposed tokens can remain in the sequence.
A short hypothetical block
Imagine a draft proposes three token IDs: A, B, and C. Verification accepts A but rejects B. The algorithm keeps the accepted prefix and supplies a correction at the rejected position. C was proposed after B, so it does not belong to the newly corrected continuation.
This is an explanatory example, not a claim that words map one-to-one to tokens or that a real model would make these exact choices. A useful implementation test uses explicit token IDs and checks the resulting prefix after every rejection position.
Sampling needs a correction rule
Let p be the target distribution and q the draft distribution for the same prefix, after the intended sampling transformations. For a token proposed from q, the standard procedure accepts it with probability min(1, p/q). On rejection, it samples from a normalized positive residual based on p minus q. This correction, rather than simple agreement testing, is what preserves the distribution in the ideal algorithm. [S2]
Greedy decoding is a different comparison: the goal is to preserve the target's argmax decisions. Do not use a greedy-match check as proof that a stochastic sampler is correct. Temperature, filtering, normalization, numerical behavior, and any implementation approximations need to be accounted for.
The cache must agree with the accepted prefix
The model state after a draft block must correspond to the tokens that actually survive. In the toy example, retaining state for the discarded continuation would leave the next step conditioned on the wrong history.
For an implementation review, I would test rejection of the first token, rejection in the middle, full acceptance, a one-token draft, and end-of-sequence handling. I would check both the returned text and the cache positions. These are proposed engineering checks, not evidence that an untested implementation is correct.
Acceptance is useful, but incomplete
Suppose two hypothetical implementations accept the same average number of draft tokens. One uses a much more expensive draft model and spends additional time moving state. Their acceptance rates can match while their request latencies differ.
The relevant budget includes drafting, target verification, sampling, state management, and any communication. Current vLLM documentation discusses speculative decoding in the context of memory-bound serving workloads and describes method selection as workload dependent. [S3]
Measure the request a reader will experience
I would compare a cached target-only baseline with the speculative implementation on the same prompts, output lengths, hardware, and sampling settings. The report should separate prompt processing from generation and include whole-request latency, token latency, throughput, and concurrency.
An isolated request can benefit from a strategy that behaves differently when many requests compete for resources. A convincing result therefore tells you which workload improved. Fewer target calls are a mechanism; lower measured latency for the intended workload is the outcome to establish.