Speculative decoding is the standard trick for making LLM inference faster: a small "draft" model guesses several tokens ahead, and a big "target" model checks them all in a single forward pass. Every guess that matches what the target would have said can be accepted without another sequential target-model step.
Say the prompt so far is "Why did the chicken" and the small model guesses eight tokens ahead:
cross · the · farm · ? · it · wanted · to · go
The big model checks all eight at once. It agrees with cross and the, but
where the draft said farm it wanted road. So it commits cross the
road. That is two accepted draft tokens plus the correction. Then it
throws away everything after the mistake. That was the part that bothered me. ? it wanted to go
is discarded without ever being looked at.
I called the idea Rejoin Speculative Decoding. How it works is that it patches in road, hold
? it wanted to go in escrow, and check whether it still fits.
What Rejoin has to beat
Reusing the discarded tail is only useful if it beats asking the draft model to make a new proposal from the corrected prefix. Repairing the escrow costs a target forward pass, and asking the draft to guess again also leads to a target verification pass. The expensive part of the work is the same in both cases.
So Rejoin only wins if the escrow is a better proposal than a fresh draft, or if the saved draft work makes up the difference. I couldn't find a measurement that made that comparison directly, so that became the experiment.
Whats makes this possible at all
Before getting to the measurements, there's one implementation detail worth explaining. It's easy to get wrong.
Transformers don't re-read the whole sentence for every new token. They keep a KV cache, which stores key and value representations for tokens already processed. Speculative decoding verifies eight tokens in one pass, so it writes eight cache entries at once.
But the cache entry for farm is invalid because farm was
rejected. Every later entry was computed from a prefix containing farm, so
those entries are invalid too. All six have to be removed.
The fix is one line: crop the cache back to just before the rejection. The accepted entries survive untouched because the model saw exactly those words in that order. There is nothing to redo. The correction has no entry yet, so it becomes the first token of the next forward pass and picks one up there.
I'm dwelling on this because the failure mode is nasty: keep one stale entry and the model silently attends to state derived from a token the target rejected. The output stays fluent and nothing crashes but you just get quietly wrong numbers. I wrote mock-model tests that assert the committed output is byte-identical to plain greedy decoding specifically so this couldn't hide.
First run: a dead end
With that in place, I ran Qwen3-4B (target) + Qwen3-0.6B (draft) on HumanEval, greedy throughout, so the committed output is bit-identical to plain target decoding no matter what the instrumentation does.
The first result was delta = survival − fresh_draft_acceptance =
−8.05, averaged over 60 rejection events. There were zero wins. The escrow did worse than redrafting every single time.
Three bugs, in the order I found them
Bug 1: the format artifact. 27% of all rejection events were the exact same thing: the instruct-tuned draft opened a Python code fence and retyped the function's docstring, while the target skipped straight to the function body. That isn't the draft failing to predict the target's reasoning. It's a stylistic disagreement about whether to use a fence at all, and it landed in the bucket with the longest possible escrow and zero survival every time. Fix: commit the fence and the docstring as a prefix before either model generates, so the disagreement can't happen.
Bug 2: I was only asking the strictest possible question. The escrow's first token comes immediately after the token that was rejected. It is the single most contaminated position in the escrow. Asking "does it match starting exactly there" can't tell you "the plan died" apart from "the plan is fine, it just needs three tokens to resync." So I added a second measurement: let the escrow reattach after a short bridge of the target's own tokens, and separately, find the longest run the escrow shares with the target's actual continuation anywhere, with slack on both sides.
Bug 3: the branch-pass measurement was pure waste. Every committed token is the target's greedy choice by construction. That means the committed text after a rejection event is already the target's greedy continuation from the corrected prefix. I didn't need to spend an extra forward pass re-deriving it. I was paying for information I already had. Once I saw that, survival and alignment both became free, retroactively, from data already being logged. It's also what unlocked the fix for bug 2: with the full continuation on hand, I could search offsets instead of only checking one.
What the fixed measurement actually shows
Across ~10,700 rejection events (HumanEval, GSM8K with full reasoning chains, and a long-block sweep), I paired each escrow with the wrong continuation as a chance control:
| escrow vs. correct continuation | escrow vs. wrong continuation (chance) | |
|---|---|---|
| share a 4+ token run | 44–62% | 0.8–2.4% |
| share an 8+ token run | 15–31% | 0–0.3% |
That's a 30–56× ratio. The escrow's content really does survive. I was wrong to think rejected suffixes were just garbage or semantic dead ends. At this point, I expected the idea to start working.
Why it still loses
It survives, but shifted. Go back to the chicken. The escrow held ? it wanted to
go. What the target actually wrote was:
? · well · it · wanted · to · go
it wanted to go was right all along. The target just said well first. A
head-on comparison scores that escrow 1, not 5. It matches the ?, hits
it against well, and stops.
That one-word slip drives the result. The ways of catching it are less useful than they first appear:
Dropping the escrow's stale first token doesn't help on its own. Now it
lands against well instead. Skipping the target's extra token doesn't help on
its own either. Now ? lands against it. You have to do both at once.
In the real data, 78% of surviving runs needed exactly that. But knowing to skip
well means already knowing the target was going to say well, which is the
thing you were trying not to compute.
That leaves four approaches. The key question is whether each one can run during live decoding, when the next target tokens do not exist yet:
| approach | extra information needed | tokens recovered per rejection | usable in live decoding? |
|---|---|---|---|
| reattach at position 0 (original idea) | nothing extra | 0.06–0.07 tokens | yes |
| batched candidate tree, trim the front | nothing extra | 0.12–0.22 tokens | yes |
| let the target bridge a few tokens first | the target's future output | 0.34–0.52 tokens | no |
| best alignment, slack on both sides | the target's future output | 1.55–3.03 tokens | no |
The last two methods are useful measurements, but not live policies. They compare the escrow with target tokens that are only known after the target has generated them. They can tell us what survived, but they cannot save that computation at the moment a decision has to be made.
Break-even sits around 0.72–0.81 tokens, even under an optimistic batched verification cost. Only the two methods that use future target output clear it. The stronger live method falls short by about 3–7×, while direct reattachment falls short by about 10–14×.
I also checked whether being selective would save it: run the escrow only when some cheap, observable signal at rejection time (how wrong the draft was, how long the escrow is, how confident the target is) says it's likely to survive. It won't. Running the policy less often scales both the total cost and the total benefit by coverage; a useful gate has to select events whose conditional gain clears the same per-event bar. I tested twenty such signals. None cleared the bar. Even an oracle gate, one that magically knows in advance which events will win, caps out at +1.3% to +2.5% overall because only about 5–7% of rejections have any winnable margin to select for in the first place.
I also checked the two conditions my own theory predicted should be the strongest case: long chain-of-thought reasoning (more structural repetition, so the plan should resync more easily) and longer speculative blocks (more room for the plan to resume within). Neither helped. Reasoning chains gave the same numbers as short Python functions. Longer blocks produced a longer absolute surviving run, but only because the escrow itself got longer. Survival as a fraction of the escrow actually dropped, from 39% down to 31%.
Why direct reattachment is structurally disadvantaged
There's a structural reason underneath all of this. Once I understood it, I
stopped looking for a smarter attachment policy. The escrow was generated
under P(continuation | prefix + the token the target just rejected). A fresh
draft is generated under P(continuation | prefix + the token the target
actually chose). Same draft model, but the fresh proposal is conditioned on
the corrected prefix, while the escrow was conditioned on a prefix the target
rejected. Without additional signal, there is no reason to expect the
worse-conditioned proposal to beat the fresh one on average. This isn't just a
policy bug. It's baked into the setup.
What I'd keep, and what I wouldn't try again
I wouldn't try to make direct suffix reattachment work. The arithmetic, not just the current numbers, says no. But the survival-above-chance result is real, and it's a small instance of a known phenomenon (recently generated n-grams tend to recur), which is a more interesting takeaway than the flat null I started with. If I revisited this space, I'd look at what sets the break-even bar. A weaker draft or a much larger target lowers the required gain, and I don't yet know whether the escrow degrades faster or slower than that bar does.
Caveats
One model pair (Qwen3-4B / 0.6B), greedy decoding only, two block lengths. ~10,700 events sounds like a lot, but they're drawn from under 300 prompts, so they're correlated and the true uncertainty is wider than the raw count suggests. The strongest live policy still misses break-even by about 3–7×, so I don't think a wider interval changes the conclusion. But I'm not claiming this generalizes past what I tested.