Engineering and empirically evaluating prompt-lookup decoding, rolling-hash speculative memory, and two-model speculative decoding in MLX-LM on Apple Silicon
Speculative decoding is typically framed around a two-model setup: a smaller draft model proposes candidate tokens and a larger target model verifies them in parallel. When draft-token acceptance is high, the target model produces multiple tokens per forward pass and throughput increases substantially. When acceptance is low, the overhead of the draft model dominates and throughput falls below baseline.
This post documents an implementation of prompt-lookup decoding (PLD) in MLX-LM, including both ngram-simple and ngram-mod. Unlike draft-model speculative decoding, these approaches do not require a second neural model. Instead, speculative drafts are generated directly from previously observed token trajectories.
An n-gram simply refers to a sequence of n consecutive tokens, such as a 3-token or 16-token window.
Two speculative strategies were implemented:
ngram-simple: exact prompt-history lookupngram-mod: rolling-hash associative memory with process-global shared stateThe work extends a generalized DraftStrategy abstraction added to MLX-LM and evaluates the practical behavior of both approaches on Apple Silicon across short-turn editing workloads, long-context conversations, a direct comparison against classic two-model speculative decoding, and cross-model warm-start experiments.
The most interesting result was not the raw throughput gain itself, but that token-space reuse could outperform a much higher-acceptance two-model draft setup on edit-heavy workloads, while also enabling runtime-learned speculative memory shared across requests and reusable across models with compatible tokenizers.
The simpler of the two implementations, ngram-simple, performs exact prompt-history lookup.
The mechanism is straightforward: the decoder maintains a running token history, extracts the most recent n-token window, and searches backward through the earlier history to find a prior occurrence of that same window. When a match is found, it copies up to num_draft_tokens tokens that followed the earlier occurrence as a speculative continuation and submits those tokens to the verifier, which accepts the matching prefix in bulk and falls back to normal generation at the first mismatch.
In effect, the draft comes from reusing previously observed token trajectories rather than from a second neural model, which makes the approach especially effective when the prompt contains recurring patterns.
For repetitive workloads such as code editing, structured generation, or iterative refinement, token trajectories recur frequently enough that the verifier can often accept several speculative tokens in bulk.
Unlike draft-model speculative decoding, no neural forward pass is required to produce the draft itself. The speculative tokens are retrieved directly from prior token history.