Fast Inference from Transformers via Speculative Decoding

3.6 Approximation Models

1. What kinds of approximation models can speculative decoding use?
   Any \(M_q\) preserves correctness. Off-the-shelf smaller Transformers about two orders of magnitude smaller than \(M_p\) worked best; negligible-cost options include n-gram models, copying from the context (for summarization or iterative editing), non-autoregressive models, and even random tokens.
   • Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §3.6
2. What did a bigram approximation model achieve for T5-XXL translation, and why is that notable?
   \(\alpha \approx 0.2\), which with \(c \approx 0\) and \(\gamma = 3\) gives about a 1.25× speedup: a trivial table-lookup model already helps, because its cost is negligible.
   • Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §3.6