Fast Inference from Transformers via Speculative Decoding

3.4 Number of Arithmetic Operations

1. What does speculative decoding cost in exchange for lower latency?
   More arithmetic: rejected guesses waste the parallel computation of \(M_p\), so total operations grow (e.g. 1.6× for a 3.7× speedup at \(\alpha = 0.8, \gamma = 5\)). Memory accesses for the target model’s weights and KV cache, however, go down, since they are read once per step rather than once per token.
   • Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §3.4
2. When is speculative decoding not helpful?
   When no spare compute is available: it trades more arithmetic operations for fewer serial steps, so it pays off mainly when decoding is memory-bandwidth bound.
   • Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §3.4