Fast Inference from Transformers via Speculative Decoding
Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202)
Subscriber deck — free during early access
1 cards in “3.5 Choosing γ” · whole deck
3.5 Choosing γ
- How should the number of guesses \(\gamma\) be chosen?Pick the integer maximizing the walltime improvement for the given \(\alpha\) and \(c\); it grows with \(\alpha\) and shrinks as \(c\) grows. Varying \(\gamma\) on the fly by predicting \(\beta\) could help further, which they leave for future work.
- Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §3.5
Source
- Yaniv Leviathan, Matan Kalman and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202, pp. 19274–19286. https://proceedings.mlr.press/v202/leviathan23a.html