Fast Inference from Transformers via Speculative Decoding
Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202)
Subscriber deck — free during early access
1 cards in “5 Related work” · whole deck
5 Related work
- How does speculative decoding improve on Blockwise Parallel Decoding (Stern et al., 2018)?Blockwise parallel decoding also produces several tokens per step, but supports only greedy decoding, requires training a custom model, and aims to preserve task quality rather than to guarantee identical outputs; speculative decoding handles general stochastic sampling with unchanged models and an exact guarantee.
- Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §5
Source
- Yaniv Leviathan, Matan Kalman and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202, pp. 19274–19286. https://proceedings.mlr.press/v202/leviathan23a.html