Fast Inference from Transformers via Speculative Decoding
Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202)
Subscriber deck — free during early access
2 cards in “2.1 Overview” · whole deck
2.1 Overview
- Outline one step of speculative decoding.(1) The approximation model \(M_q\) generates \(\gamma\) guesses autoregressively; (2) the target model \(M_p\) scores all of them in one parallel run; (3) accept the longest prefix of guesses allowed by speculative sampling, then sample one more token from an adjusted distribution to replace the first rejected guess (or to extend the sequence if all were accepted).
Three tokens for one run of the large model (invented example, γ = 4): the small model drafts, the large model verifies - Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §2.1
- Why can speculative decoding never need more serial runs of the target model than ordinary decoding?Every parallel run of \(M_p\) yields at least one new token (the correction or the extra sample), and up to \(\gamma + 1\) tokens if \(M_q\) guesses well.
- Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §2.1
Source
- Yaniv Leviathan, Matan Kalman and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202, pp. 19274–19286. https://proceedings.mlr.press/v202/leviathan23a.html