Fast Inference from Transformers via Speculative Decoding
Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202)
Subscriber deck — free during early access
1 cards in “3.3 Walltime Improvement” · whole deck
3.3 Walltime Improvement
- What walltime improvement does speculative decoding give, and when does it help at all?With cost coefficient \(c\) (time of one \(M_q\) run over one \(M_p\) run), the expected speedup is \(\frac{1 - \alpha^{\gamma+1}}{(1 - \alpha)(\gamma c + 1)}\), assuming the \(\gamma + 1\) parallel runs of \(M_p\) don’t increase walltime. Some \(\gamma\) gives a speedup whenever \(\alpha > c\).
- Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §3.3
Source
- Yaniv Leviathan, Matan Kalman and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202, pp. 19274–19286. https://proceedings.mlr.press/v202/leviathan23a.html