Fast Inference from Transformers via Speculative Decoding
Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202)
Subscriber deck — free during early access
1 cards in “4.1 Empirical Walltime Improvement” · whole deck
4.1 Empirical Walltime Improvement
- What speedups did speculative decoding achieve on T5-XXL (11B)?With T5-small (77M) as approximation model on a TPU-v4 at batch size 1: 3.4× (argmax) and 2.6× (sampling) on English–German translation, 3.1× and 2.3× on CNN/DM summarization, with identical outputs; larger approximation models gave higher \(\alpha\) but less speedup.
- Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §4.1
Source
- Yaniv Leviathan, Matan Kalman and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202, pp. 19274–19286. https://proceedings.mlr.press/v202/leviathan23a.html