Fast Inference from Transformers via Speculative Decoding
Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202)
Subscriber deck — free during early access
1 cards in “4.2 Empirical α Values” · whole deck
4.2 Empirical α Values
- What patterns appear in the measured \(\alpha\) values?Approximation models a couple of orders of magnitude smaller than the target gave \(\alpha\) between 0.5 and 0.9 (e.g. LaMDA 137B with LaMDA 8B: 0.75); \(\alpha\) grew with the size of \(M_q\) and was higher for sharper distributions (argmax); even unigram and bigram models gave non-negligible \(\alpha\).
- Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §4.2
Source
- Yaniv Leviathan, Matan Kalman and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202, pp. 19274–19286. https://proceedings.mlr.press/v202/leviathan23a.html