Fast Inference from Transformers via Speculative Decoding
Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202)
Subscriber deck — free during early access
1 cards in “2.2 Standardized Sampling” · whole deck
2.2 Standardized Sampling
- How does speculative decoding handle argmax, top-k, nucleus or temperature sampling?Each is cast as plain sampling from an adjusted distribution (e.g. argmax = zero out all but the maximum and renormalize). \(p(x)\) and \(q(x)\) denote the two models’ distributions after the same adjustment, so one sampling procedure covers all methods.
- Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §2.2
Source
- Yaniv Leviathan, Matan Kalman and Yossi Matias. 2023. Fast Inference from Transformers via Speculative Decoding. In Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202, pp. 19274–19286. https://proceedings.mlr.press/v202/leviathan23a.html