Fast Inference from Transformers via Speculative Decoding

1 Introduction

1. Why is decoding from large autoregressive Transformers slow?
   Each step of a large model is slow, and steps run serially: decoding \(K\) tokens takes \(K\) runs of the model. Yet inference is often bottlenecked on memory bandwidth and communication rather than arithmetic, so extra compute is frequently available.
   • Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §1
2. What two observations underlie speculative decoding?
   (1) Hard language-modelling tasks contain easier steps that a much smaller model approximates well; (2) with speculative execution and a new sampling method, the large model can check several of the small model’s guesses in parallel, producing several tokens per run without changing the output distribution.
   • Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §1
3. What is speculative execution, and how do Leviathan et al. generalize it?
   A processor optimization (e.g. branch prediction) that performs a task in parallel with verifying whether it is needed. Leviathan et al. extend it to the stochastic setting, where a task is needed only with some probability, and use samples from a small model as speculative prefixes for the large one.
   • Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §1
4. How does speculative decoding differ from distillation, quantization or early-exit methods?
   Those change the architecture or training, need new or retrained models, and usually change the outputs. Speculative decoding works with off-the-shelf models, needs no retraining, and guarantees exactly the same output distribution as the target model.
   • Leviathan et al., 2023, Fast Inference from Transformers via Speculative Decoding (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §1