Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters
Snell et al., 2025, Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters (The Thirteenth International Conference on Learning Representations (ICLR 2025))
Subscriber deck — free during early access
1 cards in “7 Putting it Together: Exchanging Pretraining and Test-Time Compute” · whole deck
7 Putting it Together: Exchanging Pretraining and Test-Time Compute
- How did Snell et al. compare test-time compute with pretraining a larger model fairly?With pretraining FLOPs \(6ND_{\mathrm{pretrain}}\) and inference FLOPs \(2ND_{\mathrm{inference}}\), scaling parameters by \(M\) multiplies both; the smaller model gets the matching extra inference budget. The comparison depends on \(R = D_{\mathrm{inference}}/D_{\mathrm{pretrain}}\): test-time compute won on easy and medium questions when \(R \ll 1\), pretraining won on hard questions or when \(R \gg 1\).
- Snell et al., 2025, Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters (The Thirteenth International Conference on Learning Representations (ICLR 2025)), §7
Source
- Charlie Snell, Jaehoon Lee, Kelvin Xu and Aviral Kumar. 2025. Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters. In The Thirteenth International Conference on Learning Representations (ICLR 2025). https://arxiv.org/abs/2408.03314