Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters
Snell et al., 2025, Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters (The Thirteenth International Conference on Learning Representations (ICLR 2025))
Subscriber deck — free during early access
1 cards in “3.2 Estimating Question Difficulty for Compute-Optimal Scaling” · whole deck
3.2 Estimating Question Difficulty for Compute-Optimal Scaling
- How is question difficulty defined and estimated by Snell et al.?Relative to the base model: its pass@1 over 2,048 samples, binned into five quantiles. In deployment the true answer is unknown, so the bins are instead predicted from a learned verifier’s average final-answer score on the samples, at extra inference cost.
- Snell et al., 2025, Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters (The Thirteenth International Conference on Learning Representations (ICLR 2025)), §3.2
Source
- Charlie Snell, Jaehoon Lee, Kelvin Xu and Aviral Kumar. 2025. Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters. In The Thirteenth International Conference on Learning Representations (ICLR 2025). https://arxiv.org/abs/2408.03314