s1: Simple test-time scaling

Muennighoff et al., 2025, s1 (Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing)

Subscriber deck — free during early access

1 cards in “3.2 Metrics” · whole deck

3.2 Metrics

What three metrics does s1 use to compare test-time scaling methods?
Control: the fraction of runs that stay within the intended compute budget. Scaling: the average slope of accuracy against thinking tokens, which must be positive. Performance: the best accuracy reached.
- Muennighoff et al., 2025, s1 (Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing), §3.2, p. 20278

Source

Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 20275–20321. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/2025.emnlp-main.1025 https://aclanthology.org/2025.emnlp-main.1025/