Measuring AI Ability to Complete Long Tasks
Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499)
Subscriber deck — free during early access
2 cards in “1 Introduction” · whole deck
1 Introduction
- What problems with AI benchmarks motivate METR’s time-horizon metric?Benchmark tasks are often artificial rather than economically valuable, are often filtered adversarially to be hard for current models, and saturate ever faster. Scores on different benchmarks cannot be compared, so there is no common scale to track progress from GPT-2 to today.
- Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499), §1
- What is the 50% task-completion time horizon?The length of tasks, measured by how long skilled humans take to do them, that an AI agent completes with 50% probability. It ties AI performance to an intuitive unit, human working time.
- Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499), §1
Source
- Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes and Lawrence Chan. 2025. Measuring AI Ability to Complete Long Tasks. In arXiv preprint arXiv:2503.14499. https://arxiv.org/abs/2503.14499