Measuring AI Ability to Complete Long Tasks
Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499)
Subscriber deck — free during early access
1 cards in “4 External validity and robustness” · whole deck
4 External validity and robustness
- How did Kwa et al. test whether the trend holds beyond their task suite?On SWE-bench Verified the trend was also exponential, with a shorter doubling time. On tasks rated by 16 “messiness” factors, models did worse on messier ones but improved at a similar rate. On METR’s own internal pull requests, agents matched contractor time but did worse than repository maintainers’ time predicts; contractors took 5–18× longer than maintainers.
- Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499), §4
Source
- Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes and Lawrence Chan. 2025. Measuring AI Ability to Complete Long Tasks. In arXiv preprint arXiv:2503.14499. https://arxiv.org/abs/2503.14499