Measuring AI Ability to Complete Long Tasks
Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499)
Subscriber deck — free during early access
1 cards in “3.3 Qualitative analysis” · whole deck
3.3 Qualitative analysis
- What drove the growth in time horizons, according to failure analysis?Comparing failed runs of GPT-4 and o1: much better tool use, adapting after mistakes instead of repeating failed actions, and stronger logical reasoning and code generation. Agents still struggled in “messier” settings without clear feedback or that require proactively seeking information.
- Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499), §3.3
Source
- Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes and Lawrence Chan. 2025. Measuring AI Ability to Complete Long Tasks. In arXiv preprint arXiv:2503.14499. https://arxiv.org/abs/2503.14499