Measuring AI Ability to Complete Long Tasks
Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499)
Subscriber deck — free during early access
1 cards in “2.1 Task suite / dataset” · whole deck
2.1 Task suite / dataset
- Which tasks did Kwa et al. use?170 software and research-engineering tasks spanning seconds to over 8 hours: 97 HCAST tasks (1 minute to 30 hours), 7 RE-Bench ML research-engineering tasks of 8 hours each, and 66 new short Software Atomic Actions (SWAA), single-step tasks that make even pre-2023 models measurable. All are scored automatically.
- Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499), §2.1
Source
- Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes and Lawrence Chan. 2025. Measuring AI Ability to Complete Long Tasks. In arXiv preprint arXiv:2503.14499. https://arxiv.org/abs/2503.14499