Measuring AI Ability to Complete Long Tasks
Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499)
Subscriber deck — free during early access
1 cards in “3.1 Computing time horizon” · whole deck
3.1 Computing time horizon
- How is an agent’s time horizon estimated?Binarize each run’s success, then fit a logistic curve of success against log human task time, \(p = \sigma\big((\log h - \log t_{\text{task}})\,\beta\big)\), as in item response theory but with difficulty taken from human time instead of being learned. The fitted \(h\), where predicted success is 50%, is the horizon.
Fitted success probability for a hypothetical agent with a 1-hour horizon - Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499), §3.1
Source
- Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes and Lawrence Chan. 2025. Measuring AI Ability to Complete Long Tasks. In arXiv preprint arXiv:2503.14499. https://arxiv.org/abs/2503.14499