Measuring AI Ability to Complete Long Tasks

Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499)

Subscriber deck — free during early access

12 cards

1 Introduction

What problems with AI benchmarks motivate METR’s time-horizon metric?
Benchmark tasks are often artificial rather than economically valuable, are often filtered adversarially to be hard for current models, and saturate ever faster. Scores on different benchmarks cannot be compared, so there is no common scale to track progress from GPT-2 to today.
- Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499), §1
What is the 50% task-completion time horizon?
The length of tasks, measured by how long skilled humans take to do them, that an AI agent completes with 50% probability. It ties AI performance to an intuitive unit, human working time.
- Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499), §1

Which tasks did Kwa et al. use?
170 software and research-engineering tasks spanning seconds to over 8 hours: 97 HCAST tasks (1 minute to 30 hours), 7 RE-Bench ML research-engineering tasks of 8 hours each, and 66 new short Software Atomic Actions (SWAA), single-step tasks that make even pre-2023 models measurable. All are scored automatically.
- Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499), §2.1

How were task lengths measured?
By timing skilled professionals (software, ML and cybersecurity, about 5 years’ experience on average) without task-specific context: over 800 attempts totaling 2,529 hours. A task’s length is the geometric mean time of successful attempts; 21 tasks without baselines used researcher estimates.
- Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499), §2.2

How is an agent’s time horizon estimated?
Binarize each run’s success, then fit a logistic curve of success against log human task time, \(p = \sigma\big((\log h - \log t_{\text{task}})\,\beta\big)\), as in item response theory but with difficulty taken from human time instead of being learned. The fitted \(h\), where predicted success is 50%, is the horizon.
Fitted success probability for a hypothetical agent with a 1-hour horizon
- Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499), §3.1

What trend did Kwa et al. find in frontier models’ time horizons?
Exponential growth from 2019 to 2025, doubling about every 7 months (207 days, 95% CI 166–240). GPT-2’s horizon was about 2 seconds; o3’s about 110 minutes. 2024–2025 models doubled faster (about every 4–5 months), but that estimate rests on few models.
- Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499), §3.2
Why are Kwa et al. more confident in the slope than in any single model’s horizon?
Tasks with the same human time differ widely in difficulty for models, so which tasks are sampled shifts all models’ horizons up or down together. Those errors are strongly correlated, so they largely cancel in the trend.
- Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499), §3.2
How do 80% time horizons compare with 50% horizons?
They double at about the same rate (204 vs 207 days) but are 4–6 times shorter: even models that sometimes solve long, varied tasks cannot do tasks of moderate length reliably.
- Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499), §3.2

What drove the growth in time horizons, according to failure analysis?
Comparing failed runs of GPT-4 and o1: much better tool use, adapting after mistakes instead of repeating failed actions, and stronger logical reasoning and code generation. Agents still struggled in “messier” settings without clear feedback or that require proactively seeking information.
- Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499), §3.3

How did Kwa et al. test whether the trend holds beyond their task suite?
On SWE-bench Verified the trend was also exponential, with a shorter doubling time. On tasks rated by 16 “messiness” factors, models did worse on messier ones but improved at a similar rate. On METR’s own internal pull requests, agents matched contractor time but did worse than repository maintainers’ time predicts; contractors took 5–18× longer than maintainers.
- Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499), §4

What does extrapolating the trend predict, and with what caveats?
AI agents with a one-month (167 working hours) horizon on such software tasks around 2028–2031, sooner if the faster 2024–2025 trend holds. But real tasks often lack automatic scoring, involve other agents, limited resources and changing environments, and agency training, compute limits or automated AI research could change the trend.
- Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499), §5

Why is a time horizon always relative?
It depends on the task domain and distribution and on the skill and context of the human baseliners (repo maintainers are much faster than contractors), and it is hard to measure for very long tasks or success rates near 100%.
- Kwa et al., 2025, Measuring AI Ability to Complete Long Tasks (arXiv preprint arXiv:2503.14499), §6

Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes and Lawrence Chan. 2025. Measuring AI Ability to Complete Long Tasks. In arXiv preprint arXiv:2503.14499. https://arxiv.org/abs/2503.14499