learn.ifs.name

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024))

Subscriber deck — free during early access

3 cards in “1 Introduction” · whole deck

1 Introduction

  1. What makes a good benchmark, and why are coding tasks attractive?
    Tasks must be hard enough to stump current models, yet predictions must be easy to verify. Code can be verified automatically by running tests; but benchmarks like HumanEval use self-contained problems solvable in a few lines, unlike real software engineering.
    • Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024)), §1
  2. What is SWE-bench?
    2,294 real software-engineering tasks from GitHub issues and the pull requests that fixed them, across 12 popular Python repositories: given the issue text and the full codebase, a model must produce a patch that resolves the issue.
    • Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024)), §1
  3. How well did models do on SWE-bench when it appeared?
    Very poorly: with BM25 retrieval the best model, Claude 2, resolved only 1.96% of issues; even given exactly the files the reference patch edited, it resolved 4.8%.
    • Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024)), §1

Source

  • Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. In The Twelfth International Conference on Learning Representations (ICLR 2024). https://arxiv.org/abs/2310.06770