learn.ifs.name

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024))

Subscriber deck — free during early access

2 cards in “2.3 Features of SWE-bench” · whole deck

2.3 Features of SWE-bench

  1. What makes SWE-bench harder than earlier code benchmarks?
    Long inputs (issues of ~195 words, codebases with thousands of files and ~438K lines), edits across contexts (reference patches touch on average 1.7 files, 3 functions and 33 lines) without hints about where to edit, and robust evaluation (a median of 51 extra tests guard existing functionality).
    • Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024)), §2.3
  2. Why can SWE-bench be continually updated, and why does that matter?
    The collection pipeline runs on any Python repository with minimal human effort, so new issues created after a model’s training cutoff can be added, ensuring the solution wasn’t in its training data.
    • Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024)), §2.3

Source

  • Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. In The Twelfth International Conference on Learning Representations (ICLR 2024). https://arxiv.org/abs/2310.06770