learn.ifs.name

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024))

Subscriber deck — free during early access

1 cards in “2.1 Benchmark Construction” · whole deck

2.1 Benchmark Construction

  1. How were SWE-bench task instances selected from about 90,000 pull requests?
    Three stages: scrape PRs from 12 well-maintained repositories; keep merged PRs that resolve an issue and change test files; then run the tests before and after the PR and keep only instances with at least one fail-to-pass test and no installation or runtime errors.
    • Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024)), §2.1

Source

  • Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. In The Twelfth International Conference on Learning Representations (ICLR 2024). https://arxiv.org/abs/2310.06770