SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024))
Subscriber deck — free during early access
1 cards in “2.2 Task Formulation” · whole deck
2.2 Task Formulation
- How is a SWE-bench solution evaluated?The model’s patch is applied with the Unix patch tool and the repository’s tests are run: the issue counts as resolved only if the patch applies and all associated tests pass, both the fail-to-pass tests from the reference PR and existing tests checking prior behaviour.
Resolved = patch applies and both kinds of test pass - Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024)), §2.2
Source
- Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. In The Twelfth International Conference on Learning Representations (ICLR 2024). https://arxiv.org/abs/2310.06770