learn.ifs.name

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024))

Subscriber deck — free during early access

1 cards in “4.1 Retrieval-Based Approach” · whole deck

4.1 Retrieval-Based Approach

  1. How did the baselines choose which code to show the model, and how well did it work?
    BM25 retrieval of files, filling a context limit (e.g. 27K tokens); an “oracle” setting gives the files the reference patch edited. At the 27K limit BM25 retrieved a superset of the oracle files in about 40% of cases, but none of them in almost half.
    • Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024)), §4.1

Source

  • Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. In The Twelfth International Conference on Learning Representations (ICLR 2024). https://arxiv.org/abs/2310.06770