SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

5 Results

1. How did performance depend on context length?
   It dropped as total input grew: models struggled to localize the few lines needing change among much irrelevant code, so larger BM25 contexts hurt even when they raised recall; collapsing oracle files to just the edited lines (±15) raised Claude 2 from 4.8% to 5.9%.
   • Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024)), §5
2. Did model knowledge of repositories seem to let them “cheat”?
   Mostly not: performance on issues from before and after 2023 was similar for most models, suggesting they weren’t simply reproducing later versions of the code from training data.
   • Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024)), §5
3. Why do Jimenez et al. ask models for patches rather than whole files?
   Patches are a compact representation of a change; although models often produced malformed patches, regenerating whole files was worse (Claude 2: 2.2% vs 4.8% with oracle retrieval).
   • Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024)), §5
4. How did model patches compare with the human reference patches?
   They were much shorter and simpler (about 30 vs 75 lines changed), rarely edited more than one file, wrote primitive code that ignored existing libraries and conventions, and solved the issue “greedily”, while human patches often made broader structural improvements.
   • Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024)), §5