learn.ifs.name

SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024))

Subscriber deck — free during early access

1 cards in “3 SWE-Llama: Fine-tuning CodeLlama for SWE-bench” · whole deck

3 SWE-Llama: Fine-tuning CodeLlama for SWE-bench

  1. What is SWE-Llama, and how was it trained?
    CodeLlama-Python 7B and 13B fine-tuned (LoRA on attention weights) to generate the gold patch from the issue and relevant files, using 19,000 issue–PR pairs from 37 other repositories, disjoint from the benchmark’s to avoid contamination. Off-the-shelf CodeLlama couldn’t follow the instructions.
    • Jimenez et al., 2024, SWE-bench (The Twelfth International Conference on Learning Representations (ICLR 2024)), §3

Source

  • Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. In The Twelfth International Conference on Learning Representations (ICLR 2024). https://arxiv.org/abs/2310.06770