SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

Tasks must be hard enough to stump current models, yet predictions must be easy to verify. Code can be verified automatically by running tests; but benchmarks like HumanEval use self-contained problems solvable in a few lines, unlike real software engineering.

2,294 real software-engineering tasks from GitHub issues and the pull requests that fixed them, across 12 popular Python repositories: given the issue text and the full codebase, a model must produce a patch that resolves the issue.

Very poorly: with BM25 retrieval the best model, Claude 2, resolved only 1.96% of issues; even given exactly the files the reference patch edited, it resolved 4.8%.

Three stages: scrape PRs from 12 well-maintained repositories; keep merged PRs that resolve an issue and change test files; then run the tests before and after the PR and keep only instances with at least one fail-to-pass test and no installation or runtime errors.

The model’s patch is applied with the Unix patch tool and the repository’s tests are run: the issue counts as resolved only if the patch applies and all associated tests pass, both the fail-to-pass tests from the reference PR and existing tests checking prior behaviour.

Long inputs (issues of ~195 words, codebases with thousands of files and ~438K lines), edits across contexts (reference patches touch on average 1.7 files, 3 functions and 33 lines) without hints about where to edit, and robust evaluation (a median of 51 extra tests guard existing functionality).

The collection pipeline runs on any Python repository with minimal human effort, so new issues created after a model’s training cutoff can be added, ensuring the solution wasn’t in its training data.

CodeLlama-Python 7B and 13B fine-tuned (LoRA on attention weights) to generate the gold patch from the issue and relevant files, using 19,000 issue–PR pairs from 37 other repositories, disjoint from the benchmark’s to avoid contamination. Off-the-shelf CodeLlama couldn’t follow the instructions.

BM25 retrieval of files, filling a context limit (e.g. 27K tokens); an “oracle” setting gives the files the reference patch edited. At the 27K limit BM25 retrieved a superset of the oracle files in about 40% of cases, but none of them in almost half.

It dropped as total input grew: models struggled to localize the few lines needing change among much irrelevant code, so larger BM25 contexts hurt even when they raised recall; collapsing oracle files to just the edited lines (±15) raised Claude 2 from 4.8% to 5.9%.

Mostly not: performance on issues from before and after 2023 was similar for most models, suggesting they weren’t simply reproducing later versions of the code from training data.

Patches are a compact representation of a change; although models often produced malformed patches, regenerating whole files was worse (Claude 2: 2.2% vs 4.8% with oracle retrieval).

They were much shorter and simpler (about 30 vs 75 lines changed), rarely edited more than one file, wrote primitive code that ignored existing libraries and conventions, and solved the issue “greedily”, while human patches often made broader structural improvements.

All tasks are in Python; the baselines are deliberately simple (retrieval plus generation, not agents); and passing tests doesn’t guarantee good code: generated fixes may be less comprehensive, efficient or readable than human ones.