How it works
SWE-bench Verified evaluates a coding agent against real issues drawn from open-source Python repositories. Each task is a self-contained unit: the agent gets the repository checked out at a specific commit and the text of a genuine issue, and it must produce a code patch that fixes the described problem. The harness then runs the project's own test suite against the patched code.
Scoring is deterministic and test-driven, not judged by a model:
- Check out the repo at the frozen base commit.
- Hand the agent the issue and let it edit files and run tools.
- Apply the agent's final diff.
- Run the hidden tests. A task counts as resolved only when the previously failing tests now pass and the previously passing tests still pass.
"Verified" is the distinguishing part. It is a subset (500 tasks) that human software engineers reviewed to remove items that were underspecified, had broken or flaky tests, or were effectively impossible to solve from the issue alone. What remains is meant to be a fairer, less noisy signal, so a resolved task reflects a real fix rather than a lucky match against an ambiguous spec. The headline metric is pass@1: the fraction of tasks resolved on a single attempt.
Why it matters in an agent harness
SWE-bench Verified is one of the few widely shared numbers that reflects end-to-end agent behavior, not just model quality. The score is a property of the whole loop: the model, the tool surface, the retrieval and context strategy, the iteration budget, and how the harness recovers from failed edits. Change the scaffolding around the same model and the number moves. That makes it useful as a regression signal for your own harness rather than only a vendor bragging point.
It also models the shape of work you actually care about: bounded, verifiable, reversible units. Every task has a deterministic verifier — the test suite — so success is unambiguous. That is the same property you want in production: an agent's output should be checkable by something other than the agent. When you evaluate against a benchmark with a real verifier, you are practicing the discipline of not trusting agent prose about whether it succeeded.
The limits matter as much as the signal. The benchmark is Python-heavy, issue-driven, and single-repository. It rewards patch-and-verify work and says little about multi-service changes, migrations, ambiguous product decisions, or long-lived state. A high score tells you an agent can close well-specified bugs with tests present. It does not tell you the agent is safe to hand an unbounded task with no verifier.
SWE-bench Verified vs. raw SWE-bench
The distinction changes how much weight you put on a number.
| Aspect | Raw SWE-bench | SWE-bench Verified |
|---|---|---|
| Task set | Full scraped set | Human-reviewed 500-task subset |
| Noise | Includes ambiguous, broken, or unsolvable tasks | Filtered for solvability and test validity |
| Score meaning | Depressed by bad tasks; harder to attribute | Cleaner signal of genuine resolution |
| Best use | Broad coverage, historical comparison | Comparing agents and tracking your harness |
If you are comparing tools or watching your own harness regress, Verified is the more honest number because a failed task is more likely a real capability gap than a broken prompt in the dataset.
The Rifty take
We treat SWE-bench Verified as a harness measurement, not a model leaderboard. The number we care about is our own delta: same benchmark, our scaffolding, tracked over time as we change the loop. We optimize for a verifier existing at all — a task without a deterministic check is a task we will not let an agent self-certify — and we accept that a benchmark of clean, single-repo bugs undersells the messy, unverifiable work where judgment still has to be carried by an operator.
Common failure modes
- Reading pass@1 as reliability. A 60% score means four in ten tasks fail; in production that is a park-and-review rate, not a ship-unattended rate.
- Comparing across different harnesses. Two labs reporting the same model can differ purely by scaffolding, retries, and budget. Compare like-for-like or don't compare.
- Assuming transfer. Strong Python bug-fixing does not imply strong migrations, multi-repo changes, or tasks without tests.
- Contamination drift. As tasks age, training data may have absorbed the fixes. Treat old benchmark numbers as capability ceilings, not steady-state estimates.
- Cheering the headline, ignoring the trajectory. A resolved task can still involve unsafe edits or wasted budget. Score the path, not only the outcome, when the harness will run unattended.
Frequently asked questions
What is the difference between SWE-bench and SWE-bench Verified?
SWE-bench Verified is a 500-task subset of SWE-bench that human engineers reviewed to remove ambiguous, broken, or unsolvable tasks. It gives a cleaner signal because a failed task more likely reflects a genuine capability gap rather than a flawed or underspecified item in the original scraped dataset.
How is a task scored as resolved?
Scoring is deterministic, not model-judged. The harness applies the agent's final patch to the repository, then runs the project's hidden test suite. A task counts as resolved only when the previously failing tests pass and the previously passing tests still pass, reported as a pass@1 rate.
Does a high SWE-bench Verified score mean an agent is production-ready?
No. The score measures single-repository, test-backed Python bug fixing on well-specified issues. It says little about migrations, multi-service changes, ambiguous decisions, or tasks lacking a verifier. Read pass@1 as a capability signal and a review rate, not as evidence that an agent is safe to run unattended.
Why does the score change with the same model?
Because the number reflects the whole loop, not just the model. Tool surface, context strategy, retrieval, iteration budget, and failure recovery all move the score. That is why SWE-bench Verified is most useful for tracking regressions in your own harness rather than comparing raw vendor claims.
How should I use SWE-bench Verified in my own harness?
Use it as an internal regression signal. Hold the benchmark fixed, vary only your scaffolding, and track your own delta over time. Compare like-for-like configurations, watch trajectories rather than only outcomes, and treat aging tasks as capability ceilings due to possible training-data contamination.