We Benchmarked WalnutAI’s Coding Agent on SWE-bench Verified - Here’s How It Did

We Benchmarked WalnutAI's Coding Agent on SWE-bench Verified — Here's How It Did

TL;DR: We ran WalnutAI's autonomous coding agent against SWE-bench Verified — 500 real, human-validated GitHub issues from large open-source projects. Paired with a frontier model, our agent resolved up to 87.8% of them. Because WalnutAI is model-agnostic, we ran the exact same agent across four different models. Every prediction, log, and score is public and reproducible.

What is SWE-bench Verified?

SWE-bench Verified (https://www.swebench.com/) is the industry-standard benchmark for autonomous coding. Each of its 500 tasks is a real bug or feature request from a major open-source Python project (Django, SymPy, scikit-learn, Matplotlib, Astropy, and more). To "resolve" a task, an agent must:

1. Read the issue,

2. Explore an unfamiliar codebase,

3. Write a correct code change, and

4. Pass the project's own hidden test suite.

There's no partial credit. Either the maintainers' tests pass, or they don't. It's the closest thing the industry has to asking: "Can this agent actually fix real software?"

The Results

We ran WalnutAI's coding agent

The headline number — 87.8% — puts WalnutAI's agent squarely in frontier-class territory for autonomous software engineering. But the more important story is the shape of the table.

The real differentiator: WalnutAI is model-agnostic Look at that table again. It isn't four different products — it's one agent, four different engines. Most AI coding tools lock you into a single model. WalnutAI doesn't. You choose the model that fits your budget, your latency needs, your data-residency rules, or your existing vendor contract — and the same agent orchestration drives it. Prefer a frontier model for hard refactors? Done. Need a cost-efficient or self-hostable open-weight model for bulk work, or for an air-gapped environment? Also done. That flexibility is a structural advantage: No vendor lock-in. Switch models as the field evolves and it evolves monthly. Cost control. Route cheap work to cheap models, hard work to premium ones. Deployability. Run frontier models in the cloud, or open models on-prem for regulated and air-gapped environments. The benchmark proves this isn't marketing the agent posts strong numbers regardless of which engine you plug in.

Methodology & integrity

A benchmark number is only worth as much as its reproducibility. Ours is fully open: Real evaluation harness. We scored every run with the official SWE-bench evaluation harness the same one used for the public leaderboard. Tests pass or they don't; we don't grade our own homework. Contamination guards. During each run, the agent solved from the repository alone. Web search and web fetch were disabled, git history was stripped so the fix commit was unreachable, and outbound access to code hosts was blocked so the agent could never look up the "answer." Everything is public. Predictions, per-task logs, agent trajectories, and final scores are all in our open repository. Anyone can re-run the grading and get the same numbers.

Reproducibility bundle:** github.com/walnutai-labs/walnutai-swe-bench

What this means if you build software

A benchmark score is a proxy. Here's what it translates to in practice the things WalnutAI does for your team every day:

• Generate requirements from any BRD, Figma, or Jira file

• Build production apps from a prompt or a document

• Auto-generate tests just by visiting your app's URL

• Cloud dev agents that write code and ship PRs 24/7

• Gap analysis that surfaces real code risk before it ships
  
  The same engine that resolves real GitHub issues on SWE-bench is the one reviewing your PRs, writing your tests, and shipping your features.

Try WalnutAI free

The fastest way to judge a coding agent is to point it at your own codebase. Install our VSCode extension and let it work on a real issue in your repo.

Create your free account →(https://app.walnutai.ai/signup)

• 14-day free trial

• 500 free AI credits to start No credit card required
  
  See for yourself why our agent scores the way it does on your code, not ours.