Methodology

What we run

We use the harness in nexrall-code/packages/bench/swebench (the same repository that ships the nex CLI), which has two phases:

1. Inference (run_inference.py) — for each of the 500 SWE-bench Verified task instances: clone the target repository at the task's pinned base_commit, run nex -p "<issue text>" --output-format json in headless mode against it, and capture git diff as the candidate patch.
2. Evaluation (evaluate.sh) — hand the resulting preds.jsonl to the official, unmodified swebench.harness.run_evaluation module from SWE-bench/SWE-bench. This is the same grading code the public leaderboard uses: it builds a per-repo Docker container, applies the candidate patch, runs the task's hidden test_patch, and checks the specific FAIL_TO_PASS / PASS_TO_PASS test IDs. We do not modify this code in any way.

What the agent sees — and doesn't

• The agent receives only problem_statement (the public GitHub issue text). It never receives test_patch (the hidden tests used to grade it).
• It also never receives hints_text — maintainer/PR discussion that often gives away the actual fix. The official SWE-bench submission checklist requires "Does not use the hints field"; we hold ourselves to the same rule even though we aren't submitting to that leaderboard.
• The agent has a fetch_url tool it can use to browse the web. For every benchmark run we set NEXRALL_FETCH_BLOCKLIST=github.com,... (and its API/raw-content/CDN subdomains), which makes fetch_url refuse any request to those hosts. This closes the obvious way an agent could otherwise look up the real PR that already fixed the issue it's being scored on — the same requirement the official checklist places on leaderboard submissions ("has taken steps to prevent lookup of SWE-bench solutions via web-browsing").

Attempt policy

Every result on this site is pass@1: one attempt per task instance, no retries, no best-of-k selection. run_inference.py runs each instance_id exactly once by construction — there is no code path that attempts an instance twice and reports the better outcome.

What we don't do

• We do not use SWE-bench test knowledge (FAIL_TO_PASS/PASS_TO_PASS) to guide the agent — those are only used by the harness, after the fact, to grade the patch the agent already committed to.
• We do not hand-pick which instances to run or exclude any instance from the published count. A run is either the full 500-instance Verified split or is explicitly labeled as a smaller "smoke test" subset.
• We do not alter the evaluation harness's pass/fail logic.

Cost & infrastructure

Each run executes on a single Linux x86_64 box (typically an AWS EC2 c6i.4xlarge). Inference cost (the actual model API spend across 500 multi-turn agentic sessions) is reported alongside each result where available.

Why we don't submit to swebench.com

As of November 18, 2025, the official SWE-bench Verified and Multilingual leaderboards only accept submissions from academic teams or research institutions with a peer-reviewed publication or arXiv preprint, with at least one author affiliated with an academic institution or established research lab. Maxrall, Inc. is a commercial company; we do not meet that criterion, and we aren't going to misrepresent an affiliation to satisfy it.

We publish here instead, using the identical open-source evaluation harness, so the numbers mean exactly what they'd mean if they were on that leaderboard — and so anyone can independently reproduce them. See Reproducing these results.