Reproducing these results

Everything needed to independently re-run and check a result published here will be linked from this page once a run completes:

• Predictions (preds.jsonl) — the raw model-generated patch for every task instance, including any that errored out.
• Evaluation logs (logs/run_evaluation/<run_id>/) — per-instance patch.diff, report.json (pass/fail against FAIL_TO_PASS/ PASS_TO_PASS), and test_output.txt (the actual test run inside the grading container), exactly as produced by the official harness.
• Result summary (nexrall-code-<model>.<run_id>.json) — the harness's own aggregate output: resolved / unresolved / error counts.

Running it yourself

The harness is the same one Nexrall Code uses internally — no special access required beyond a Nexrall account (for CLI auth) and standard SWE-bench infrastructure (Docker, ~120GB disk, a Linux x86_64 box).

# 1. One-time box setup (installs Docker, the official SWE-bench harness, and
#    the nex CLI via npm).
bash setup.sh
source ~/swebench-venv/bin/activate
export NEXRALL_TOKEN=...          # headless CLI auth — from your Nexrall account

# 2. Smoke test (20 instances) — cheap, proves the pipeline end-to-end.
python3 run_inference.py --out preds20.jsonl --model claude-sonnet-5 --limit 20 --resume
bash evaluate.sh preds20.jsonl smoke20 4

# 3. Full 500-instance run.
python3 run_inference.py --out preds_full.jsonl --model claude-sonnet-5 --resume
bash evaluate.sh preds_full.jsonl full-claude-sonnet-5 8

Because Phase B (evaluate.sh) invokes the unmodifiedswebench.harness.run_evaluation module from SWE-bench/SWE-bench, re-running it on our published preds.jsonl will reproduce the same pass/fail outcome per instance (modulo the harness's own documented flakiness on a small number of known-flaky instances, same as it would for any submission on the official leaderboard).

Questions about a specific result

If a number here doesn't reproduce for you, or you'd like to run a random subset yourself to spot-check a published result, open an issue or reach out — contact details are on nexrall.com.