CWE-bench v2: top agents fix 67% of security flaws on the first try

Collinear AI published CWE-bench v2 on the Hugging Face blog on 3 November 2026, opening a public set of 100 tasks that test whether a coding agent can find and fix a security weakness without breaking the software. Of 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, just ahead of Claude Opus 5.5 at 67.3% and 78.0%, the company reports. The dataset card dates version 2.0, the one the leaderboard reports, to 30 October.

What a task looks like

The agent gets a checkout of a real open-source repository and a short instruction. The authors' arXiv paper says agents are not told what the weakness is, where it sits or how many vulnerable sites there are. The 100 public tasks cover 91 types in the Common Weakness Enumeration (CWE), the standard catalogue of software flaws, across eight languages: 27 tasks in Go, 15 each in Java and JavaScript, 14 in C++, 10 each in C and Python, 5 in Rust and 4 in TypeScript.

Grading is a program, not a model. The dataset card says a patch scores 1 only if every exploit input is closed and every legitimate case still works, and 0 otherwise. Collinear says the checks reject memorised fixes such as the known upstream patch. Each agent has one hour per task, with no network access beyond its model's API host.

The scores and the bill

• Gemini 4 Argon: 67.8% pass@1, 81.0% pass@4, $3.92 per rollout.
• Claude Opus 5.5: 67.3% pass@1, 78.0% pass@4, $1.83 per rollout.
• DeepSeek V4.1 Flash (1017): 57.0% pass@1, $0.04 per rollout, the top open-weight model on the dataset page.
• Other open-weight scores on the dataset page: Tencent Hy4-preview 52%, Mistral Large 4 46.3%, GLM-5.3 44%.

Collinear says those three models form the cost frontier. A full run is 400 trials, four attempts at each task. On Collinear's per-rollout prices, my arithmetic puts that at about $16 for DeepSeek V4.1 Flash, $732 for Claude Opus 5.5 and $1,568 for Gemini 4 Argon. At least one model solved 95 of the 100 tasks. The leader still failed 19% of the set on every attempt.

The paper's abstract adds two findings a buyer can use: agents with similar pass@1 differ widely in how much retries help and in cost, and no single agent is best across all languages or weakness classes. A Go shop and a Java shop may not want the same model.

The catch

Collinear built the benchmark, curated the checks and ran the models. The post does not name the agent harness behind each score or give token counts; a sample results entry on the dataset card for a 57.0% score notes the OpenCode harness. The paper describes 250 tasks, of which 150 are held out, and the post gives no held-out scores. The post predates Claude Fable 5.2, which Anthropic released on 9 November, so the newest models are absent from the figures quoted here.

Each task also contains one weakness in a repository the agent is told to audit. That measures repair under known conditions. It does not measure how an agent behaves on your codebase, where nobody knows how many flaws exist.

Running it yourself

The tasks are released under Apache 2.0 on a gated Hugging Face repository; access is granted automatically after accepting conditions that bar training models that will be evaluated on the tasks and republishing solutions. Each task is a Docker environment with a default of 2 CPUs and 2 GB of memory, run with the Harbor framework's command line. A team choosing a model for automated security fixes can run the public set with its own agent and compare its cost per rollout with Collinear's.