CWE-bench v2: best model fixes 67.8% of security tasks, cheap one 57.0%

Collinear AI published CWE-bench v2 on Hugging Face on 3 November 2026. It is a public set of 100 defensive security tasks that anyone can run. Collinear reports that, of 11 models it evaluated, Gemini 4 Argon scores highest at 67.8% pass@1, meaning the share of tasks fixed on the first attempt. Claude Opus 5.5 follows at 67.3%. The company, which builds the benchmark, also publishes the cost per attempt.

What the benchmark tests

According to Collinear, an agent receives a checkout of a real open-source repository and a short instruction. Its job is to find a security weakness and fix it without breaking existing functionality. The 100 tasks cover 91 different weakness types from the CWE catalogue, the standard list of software flaw categories, across 8 languages, per the dataset page.

Scoring is programmatic, so results are deterministic and cost nothing extra to reproduce. Collinear says each task was checked so that the starter code fails, the reference fix passes, and the original upstream patch does not simply pass by memorisation. It says every task was also reviewed by hand and that LLM judge panels helped shape the checks. It keeps a separate held-out set that is not public.

The scores and the price

Collinear reports that 95 of the 100 tasks are solved by at least one model, which it takes as evidence the tasks are solvable. Even the leader fails 19% of the set on every attempt. Gemini 4 Argon reaches 81.0% pass@4 (at least one success in four tries), against 78.0% for Claude Opus 5.5.

On cost, Collinear says DeepSeek V4.1 Flash (1017), Claude Opus 5.5 and Gemini 4 Argon form the Pareto frontier, the set of models that no other model beats on both score and price. DeepSeek reaches 57.0% pass@1 at $0.04 per rollout. Opus adds ten points at $1.83, and Gemini edges ahead at $3.92. By my arithmetic from those figures, Opus costs about 46 times as much per attempt as DeepSeek for roughly ten more points.

How to run it

Each task is a Dockerised environment with an instruction and a verifier, packaged for the Harbor evaluation runner. Collinear's post gives a command that runs the whole set from the Hugging Face repository with four attempts per task, and the reported score is the mean over all 400 trials. The dataset is gated, so you accept its access conditions and save a Hugging Face token first. The licence tag on the dataset is Apache-2.0, and the tasks are also tagged for use as reinforcement-learning environments.

What the numbers do not tell you

• The results are Collinear's own, and the post does not say how it configured each model's agent or whether vendors checked the runs. Treat the ranking as one lab's measurement.
• Gaps of half a point between the top models on 100 tasks are small, and the post gives no confidence intervals.
• The leaderboard tables appear as images in the post, so I could not check the other nine models' scores.
• Passing means the exploit stops and legitimate behaviour survives. It does not tell you whether a patch fits your codebase or your review process.

What to do with it

If you are weighing an agent to patch vulnerabilities, run the public set with your own harness and the model you actually use, and record the cost per attempt alongside the score. Compare pass@1 with pass@4: a wide gap means the agent needs retries and a human or automated check to pick the working fix. A cheaper model that scores ten points lower may still suit a first pass where every patch gets reviewed.