CWE-bench v2: More coverage, quality, and transparency

Collinear AI has released CWE-bench v2, a cybersecurity benchmark whose public set of 100 tasks is now open for anyone to run, according to a November 3 post on Hugging Face. Each task hands an AI agent a checkout of a real open-source repository and a brief instruction, then asks it to locate and repair a security weakness without breaking existing behavior. The tasks span 91 distinct CWE categories, and grading is entirely programmatic, which the authors say makes results deterministic and reproducible. A separate held-out set remains private so progress can be measured on unseen tasks.

Collinear reports that 11 models have been evaluated so far. Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, narrowly ahead of Claude Opus 5.5 at 67.3% and 78.0%. At least one model solved 95 of the tasks, which the team takes as evidence the set is solvable, though even the leader fails 19% of tasks on every attempt. On cost, the authors place three models on the Pareto frontier: DeepSeek V4.1 Flash at 57.0% pass@1 for $0.04 per rollout, Opus at $1.83 and Gemini at $3.92.

The team says LLM judge panels helped curate the checks, that every task was reviewed by hand, and that verifiers are meant to reject memorized fixes such as the known upstream patch. The benchmark runs through Harbor directly from a gated Hugging Face dataset, and its environments can also supply rewards for training defensive agents, giving researchers an open way to compare models on vulnerability repair.