ExploitBench
How it works
The exploitation process is decomposed into 16 measurable capabilities arranged as a ladder of increasing difficulty: from code coverage and triggering a crash, through sandbox primitives, arbitrary memory read/write and control-flow hijack, up to arbitrary code execution. Each task is built on a real V8 engine bug. Reaching a given step is confirmed by a deterministic oracle, and result reliability is strengthened by randomized challenge variants and differential execution analysis that limit false positives. A model or agent is characterized by the highest ladder step it reliably reaches.
Problem solved
Existing LLM security benchmarks typically equate a crash with successful exploitation, collapsing the hardest parts of an attack: the transition from triggering a bug to building reusable primitives and actually taking control of the target. ExploitBench addresses this with a graded, multi-step measure of real exploitation capability.
Components
A 16-step scale from code coverage and crash, through sandbox primitives, arbitrary read/write and control-flow hijack, up to arbitrary code execution.
41 real V8 JavaScript engine vulnerabilities used as exploitation tasks.
Randomized challenge variants and differential execution analysis that confirm a step was reached and limit false positives.
Evolution
Benchmark published on arXiv (2605.14153); evaluation of 8 public frontier models and 1 private model on 41 V8 engine bugs.
Hyperparameters (configurable axes)
Number of measurable exploitation capabilities composing the ladder.
Number of bugs used as exploitation tasks.
Models and agents evaluated on the benchmark.