AWS Continuum Hits 89% on CyberGym-E2E, but a Catch Remains

AWS made Continuum for code vulnerabilities generally available on Thursday. The service finds and validates bugs in a team's repositories, checks them against the customer's AWS environment, and opens pull requests with proposed fixes.

The headline number comes from a benchmark AWS published on October 5, more than two weeks before launch. Continuum passed 819 of 920 tasks on CyberGym-E2E, an 89.0% end-to-end success rate, compared with a previous public high of 65.9%. AWS CEO Matt Garman wrote on X that AI is "genuinely transformative for defenders" and that Continuum scored higher than any other AI agent on the test.

How the benchmark works

CyberGym-E2E gives an agent 90 minutes to find and fix a vulnerability in one of 139 open-source projects. It gets no bug description, proof of concept, crash log or existing patch. The agent must identify the flaw, trigger it, and produce a fix that stops the crash without breaking existing tests. All tasks involve memory-safety bugs in C or C++, which leaves out many other languages and vulnerability types, as AWS acknowledges.

Grading runs in four cumulative stages. Continuum found and reproduced a bug in 92.5% of tasks, stopped the crash in 89.6%, and kept the project's tests passing in 89.0%. The fourth stage checks whether the patch also fixes the historical vulnerability the benchmark selected. There, Continuum succeeded in 37.8% of tasks, against a previous public high of 26.2%.

The catch

The agent is never told which historical vulnerability is being checked. It only has to find and repair a real flaw. Because a repository can contain several vulnerabilities, fixing a different one is still useful work, which helps explain the gap between the two scores. What remains unknown is how well Continuum performs when a team points it at a specific vulnerability.

AWS ran the tests itself and says Continuum followed the benchmark's rules, including limits on internet access. The company also checked that the agent was not simply looking up fixes that were already public. With more time than the 90-minute limit, AWS says the success rate rises to 93.7%. The benchmark and testing tools are public, but so far the results are AWS's alone.