Behind the LLM Cybersecurity Hype

Introduction

#

The same vendors publishing breakthrough numbers on LLM cyber capability are publishing the disclaimers in the same posts. Read both halves and the picture changes.

The numbers that get the headlines

#

Anthropic’s Claude Mythos Preview evaluation by AISI¹ reports 73% success on expert-level CTFs and three full completions of a 32-step autonomous attack chain that AISI estimates at roughly 20 hours of human effort (a later checkpoint managed six of ten²). An April arXiv preprint on multi-agent harness synthesis³ claims ten zero-days in Chrome, including two “critical sandbox escapes”, though Chromium rates one of the two (CVE-2026-5280) High, and as code execution inside the sandbox. Anthropic’s Firefox 147 benchmark, as cited by Vincenzo Iozzo⁴, saw Mythos produce 181 working exploits where Opus 4.6 produced two. AISI’s follow-up² put the doubling time of cyber task length at 4.7 months as of February, down from eight months the previous November, and says Mythos Preview and GPT-5.5 have since beaten that trend.

Those numbers travel. They show up in press releases, board decks, and policy papers.

The numbers that don’t

#

The same AISI evaluation notes, in the same post, that the test ranges have no active defenders, no defensive tooling, and no penalties for triggering alerts. Anthropic’s much-cited “thousands of critical vulnerabilities” claim was extrapolated from 198 manually reviewed cases⁵; in at least one prominent case, an FFmpeg bug, Anthropic’s own analysis concluded it was not critical. Mythos surfaced real Linux kernel bugs and then could not exploit any of them because of the kernel’s defense-in-depth. Mozilla credited Mythos with 271 Firefox 150 vulnerabilities⁶, but the interesting question (what kind of bugs, how serious, do they shift the balance) went unanswered.

Iozzo’s Opus 4.6 experiment (n=1, by his own caveat) points the same way: the model found the bug by matching a known pattern, then confabulated an account of how it reasoned. “The only thing keeping us honest in this experiment was the ability to spot-check — and that only worked because I already knew the answer.”

The clearest signal

#

The most useful data point of the last month came from XBOW⁷, an autonomous-pentest vendor whose entire business depends on this technology working. They disclosed CVE-2026-45185, an unauthenticated Exim RCE, and structured the disclosure as a head-to-head: their autonomous track against a senior exploit developer using LLMs as an assistant. The autonomous system succeeded against simplified CTF-style builds. Against the production Exim binary, with ASLR and PIE enabled, it never even got as far as a leak; the human side got one stack address, and no full exploit either. Federico Kirschbaum, head of XBOW’s Security Lab, summarized it:

Honestly, I don’t think LLMs alone are quite ready to write exploits against real-world software yet.

When the vendor whose business depends on autonomous exploitation publicly concedes the autonomous part isn’t ready, the headline benchmark numbers deserve a second read.

What’s actually shipping

#

Strip the hype and the picture is consistent. LLMs are now industrial-scale bug finders and triage assistants: useful enough that curl, the Linux kernel, and Firefox are all dealing with their output at volume, sometimes productively⁸, sometimes as a moderation problem⁹. They are not yet the autonomous attackers that the “AlphaGo moment” framing suggests. And dbreunig’s “proof of work”¹⁰ economics (roughly $12,500 per 100M-token attempt on AISI’s range, $125,000 for all ten runs) describe a landscape where well-funded defenders may scale this technology before opportunistic attackers do.

The hype is interesting. The disclaimers are useful. Pay more attention to the second.