The One-Armed Zero-Day Bandit
A Self-Licking Tokenmaxxing Vuln Finding Lollipop
Bernie’s Bounding Box asked whether the fix reaches production when AI finds a vulnerability in an open-source library. It doesn’t — Anthropic’s own dashboard showed six percent of disclosed findings patched, less than half a percent of found. The dependency graph between the patched library and your running application is dead in the middle, and the fix can’t cross it.
This piece asks the question upstream: when are we done? When does the enterprise get to hang up a banner that says this application is secure, that any real vulnerabilities have been enumerated and mitigated? How many licks does it actually take to scan through the application attack surface? The world may never know.
Same Code, Different Dice
The same frontier model scans the same repository on consecutive days and returns materially different finding sets — not different confidence scores on the same findings, but different findings entirely. Vulnerabilities that appear on Tuesday, vanish on Wednesday, and resurface on Friday with a different severity rating and a slightly different description that makes you wonder whether the model is gaslighting you or whether you’ve been staring at dependency graphs long enough to gaslight yourself.
This is not an implementation bug. It is a structural property of how these models reason about code. Each scan is a pseudorandom walk through the model’s attention window — following patterns learned from training data, and anchored in the context provided for a given run. Different walk, different findings. On a sufficiently large codebase, repeat scans will continue to surface new findings for a very long time, because the space of possible attention paths through the code is larger than any finite number of scans can exhaust. Attention is all you have — and on a codebase of any real complexity, the only thing you can know for sure is that you may never know when you have captured the full attack surface for any given class or severity of vulnerability.
Run the same model against the same codebase thirty times instead of once and you go from solving a quarter of the benchmark’s known vulnerabilities to solving two-thirds — same model, same code, same vulnerabilities sitting in the same files waiting to be found, and the single scan missed most of them. The enterprise that runs one scan and files the ticket has pulled the lever on a slot machine once and reported the result as a census of every prize in the drum.
The multi-armed bandit is a well-understood pattern in ad-tech — an optimization problem where you have multiple slot machines with unknown payouts and have to decide which to pull, knowing you’ll never play enough rounds to learn the full distribution. Ad-tech learned decades ago that you never exhaust the exploration space. Vulnerability scanning is still pretending one pull of one lever tells you what’s in the drum. Coverage may be buyable — thirty runs, multiple models, measurable gains — but the exercise is expensive, and the economics get worse with the breadth of the scan portfolio. The security industry doesn’t have a definition of meaningful coverage, isn’t disclosing known gaps, and hasn’t priced what it would cost to start filling them in. Incremental recall is buyable. Completeness is not.
A scan is a sample, not a census.
Practitioners inside the institutional scanning programs know this. A scan gets reported as complete when it’s a single sample, and no framework anywhere makes anyone say so. The operational awareness exists. The governance disclosure does not.
Four Models Walk Into a Codebase
Four frontier programs now enable scanning at scale — Mythos, GPT-5.5-Cyber, MDASH, Sec-Gemini — plus a growing tail of open-weight models and agentic harnesses that reproduce the headline findings at a fraction of the cost and none of the coverage guarantees. Coordination issues aside, the CyberGym leaderboard makes them look like they’re converging.
But are they? When Berkeley’s CyberGym researchers actually measured the overlap, the confirmed zero-day findings had only a sixteen percent overlap. The models didn’t expand a shared finding set. They explored different regions of the same vulnerability surface and came back with largely disjoint maps. The newer model didn’t find everything the older one found plus more — it found different things, and some of the older model’s discoveries vanished in the upgrade.
Nearly three-quarters of the known vulnerabilities in a benchmark set were missed by every frontier model tested. The unknown unknowns are not an edge case. They are the supermajority. The models diverge on what they find. They converge on what none of them find.
When two models independently find the same vulnerability, the organizational instinct is to treat it as corroboration — two independent witnesses confirming the same event. Twenty years as a developer tells me something different: two developers making the same mistake isn’t independent validation, it’s a shared tutorial.
The models agree on the findings they share because they learned the same patterns from substantially overlapping training corpora — the same CVE databases, the same disclosed vulnerabilities, the same patches. Their “independent” convergence reflects shared exposure, not independent reasoning. And they fail together — miss the same vulnerabilities, in the same proportions, across architecturally diverse models — because the patterns they learned have shared holes.
Semgrep tested this directly — took Mythos’s headline discoveries, the FreeBSD and OpenBSD vulnerabilities that launched a thousand pitch decks, and tried to reproduce them under realistic conditions. Whole-file context, no hints, no scope narrowing. No model found either vulnerability. Other teams reproduced them under narrower, scoped conditions — and only a few recovered both. The reproductions that succeeded required exactly the narrowing that open-ended discovery doesn’t have. The industry is benchmarking verification and selling discovery.
And the bug that proves the ceiling wasn’t found by a model at all. Sumedh Thakar’s team at Qualys found CVE-2026-46333 through manual code review — a vulnerability that only became dangerous when combined with specific userland programs, a compositional exploit that required reasoning across a system boundary no single model can see. The frontier models found 23,000 vulnerabilities and missed the one that required thinking across a boundary none of them crossed.
The paradox of the bandit: the models don’t work well enough to enumerate the issues across your code, but they work fast enough to drown you in what they did find.
The House Edge
Even for the vulnerabilities the models do find, the classification is unreliable at the only level that matters.
Anthropic publishes a severity agreement matrix — Claude’s assessments versus external security researchers. Overall exact agreement: 58.7%. But at the critical level — the classification that triggers the emergency response, the weekend page, the board notification — twelve percent concordance with expert reviewers. Of 192 findings Claude rated critical, external researchers agreed on 23. The model says critical. The human says not so much. Inflated seven times out of eight.
For open-source code, which is the entire scope of the CVD program, the information required to make a project-specific severity judgment is public — the code, the community, the discourse, the usage statistics, the project’s own security policy. The model could consume that context before assigning the label that drives triage priority. It doesn’t. Consuming project context for every finding would slow the scan, reduce throughput, and produce a smaller number on the dashboard. The system rewards the number that grows at the expense of the number that’s right.
Berkeley’s Vulnerability Initiative tracks this daily: among agentic CVEs, severity has lost any meaningful relationship to exploit risk. CISA’s BOD 26-04 retired CVSS-based remediation deadlines entirely. The federal government and the academic community have both moved away from the metric these models produce. The enterprise remediation queue hasn’t caught up.
And then there is the floor. Bernie’s Bounding Box defined the ceiling — the dead middle of the dependency graph that will never forward the fix. The bounding box also has a floor.
Anthropic’s true positive count includes “won’t fix” findings — real bugs the maintainer evaluated and consciously decided to live with. Anyone who has maintained an open-source project knows this conversation: yes, the bug is real; no, I’m not restructuring the parser for an attack that requires the attacker to already control the config file. The scanning program counts it as a discovery. The dashboard counts it as a true positive. The six percent patch rate carries it in the denominator forever.
The remediable surface — findings that are real, correctly classified, where the maintainer will fix, where the fix will propagate through a dependency graph whose middle is alive — is a fraction of a fraction of a fraction of the total output. The dashboard shows the numerator. Nobody reports the denominator.
High Rollers & Infinite Flywheels
Tokenmaxxing — the practice of optimizing for AI consumption volume as a proxy for productivity — migrated from engineering leaderboards at Meta and others to the vulnerability management pipeline without anyone noticing the transfer.
The flywheel is simple and self-sustaining. Frontier models scan at computational speed, producing thousands of findings per day. Human triage can only validate a fraction — Anthropic itself reviewed fewer than 2,000 of 23,000 candidates. The validated findings enter a dependency graph that patches six percent. The backlog grows. The growing backlog justifies more scanning because the problem is clearly not solved and more scanning is clearly needed. Daniel Stenberg closed curl’s vulnerability intake for the summer — report rate at one every eighteen hours — because the maintainer of one of the most critical libraries on earth couldn’t keep up with the volume. The ecosystem is not absorbing the output. The ecosystem is drowning in it.
The counterargument writes itself: GLM 5.2 — open-weight, MIT-licensed, a sixth of the cost, runnable on your own hardware with no provider logging — breaks the lollipop because the token meter stops running to a provider. Except tokenmaxxing here is organizational production of vulnerability findings as a proxy for security improvement, and owning the slot machine doesn’t change the game. GLM 5.2 shipped as a free download right as Mythos was disabled worldwide for eighteen days. Neither changes the six percent. And if attackers now have unmonitored Mythos-class scanning, the urgency of closing the remediation gap increases from both sides simultaneously — which is exactly what Mandiant’s negative-seven-day mean time to exploit already measures. Exploitation before the patch exists. The timer the lollipop runs on is already expired.
The cost model follows inexorably. Each scan is a pseudorandom walk through the model’s attention window — following patterns learned from training data, and anchored in the context provided for a given run, but never guaranteed to exhaust the search space. Scanning becomes perpetual OpEx with logarithmic returns. The findings are real — ninety percent true positive rate, remote code execution in FreeBSD — so not scanning means the adversary finds what you didn’t. Mandiant’s mean time to exploit is now negative seven days, highlighting how the frontier has now become the expectation. For first-party code, there is no consortium to share the scan cost and every finding now competes with the feature roadmap — the implications for your COGS are clear. You can’t afford to do it. You can’t afford not to.
Blind Trust and Shared Responsibility
And then there is the code you run but didn’t write.
Third-party and COTS applications are opaque by definition — someone else’s code, design choices, and security maturity becomes your problem the moment the installer drops a shortcut on your screen. The pseudorandom walks that give you probabilistic coverage of your own codebase give you zero coverage of the vendor’s. You inherit their propagation failure blind — their dead dependencies, their won’t-fix decisions, their severity theater. The bounding box for vendor software isn’t just bounded — it’s blind.
SaaS is worse still — what you can see, let alone defend, is predicated on available vendor APIs, tenancy, and logging depth. The application runs in someone else’s environment, on someone else’s stack, with someone else’s dependency graph, and your governance is contractual, not architectural.
This is the gradient the scanning conversation ignores entirely. The security industry frames vulnerability management as though the enterprise has a unified code surface it can scan, classify, prioritize, and remediate. It does not. It has first-party code it can scan probabilistically at unbounded cost, vendor code it cannot scan at all, and SaaS it cannot even fully observe. The bounding box has a different shape for each, and the shape gets worse as you move from what you wrote to what you bought to what you rent.
Tiers for Fears — The App Defense Onion
The prescription is not scan harder. The institutional move is to reprice scanning from assurance to sampling — an input to the risk model, not a certificate of completion. Because code assurance is permanently stochastic, and the threat model can never be finalized, assurance cannot depend on code inspection alone — it also has to come from the behavior the exploit produces, not only the vulnerability it exploits.
Shrink the target. Minimize the codebase. Minimize dependencies. Every line of code you don’t write is a random walk you don’t need. Every dependency you don’t take is a dead node you’ll never propagate through. Reusable application platforms concentrate repeated assurance on a shared foundation and inherit validated controls across the portfolio — the per-application cost collapses from unbounded to amortized. Snowflake count is the master variable — the one that improves scanning economics, instrumentation coverage, and assurance inheritance simultaneously. It is also the only item on this list a CISO cannot buy, because it is an engineering-org outcome, not a security purchase. If the defense program lands and consolidation doesn’t, the program is a tax on complexity you chose not to remove.
Rebuild the supply chain. The prescription from Post 2 — hardened images rebuilt from patched sources to bypass the dead middle entirely — isn’t a separate recommendation. It’s the supply chain tier of the same architecture.
Classify at deployment, not at discovery. The model sees code, not topology. Move severity classification to the point where deployment context exists — inside your environment, against your asset inventory, with your exposure data. BOD 26-04 already writes this into binding federal policy — its four-variable risk model assigns three variables to CISA and the fourth, asset exposure, to the asset owner. The enterprise should follow.
The first three follow from the earlier posts. The next three address the residual risk neither discovery nor remediation can eliminate.
Evolve the SDLC. Agentic harnesses scanning first-party code for vulnerabilities as part of the development pipeline, not as an afterthought. Reachability analysis complementing SAST, SCA, and frontier findings across first-party and open-source libraries — because a vulnerability that’s present in the code but unreachable in the execution path is a different priority than one that’s live. Pre-production offensive ranges for critical applications testing the deployed stack with the attacker’s frontier toolkit before the attacker does. If the scan is a sample, sample from the attacker’s seat — their tools, their models, their frontier access — and deny what you find before they pull the lever. Not convergence. Not assurance. But the attacker faces the same sampling problem, and every exposure you close is one they have to find the hard way.
Harden the stack. Agentic attack harnesses don’t need a single critical vulnerability — they chain low-severity findings across system boundaries into composite exploits that no individual scan would flag. The Qualys CVE was the preview: a vulnerability that only became dangerous in combination with specific userland programs. Exposure management and attack surface management need renewed investment because the attack surface the chaining traverses is the one the fundamentals control — identity, segmentation, configuration, lifecycle management. These have been underinvested for a decade. The agentic attacker makes the bill come due.
Defend the application plane. Defense in depth used to stop at the endpoint and the application perimeter. This is no longer viable. Invest in layered application denial. Runtime instrumentation at the core. eBPF/stack walking as the infrastructure profiler. API security as the gatekeeper. WAF/edge denial at the perimeter. Endpoint detection on entirely separate telemetry as the common-mode backstop. SaaS security platforms providing assurance at the shared responsibility boundary.
This is more expense, not less — but it is bounded expense complementing the unbounded self-licking scan lollipop. The coverage degrades with ownership but doesn’t vanish: vendor software gets edge and OS-boundary tiers without application integration, SaaS gets posture management and detection capabilities from a different architectural toolset.
The questions that matter now are not about scanning. They are about what happens when scanning isn’t enough.
Four Questions for Leadership
What defends your applications from the threats the attackers will stumble on tomorrow using a commodity model but your frontier scanning program hasn’t found yet — and from the vendor code you can’t scan at all?
If “certifiably secure” now means continuous AI scanning that never converges, what does that do to your application development velocity and your operating model?
How many distinct snowflake application stacks in your portfolio — and what would it take to consolidate them to a set of platforms that can be scanned and secured consistently?
What does continuous scanning cost per application per year at the coverage depth the auditors are about to expect — and is that number in anyone’s forecast?
Three posts, one cascade: the fix doesn’t get funded, the fix doesn’t propagate, and the findings never stop arriving because there is no finish line for the scanning. The six moves above are the same answer to all three — stop trusting the scan as a census, and build for a world where the code is never fully knowable.
Stop pulling the bandit’s lever. Build the architecture that survives inside the bounding box — because the slot machine will never tell you what it missed.
This is the third in a series on the structural failures in vulnerability management that technology alone does not solve. The first, Off the Beaten Patch, argued that discovery is not the bottleneck — remediation is. The second, Bernie’s Bounding Box, argued that even remediation at the root is insufficient when the dependency graph has no mechanism to propagate fixes to production. This piece argues that the discovery itself is a sample with unknown coverage — and that the prescription is not more scanning but the architecture that makes the bounding box survivable.