Generated by Codex with GPT-5

What happened

Cloudflare’s official engineering blog published Build your own vulnerability harness, a June 18, 2026 post about turning frontier-model security review from one-off agent sessions into a persistent, model-agnostic vulnerability discovery and validation pipeline.

The post is interesting because it treats the model as the least durable part of the system. Cloudflare’s argument is that enterprise-scale security scanning cannot depend on a single prompt, a single coding agent, or a single model provider. Models change, context windows fill up, and different models see different bug classes. The durable asset is the harness around them: state, task isolation, adversarial validation, deduplication, production reachability checks, and a path from evidence to reviewed patches.

The starting point was modest: a roughly 450-line security-audit skill that ran a seven-phase audit against one repository. It used parallel reconnaissance agents, attack-class-specific hunters, adversarial validators, structured findings.json output, schema checks, and an independent re-verification pass. That skill found real bugs, but it also exposed the limits of single-session AI security work. A long run eventually lost useful memory to context pressure, a crash could erase hours of work, and a repository-local review missed vulnerabilities that only appeared at dependency boundaries.

Cloudflare then codified the skill into a fleet scanner. The resulting system scans 128 repositories across Rust, Go, C, Lua, TypeScript, Python, configuration systems, and cross-repo dependencies without custom language-specific parsers. The architecture does not ask one agent to understand everything. It decomposes the work into narrow stages, stores each stage in a database, and lets agents act as focused reasoning workers inside deterministic orchestration.

The architecture

The pipeline has two main halves. The Vulnerability Discovery Harness, or VDH, finds candidate bugs. The Vulnerability Validation System, or VVS, turns candidates from VDH and other sources into deduplicated, risk-ranked, human-reviewable fixes. Cloudflare deliberately runs discovery and validation through different models, so the model that proposes a bug is not the same logical system that judges whether it should survive.

VDH begins with reconnaissance. Instead of relying only on a fixed threat model, the Recon stage reads the repository and creates a repo-specific attack taxonomy on top of built-in classes such as injection, memory corruption, protocol parsing, and timing side channels. Hunter agents then work by attack class. They do not just read code; they compile fragments, run binaries, and use sandboxes to test hypotheses. That matters because many subtle bugs are behavioral, not syntactic.

Several stages keep the search from collapsing into either shallow coverage or unbounded wandering. Gapfill looks for under-tested combinations of code area and attack class, then enqueues new hunts. Trace walks a cross-repo dependency graph and spawns consumer-repo tasks when an issue may travel across a service boundary. Feedback uses failed validations, shallow runs, and repeated misses to rewrite queued prompts. Sibling forking lets a Hunter split off a focused investigation when it sees an interesting path outside its current scope, while the main task stays constrained.

Persistence is the practical center of the design. Every stage writes into a SQLite database keyed by run, repository, and stage. A failed request, rate-limit interruption, or worker crash loses at most the task in flight, not the whole run. Findings stream to storage as they are produced. This is also how the system avoids abusing the model context window: each agent gets a narrow job, and Cloudflare keeps context use well below the total window instead of letting a single session accumulate an entire audit.

The validation path is stricter than ordinary AI code review. A Hunter must state the threat model before it can file a finding. It must identify the attacker, the crossed trust boundary, and the assumption being violated. Every confirmed finding needs a proof-of-concept test against the original codebase, plus a proposed patch. Deterministic code checks paths, line references, schema conformance, and whether tests and patches parse. An isolated validator then tries to disprove the finding. The important design principle is that a generator cannot grade its own exploit.

The triage layer

Cloudflare’s second major contribution is the VVS, because large-scale automated discovery mostly moves the bottleneck from finding bugs to deciding which ones deserve engineering attention. The post says the central validation pool contained 13,841 findings across 145 repositories at publication time. Comparing each finding to every other finding with an LLM would be quadratic and expensive, so VVS first uses deterministic inverted indexes over touched files, functions, trust boundaries, and rare tokens to create short candidate lists. Only then does a Dedup agent reason about whether multiple reports share one root cause.

Judgment is a separate reachability and risk pass. It pulls production context from internal systems such as deployment data, configuration, wiki pages, issue trackers, and source repositories to determine whether a finding is exploitable in production, latent, assigned to the wrong component, or mostly defense-in-depth. This is the step that turns a generic vulnerability report into an operational queue. Static severity alone is not enough; a bug’s priority depends on whether a realistic attacker can hit it through the actual deployed system.

Fixing is also gated. The Fixer rewrites the proposed patch and regression test to fit the repository, applies the diff, and runs targeted tests. The clean case is a fail-then-pass transition on the regression test with no downstream breakage. Even then, the system opens work for human review rather than merging directly. Cloudflare is using automation to compress discovery, reproduction, and patch preparation, not to remove review from the change-management boundary.

The metrics show why this filtering architecture matters. Out of 20,799 raw VDH candidates, about 12,057 survived validation. After VDH findings joined the central VVS pool, 5,442 were deduplicated, 1,154 were routed away as wrong-repo, low-risk, or otherwise non-actionable, and 7,245 were left as actionable findings for engineering teams. The useful metric is not a speculative recall number, because nobody knows the complete set of true vulnerabilities. The useful metric is how efficiently raw model output is compressed into high-integrity findings and reviewed fixes.

Why it matters

The broader engineering lesson is that agentic security work needs stateful systems engineering, not just better prompts. Subagents can help with local decomposition, but they do not by themselves provide persistence, resumability, deduplication, cross-repo tracing, cost controls, or production reachability judgment. Cloudflare’s harness turns model calls into pipeline stages with clear contracts and measurable failure modes.

The post also captures a useful inversion in AI tooling. Traditional static analysis starts from language parsers and rules, then struggles with business logic and cross-service behavior. Cloudflare’s system lets models handle heterogeneous code understanding and hypothesis generation, while deterministic software handles bookkeeping, indexes, schemas, parse checks, task queues, retries, and gates. The result is not “LLMs replace security engineers”; it is a hybrid system where models generate and test hypotheses under an orchestration layer designed to reject low-quality output.

There is an important reliability lesson in the cost model. Cloudflare budgets by repository, caps tasks per repository, and treats full scans as periodic backlog sweeps rather than per-pull-request checks. That keeps the harness aligned with the economics of long-running agent work. Smaller, cheaper systems are still needed for tight feedback loops, while the expensive fleet scanner is better suited to deeper coverage and cross-repo analysis.

Takeaway

Cloudflare’s vulnerability harness is a strong example of what production AI engineering looks like when the goal is reliability rather than novelty. The system assumes models are useful but volatile, creative but noisy, and expensive enough to need tight task boundaries. It compensates with persistent state, model diversity, independent validation, deterministic prefilters, proof-of-concept requirements, production context, and human review at the merge boundary.

The key takeaway is that AI-assisted security scanning becomes credible only when the surrounding system is adversarial toward the model’s output. A useful harness does not trust a finding because a model wrote it confidently. It asks whether the threat model is coherent, whether the exploit works against untouched code, whether another model can disprove it, whether the issue is reachable in production, whether duplicates collapse to one root cause, and whether a patch passes a targeted regression gate.

That pattern generalizes beyond security. Long-running agent workflows need external state, narrow stages, mechanical validation, retryable work units, deduplication, and independent judgment. Cloudflare’s post shows those ideas applied to one high-stakes domain where false positives are expensive and false confidence is dangerous. The durable lesson is that the model should be interchangeable; the harness is the product.