Generated by Codex with GPT-5

What happened

Cloudflare’s official blog published Scaling Security Insights: how we achieved a 10x increase in global scanning capacity, a June 12, 2026 engineering post about turning a strained security-scanning pipeline into a system that could run for every account and zone by default.

Security Insights is Cloudflare’s system for regularly scanning accounts, zones, and DNS records for risky configuration states. The operational problem was straightforward: scans were too infrequent, often weekly or biweekly, and many free-plan accounts were not automatically scanned at all. To make scanning universal and more frequent, Cloudflare estimated that the pipeline needed to move from roughly 10 scans per second to around 100 scans per second.

The post is valuable because the team did not frame the work as a wholesale platform rewrite or a generic “scale out” story. The original architecture already had recognizable building blocks: a scheduler emitted scan messages into Kafka, specialized Go checkers consumed those messages, and checkers sent discovered insights to an internal API backed by Postgres. The hard part was that each layer had accumulated hidden bottlenecks. Kafka lag grew into millions of events, API calls timed out, and checker processes crashed under load.

The resulting 10x improvement came from understanding those bottlenecks precisely enough to avoid blunt fixes. Cloudflare improved concurrency inside the Kafka consumers, separated slow and fast work, replaced row-at-a-time database writes with set-oriented paths, moved API traffic closer to the primary database, and redesigned scheduling so demand arrived smoothly instead of in account-shaped bursts. The system now sustains more than 120 scans per second during peak scheduling, which was enough to enable automatic scans for all plans and increase scan frequency across tiers.

The mechanism

The first lesson is that Kafka was not behaving like an elastic work queue. In a Kafka consumer group, each partition can only have one active consumer, and messages within a partition are consumed in order. That meant a slow message could block everything behind it, and each checker could not scale beyond the partition count without changing the topic topology. Cloudflare could have added partitions, but that would have pushed resource cost onto a shared broker before the team had proven that the application layer was efficient.

The team instead introduced batch consumption with parallel per-message processing inside each checker. The consumer still respected Kafka’s ordered consumption model, but after reading a batch it could process messages concurrently in goroutines. That increased useful parallelism without immediately changing the broker. The tradeoff was explicit: a crash in the middle of a batch could force more work to be retried, and each process would hold more in memory. For this workload, retrying scan messages was acceptable, and the memory increase was smaller than the benefit of unblocking throughput.

Parallelism alone did not solve head-of-line blocking because some scan messages represented much larger accounts or zones than others. A single large account could take minutes or hours while typical scans completed in seconds or milliseconds. Cloudflare split the checkers and consumer groups into fast and slow lanes. The fast lane could quickly identify work that belonged in the slow lane and skip it, while the slow lane had dedicated capacity for long-running scans. That is a simple design, but it matters because it keeps tail latency from consuming the same resources as the median case.

The next bottleneck was Postgres. Checkers wrote insights through an internal API endpoint that looped over every issue and executed an INSERT ... ON CONFLICT DO UPDATE call one row at a time. For a large result set, the endpoint could perform hundreds of thousands of database round trips inside a single request. Cloudflare first tried the common bulk-loading answer, COPY into a temporary table, but it caused bloat in Postgres system tables. The final solution was hybrid: use UNNEST for result sets below a threshold, and use COPY for very large sets. That kept small writes in the millisecond range while still making huge writes complete in seconds.

The API timeout story is a reminder that distributed placement can dominate code-level optimization. Cloudflare’s primary database was in Portland, while the internal API was active-active in Portland and Amsterdam. Load balancing sent some checker connections to the Amsterdam API, which then had to make database calls across a transatlantic round trip. The average API call was about 10 ms in Portland and nearly 3 seconds in Amsterdam. Worse, long-lived checker connections meant some Kafka partitions were effectively bound to the slow path, so lag accumulated unevenly by partition. Switching the API to active-passive, with the active instance following the primary database, removed the latency problem without changing the data model.

Finally, the scheduler had to be made smooth enough for the improved pipeline to matter. Its previous logic scheduled scans from account-level timestamps. That produced bursts when many accounts had similar last_scheduled_at values, and accounts with many zones created cascades of zone scans. Shortening the scan interval made the problem worse because a large fraction of the fleet would suddenly become due.

Cloudflare changed the scheduler in three ways. Zones received independent scheduling timestamps instead of inheriting account-level bursts. Existing timestamps were randomized so the current database state did not preserve old synchronization artifacts. An adaptive rate limit recalculated every half hour based on account counts, zone counts, and target scan frequencies. That rate limiter let the scheduler spread due work across time while still increasing automatically as the customer base grew.

Why it matters

The broader engineering takeaway is that capacity is often trapped in mismatched assumptions, not only in insufficient hardware. Kafka partitions looked like the natural scaling boundary until the team separated consumption order from concurrent processing. Postgres looked overloaded until the API stopped paying one round trip per insight. The API looked unreliable until metrics showed that half the traffic path was accidentally remote from the database. The scheduler looked like a background process until its burst shape threatened Kafka retention and downstream stability.

The post is also a useful example of preserving a system’s basic architecture while changing the pressure points that determine whether it can operate at product scale. Cloudflare still has a scheduler, Kafka, Go checkers, an API, and Postgres. What changed is the control over variance: slow messages are isolated, writes are batched by size, database locality is intentional, and scan issuance is rate-shaped. Those choices are less dramatic than a rewrite, but they are exactly the sort of changes that make production systems dependable.

For security engineering, the result is more than a throughput chart. Configuration risks are time-sensitive; a misconfigured zone that waits two weeks for detection gives attackers a larger window. By making scans default and more frequent, Cloudflare turned pipeline capacity into security coverage. The operational win therefore improves the product’s defensive posture, especially for customers who would not have opted into scanning themselves.

Takeaway

Cloudflare’s Security Insights work is a clean production lesson: scale the invariant the product actually needs, not the component that first appears saturated. The invariant was regular, universal scanning with bounded backlog. Achieving it required work at every boundary where the old system amplified variance: Kafka partition order, outlier scan sizes, row-oriented writes, cross-region database calls, and bursty scheduling.

The strongest part of the post is the restraint. The team considered heavier options, including more Kafka partitions and a larger architectural replacement, but first used metrics, logs, SQL inspection, and workload shape to find local fixes. The result exceeded the 10x target without simply adding more pods or relaxing timeouts.

That pattern generalizes well beyond security scanning. Any high-volume background platform that mixes many small jobs with occasional large ones needs to protect the fast path, batch the write path, keep latency-sensitive services close to their state, and shape scheduling demand before it reaches the queue. Cloudflare’s post shows how those familiar ideas combine into a practical scaling strategy when the goal is not peak benchmark throughput, but dependable coverage for millions of real accounts and zones.