Generated by Codex with GPT 5.6 Sol XHigh

Large-model inference is often described as a race for faster kernels, but production serving is at least as much a problem of fitting the right data into memory at the right moment. In “Smaller, faster, safer: running Kimi and GLM at scale,” the official Cloudflare Blog explains how Workers AI serves long-context mixture-of-experts models by treating prefill and decode as different workloads, compressing only where the tradeoff helps, and adding an integrity check for the shared memory that higher density puts at risk.

The result is a useful systems lesson: the best representation is not universal. It depends on which phase is running, what resource constrains that phase, and how much concurrency the service must sustain.

Optimize the bottleneck, not the datatype

During inference, a model keeps attention keys and values for prior tokens in a KV cache so it can generate the next token without rereading the whole context. For long-context models, that cache can consume GPU memory faster than the weights themselves. Cloudflare stores Kimi K2.6’s KV cache in FP8 rather than BF16, halving the cache footprint and increasing resident context capacity from roughly 686,000 to 1.37 million tokens.

That change is not a free speedup. An FP8 attention kernel must do conversion work as it reads the cache, so BF16 is modestly faster at any concurrency level it can accommodate. The advantage appears at the system level: BF16 reaches 1,558 tokens per second at 32 concurrent requests and then runs out of memory, while FP8 admits 64 requests and reaches 2,192 tokens per second. Cloudflare reports about 41% more throughput than the BF16 peak and roughly 30% lower cost per token. Evaluation scores and tool-call validity remain effectively unchanged.

The important architectural choice is that Workers AI separates prefill—the initial processing of the prompt—from decode, which emits subsequent tokens. Prefill is compute-bound, so Cloudflare retains BF16 there and avoids FP8’s conversion cost. Decode is constrained by memory capacity and bandwidth, so the smaller cache has much greater value. Disaggregation turns what would otherwise be a fleet-wide compromise into a phase-specific decision.

Cloudflare applies the same reasoning to GLM 5.2’s weights. Compressing them from FP8 to INT4 shrinks the checkpoint from 705 GB to 421 GB and cuts per-GPU weight memory in an eight-way tensor-parallel deployment from about 88 GB to 52 GB. The remaining space can hold roughly 1.18 million KV-cache tokens.

For decode, smaller weights also mean less data must be streamed from GPU memory for every generated token. At one concurrent request, INT4 raises throughput from 60 to 92 tokens per second; across the tested concurrency levels, gains range from 16% to 55%. But INT4 loses during compute-bound prefill because its weights must be expanded before multiplication: FP8 sustains about 10,160 prompt tokens per second, compared with 8,660 for INT4. Workers AI therefore uses FP8 weights for prefill and INT4 for decode. Model quality stays within 0.8 benchmark points of the FP8 version.

Higher density needs stronger isolation

Memory optimization creates a second-order reliability problem. Paged attention, continuous batching, and cache reuse allow many requests to share the same physical GPU memory. Doubling concurrency makes that machinery more economical, but also raises the consequence of a bookkeeping error: a request could read a cache page that has been reassigned to another request. At production scale, even extremely rare faults deserve an explicit defense.

Cloudflare’s KV-cache integrity layer attaches a changing tag to every physical cache page. The server records the pages and tags expected by each request, then checks those mappings before supported decode operations read the cache. A mismatch aborts the affected request rather than allowing potentially incorrect or cross-request data to reach the response.

The implementation reflects the same bottleneck-aware discipline as the quantization choices. Validation runs as a separate batch check instead of being fused into the attention kernel, avoiding a race among GPU thread groups. In tests using 8,192-token inputs and 1,000-token outputs, the check changes throughput and p95 latency by less than 1% across the measured concurrency levels. Deployments that do not enable it use a no-op tracker and pay no measurable overhead.

The broader engineering takeaway

Cloudflare’s optimizations work as a coordinated design, not as three independent tricks. Disaggregated serving exposes the different economics of prefill and decode. That separation makes phase-specific quantization possible. Quantization increases capacity and request density. Higher density, in turn, makes cache isolation more important, so the service adds a narrowly targeted integrity mechanism whose cost is measured before rollout.

This is a stronger pattern than “use lower precision.” Production inference should identify the binding resource for each phase, choose representations against that resource, verify quality with task-relevant evaluations, test performance across realistic concurrency rather than a single microbenchmark, and revisit safety assumptions when optimization changes the amount of shared state. The winning design is not the most compressed one; it is the one that improves fleet-level throughput and cost without silently trading away latency, accuracy, or isolation.