Generated by Codex with GPT 5.6 Sol High

OpenAI’s official engineering blog published “Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II)” on October 23, 2026. The post explains how Habitat, OpenAI’s application-storage platform, grew into a globally distributed layer holding more than 500 petabytes while insulating product teams from the operational details of Azure Cosmos DB.

The central lesson is not simply that OpenAI needed a bigger database. Its harder problem was choosing where to place responsibility. A storage design can isolate every workload and make each product team plan its own capacity, or pool workloads and let a platform team exploit aggregate demand. Habitat tried both extremes. The system eventually settled on shared infrastructure with stronger controls, because technical isolation that pushed continual capacity planning onto every product team did not scale organizationally.

The path from sharing to isolation and back

Habitat initially placed many workloads in a shared Cosmos DB container. Cosmos partitions data by a logical partition key, but enforces throughput limits at the underlying physical-partition level. Two consequences emerged at OpenAI’s scale. Common logical keys such as user_id collided across otherwise unrelated product features, and a hot workload could consume the capacity of a physical partition shared with critical paths such as login or signup.

Storage growth created another failure mode. As a physical partition filled, Cosmos split it, while the container’s provisioned throughput stayed fixed. Each child therefore inherited a smaller share of the same throughput budget. A blob-like workload could become slower merely because its stored data grew, even if its request rate did not.

OpenAI responded by moving important workloads into dedicated containers. This reduced blast radius, but it replaced shared-system failures with a coordination problem. A feature that rose from test traffic to a large launch barely moved demand in a huge pooled container, yet could exceed a small dedicated container by orders of magnitude. Product teams had to predict launches, provision headroom, and keep revisiting capacity. Misses turned launch days into incident-response sessions.

The final design returned to multi-tenancy by default, but with better primitives. Hot-key throttling sheds the most concentrated demand before it harms unrelated keys. Quality-of-service tiers throttle bulk inserts and extracts before interactive requests. An improved serverless mode lets each physical partition reach 10,000 request units per second and splits partitions based on request-unit demand as well as storage size. Together, these mechanisms recover the utilization and operational simplicity of pooling without accepting the original noisy-neighbor behavior. OpenAI reports that the shared design has run for months without provisioning or noisy-neighbor incidents despite faster product development.

Managed services become joint systems

At very large scale, the boundary between customer and cloud provider blurred. OpenAI’s largest Cosmos account stores tens of petabytes, serves more than 20 million queries per second, and exhibits failure modes that normal provider testing does not reproduce.

A regional power outage exposed that gap. Failover of the largest accounts would have taken hours. During a later failback, clients stampeded the recovering Cosmos gateway because they immediately opened connections and sent requests to the original region even though only some physical partitions had moved there. The incidents were not isolated configuration mistakes; they showed that neither organization could operate the system safely through conventional support channels.

OpenAI and Azure shifted to co-management: shared dashboards, production-scale non-production testing, joint SDK work, direct engineering communication, and paging across company boundaries. They improved gateway load protection and metadata caching, increased connection reuse across Habitat’s service fleet, colocated workloads where possible, and separated critical and offline traffic into different virtual networks. The broader engineering takeaway is that a sufficiently large consumer of a managed service effectively becomes a participant in that service’s design and operations.

Hiding geography without hiding consistency

Once reliability improved, global latency became the constraint. Compute and storage could not always occupy the same region because hundreds of petabytes compete with serving capacity for datacenter power, cooling, and space. Habitat therefore uses a layered strategy instead of attempting universal replication.

Small, latency-critical datasets such as authentication metadata receive broad geographic replication. Other data is placed near users when capacity and residency rules allow. The remaining read-heavy traffic goes through distributed Valkey lookaside caches located near application compute.

The important detail is invalidation. Long time-to-live values improve hit rates but risk serving stale state; short values preserve freshness but surrender much of the latency benefit. Explicit invalidation from application code is vulnerable to partial failures and multi-region races. Habitat instead consumes Cosmos DB’s change feed and propagates invalidations to regional caches within seconds. The next read misses and fetches the current object. This keeps the benefits of long-lived cached data while reducing prolonged staleness.

Combined with selective replication and a rewrite of the Habitat service from Python to Rust, the approach cut global median read latency from roughly 120 milliseconds to roughly 10 milliseconds—a 12× improvement. Tail requests can still cross regions when data is absent from cache, so placement remains a deeper problem rather than something caching completely solves.

Change capture as a platform

The same change stream that keeps caches fresh also feeds secondary indexes, data movement, analytics, Kafka, and real-time features. Habitat processes millions of changes per second from thousands of containers and millions of partitions through more than one hundred Kafka topics.

That scale turns change data capture into its own distributed platform. A control plane discovers which containers require capture, selects replica regions, and assigns work across thousands of workers. Lower-priority reads prevent CDC from competing with interactive storage traffic. The system isolates failing streams, sends permanent delivery failures to dead-letter queues to prevent head-of-line blocking, moves readers during regional outages, and batches checkpoints in memory to reduce pressure on Cosmos. Product teams see a callback-oriented interface rather than the routing, leases, worker scaling, and failure recovery beneath it.

The durable pattern

Habitat’s evolution is a useful counterexample to designing the final architecture too early. OpenAI first used a shared container, then isolated workloads to buy reliability, then returned to pooling after the database and control plane could enforce fairness. Each stage solved the most urgent constraint while exposing the next one.

The result is not a universal endorsement of multi-tenancy, caching, Rust, or any particular database. It is a sequencing strategy: make the system dependable, remove the most damaging bottleneck, and only then optimize the operating model. At extreme scale, the decisive abstractions are the ones that prevent product teams from needing intimate knowledge of storage topology while still giving the platform enough control to contain hot keys, prioritize interactive work, move data, and recover from failures.