Generated by Codex with GPT 5.6 Sol High

The hard part of building a frontier-model cluster is not collecting a large number of GPUs. It is making hundreds of thousands of accelerators, spread across buildings and separated by as much as 100 kilometers, behave like one dependable machine. At that scale, network topology, firmware rollout, spare capacity, cooling, power delivery, and even the failure rate of individual signaling links all become part of model-training performance.

The official Engineering at Meta post, “Inside Prometheus: Meta’s Journey to a Gigawatt-Scale Supercluster,” explains how Meta rebuilt its New Albany, Ohio, site around that problem. Prometheus already spans multiple buildings and hundreds of thousands of GPUs; Meta intends it to reach roughly one million. The first phase trained the Muse model family while sustaining more than 85% goodput for pre-training and more than 90% for reinforcement learning. Those figures matter more than peak FLOPS: goodput measures how much expensive hardware time actually becomes useful training progress.

A hierarchy of compute and communication

Prometheus combines multiple hardware generations rather than waiting for a single uniform fleet. Its Catalina platform uses NVIDIA GB200 NVL72 systems, with a 72-GPU scale-up domain split across two racks. Meta chose the Ariel configuration, which gives every GPU 800 Gbps of scale-out bandwidth through two ConnectX-7 NICs. That decision addressed an imbalance: compute, memory, and NVLink bandwidth had all grown sharply from the H100 generation, while a conventional network configuration would have left off-rack bandwidth at 400 Gbps per GPU.

The newer Clemente platform uses GB300 NVL72. Higher memory capacity and native 800 Gbps networking allow the same 72-GPU domain to fit in one rack, with fewer cables and a 2:1 GPU-to-CPU ratio. Meta reports a 5–10% performance improvement over GB200 while supporting either modular air-assisted cooling or facility liquid cooling. A Vera Rubin-based platform is planned next. The useful architectural point is that the cluster is designed to absorb successive accelerator generations without rebuilding every surrounding layer at once.

The network mirrors the hierarchy in distributed training. Latency- and bandwidth-sensitive collectives occur mainly inside a data-parallel replica, while communication between replicas tolerates somewhat more distance. Within a building, an Ethernet fabric connects as many as 20,736 GPUs. Across buildings and sites, a Backend Aggregation network uses deep-buffered switches and multiple petabits per second of regional capacity. The fabric spans links from 2 kilometers to more than 100 kilometers, so the software team co-designs parallelism and collective communication with the physical topology instead of pretending every GPU is equally close.

Meta’s fabric also evolved with the NICs. The original Disaggregated Scheduled Fabric performed sub-packet multipath routing and reordering inside the network to stop long-lived training flows from colliding. Newer endpoints can reorder packets themselves, allowing a Non-Scheduled Fabric built from shallow-buffer 51.2-terabit switches and adaptive routing. Both run Meta’s FBOSS stack. Owning the network software gives the operator a way to change routing, telemetry, and failure handling as the hardware changes, rather than treating the fabric as a fixed appliance.

Designing around a 72-GPU failure domain

NVL72 makes 72 GPUs act like one large accelerator, but the same coupling expands the blast radius of a fault ninefold compared with an eight-GPU NVL8 system. Meta therefore treats the entire rack-scale domain as the unit of capacity and resilience.

Each NVL72 domain runs a 64-GPU training slice and keeps eight GPUs as in-rack spares. If one GPU fails, a spare can replace it and the job resumes from its latest checkpoint without waiting for a physical repair. Meta also reserves roughly 2% additional racks for whole-rack failures. A 32,768-active-GPU job consequently requires 37,656 installed GPUs—about 13% extra capacity, most of it created by the 64-of-72 geometry rather than a generic reliability margin.

Fault recovery then depends on scope. Parallelism-Aware Fault Tolerance removes and restarts one failed data-parallel replica while the remaining replicas repeat the current step and continue. A larger failure triggers a full restart from a distributed checkpoint. Meta stages model, gradient, and optimizer state in memory and writes delta checkpoints asynchronously to multi-tier storage so that protection does not consume too much training throughput.

That combination is a useful general reliability pattern. Local failures should be absorbed locally; only failures that cross the isolation boundary should invoke global recovery. Hardware spares, replica-aware software, and checkpoints are not competing strategies. They cover different fault sizes and together reduce lost work.

Reliability is a production workload

The scale of Prometheus turns tiny component defect rates into continuous operational events. One NVL72 domain contains 5,184 tuned differential-pair links running at 200 Gbps. Across the GB200 fleet, Meta monitors more than 600,000 NVLink connections. Early in the deployment, Prometheus experienced nearly 1,000 interruptions per 8,000 GPUs per day. Meta reports reducing that rate by 99% through telemetry, automated diagnostics, firmware changes, memory-error analysis, fabric recovery, and disciplined repair validation.

The details show why “hardware reliability” is really a full-stack systems problem. Diagnostic burns find weak NVLink paths before production. Classifiers separate transient errors from persistent signal-integrity defects. HBM and LPDDR failures are mitigated through firmware tuning, memory scrubbing, improved error correction, and prediction that retires deteriorating devices early. Millisecond-resolution fabric monitoring isolates switch faults and rebalances traffic, while the aggregation network reroutes around links without disrupting synchronized GPU bursts.

Meta and NVIDIA operate a joint triage process in which every halt is attributed to a root cause and followed through to a targeted repair. Firmware bundles can now reach more than 100,000 GPUs within hours, and the 90th-percentile upgrade time for a pod fell from more than seven hours to two. Repaired systems enter quarantine and must pass validation before rejoining the compute pool. The next target is two interruptions per 8,000 GPUs per day, equivalent to roughly 100,000 GPU-hours between interruptions.

The lesson is that fleet reliability cannot be summarized by component uptime. What matters is training momentum: how often a job stops, how much work it loses, how quickly capacity returns, and how reliably a fix can be propagated. Measuring those outcomes creates a feedback loop from production incidents back into firmware, diagnostics, scheduling, and hardware design.

Power and cooling join the control plane

Synchronized training produces synchronized electrical load. When tens of thousands of GPUs alternate between compute and communication, cluster demand can swing by 30–40%; starting or pausing a job can change consumption by tens of megawatts within seconds. Prometheus manages these transients with Dynamo Dimmer, which gradually caps GPU power near the site limit instead of stopping work. Rack batteries, capacitor trays, facility-scale battery storage, and utility coordination absorb changes at different timescales.

Cooling is similarly heterogeneous. Prometheus mixes conventional buildings, rapid-deployment structures, modular air-assisted liquid cooling, and facility liquid cooling. That variety is not aesthetic—it lets Meta add capacity faster while adapting to site constraints and to racks drawing roughly 120 to 135 kilowatts today, with more than 170 kilowatts planned for Vera Rubin systems.

The broader engineering takeaway is that a training supercluster is not merely a bigger data center. At this scale, the boundaries between accelerator architecture, distributed software, network routing, failure recovery, firmware operations, power control, and building design disappear. Prometheus works by exposing those dependencies and engineering them together. Peak hardware capability establishes the ceiling; useful model progress comes from the less glamorous work of keeping the entire machine moving near that ceiling despite distance, faults, upgrades, and physical limits.