Generated by Codex with GPT 5.6 Sol High

Reinforcement-learning infrastructure has an unusual performance problem: average sandbox speed is not enough. Many training algorithms launch groups of rollouts in parallel and wait for the slowest member before taking the next gradient step. A host can therefore appear healthy at the median while a small tail of delayed environments quietly determines the pace and cost of the entire training run.

The AWS Compute Blog published “Scaling RL rollouts with Firecracker on Amazon EC2 metal”, a measurement-driven guide to finding the highest useful density for isolated training sandboxes. Its central result is simple but consequential: on the tested systems, tail latency stayed nearly flat until density reached roughly one Firecracker microVM per vCPU, then rose sharply even though median latency barely moved.

The sandbox host as a scheduling system

Large-scale reinforcement learning with verifiable rewards can require thousands of disposable environments. Each rollout may run code, execute tests, modify files, or maintain episode state, so isolation is essential: one attempt must not leak state into the next. Firecracker provides that boundary with lightweight microVMs, each running its own kernel, while EC2 metal exposes the physical machine closely enough to control CPU and memory placement.

The reference architecture places a host agent on every bare-metal Kubernetes node as a DaemonSet. The agent discovers the machine’s CPU topology, creates a cgroup hierarchy for each microVM, launches Firecracker through its jailer, and maintains a warm pool of already-booted guests. When work arrives, it dispatches into that pool instead of paying a cold-boot penalty. Per-guest controls cover CPU pinning, NUMA-local memory, memory size, and placement within cache domains.

Those controls cannot be reduced to one universal configuration. Intel-based m8i exposes simultaneous multithreading, while m8a and m8g do not. L3 cache layouts and NUMA boundaries also differ. Isolated cgroup partitions can keep kernel workqueues away from pinned cores, but require a recent Linux kernel. The useful optimization surface is therefore the interaction between a particular host topology and a particular rollout workload.

Measuring the density cliff

AWS built an open measurement harness called labsweep to map that interaction. It reads host topology from sysfs, generates a matrix of placement configurations and microVM densities, launches jailed guests with bounded concurrency, runs the workload in every guest at once, and records raw samples plus percentile and variance statistics. Results carry the instance type, CPU model, kernel, and Firecracker versions, and failed guests remain visible instead of disappearing from the dataset.

The study collected more than 100,000 samples across over 500 configurations. Workloads included short verification jobs, build-and-link tasks, complete replayed 14-turn RL episodes, and a synthetic compute loop used as a control. Storage design and network latency to the model endpoint were held constant, so the comparison isolates CPU and memory placement. The published latency results cover m8i and m8a; m8g measurements were still pending.

Across the tested workloads, the important boundary was about 1.0 microVM per vCPU. At 0.94 microVM per core, one representative rollout had a median latency of about 7.74 milliseconds and a p99 of 8.04 milliseconds. The median remained essentially unchanged as density increased, but p99 climbed to 9.91 milliseconds at 1.15x density and roughly 15.6 milliseconds at 1.44x. A dashboard centered on averages or medians would miss the collapse.

That tail becomes more damaging when the training algorithm synchronizes a rollout group. With 64 rollouts per group, a one-percent tail event appears in nearly half of all groups. In AWS’s measurements, the estimated group-completion amplification rose from 1.04x at 0.94 density to 1.32x at 1.15 and about 2.0x at 1.44. Packing more guests onto the host therefore stopped increasing useful training throughput well before the machine looked overloaded by ordinary metrics.

Pinning, SMT, and workload shape

Core pinning did not help the synthetic compute loop at equal density, a useful control against treating affinity as magic. It did tighten the p99-to-p50 spread for real sandbox workloads, especially near the density ceiling, because it removed scheduler interference from the critical tail. That is the quantity a synchronized RL step actually experiences.

Simultaneous multithreading produced a more nuanced tradeoff. Many agent sandboxes spend substantial time waiting on process creation, file operations, Git, or other I/O. For those workloads, SMT exposed more vCPUs and delayed crossing the one-microVM-per-vCPU boundary; at high density it delivered as much as 1.49x throughput. For compute-heavy loops, disabling SMT yielded tighter and more predictable tails. Even within the tested RL workload set, enabling SMT improved peak throughput by different amounts: 1.14x for verification, 1.13x for builds, and 1.17x for replayed episodes.

Memory can move the ceiling as well. The M-series systems in the experiment provide 4 GiB per vCPU, and every tested workload exhausted CPU capacity before memory. Guests with larger working sets could reverse that relationship, making a memory-optimized instance preferable even though it offers fewer vCPUs and therefore a lower maximum sandbox count.

Optimize the training step, not the guest

The broader engineering lesson is to measure the unit the system is trying to accelerate. For grouped reinforcement learning, that unit is not an individual microVM invocation or even total guest completions per second. It is the synchronized training step after stragglers have been amplified by group size.

That changes capacity planning. The best operating point is the highest density at which group-completion amplification remains flat, not the highest density that boots successfully or preserves median latency. Hardware specifications and generic tuning rules can narrow the search, but only a sweep using the real root filesystem, workload mix, group size, and target instance can reveal the actual boundary. By publishing the harness, deployment stack, and raw methodology, AWS turns that principle into a repeatable test rather than a rule of thumb.