Generated by Codex with GPT-5

What happened

NVIDIA’s official technical blog published How the NVIDIA Vera Rubin Platform is Solving Agentic AI’s Scale-Up Problem, a post about the hardware, networking, compiler, and serving-stack design needed to make long-context agentic inference both fast and economical at frontier scale.

The post starts from a useful premise: agentic inference is not just ordinary batched inference with more tokens. A single user session can expand into a sequence of model calls, tool invocations, observations, retries, subagents, and long conversation state. Each branch carries its own system prompt, tool definitions, accumulated KV cache, and new tokens. When that state is routed through trillion-parameter mixture-of-experts models, the serving system has to move activations and cache-dependent work across many accelerators while still keeping per-token latency low enough for an interactive product.

That changes the network requirement. Large training runs and high-volume inference can often tolerate some jitter because big batches and throughput-oriented scheduling smooth it out. Agentic decode has a harsher shape: smaller batches, many dependent turns, long context, and user-visible latency that compounds across hundreds of requests. If every cross-chip hop is merely statistically bounded, tail latency can become product latency. NVIDIA’s framing is that the scale-up network has to be co-designed with the silicon and compiler, not treated as a generic fabric that absorbs contention at runtime.

The scale-up piece in the post is NVIDIA Groq 3 LPX and its LPU chip-to-chip interconnect. Instead of relying on a runtime-arbitrated network, the design extends Groq’s deterministic execution model across many LPUs. The mechanism has three parts: high-radix point-to-point links, compiler-scheduled data movement, and hardware-driven near-synchronous timing.

The point-to-point fabric gives each LPU 96 chip-to-chip links at 112 Gbps, which NVIDIA describes as roughly 2.5 TB/s of scale-up bandwidth per LPU and 640 TB/s at the rack level. The topology uses direct peer connections, symmetric routes, and low hop counts so collective communication can be planned rather than discovered under load. This is the first important architectural choice: the network is shaped to look less like a shared packet fabric and more like a predictable extension of the execution surface.

The compiler then takes responsibility for communication. Data moves between LPUs in fixed 320-byte vectors, the same unit used for computation, and those transfers are scheduled as first-class operations beside matrix, vector, and switch execution. The compiler decides when a vector leaves, which link it takes, and when it should arrive. In other words, route selection, load balancing, and synchronization happen ahead of time rather than being delegated to hardware schedulers reacting to contention.

That only works if timing is stable enough for the plan to hold. The post’s third mechanism is a plesiosynchronous chip-to-chip protocol: each LPU has its own clock, but the system manages clock drift so many LPUs can behave like one coordinated execution cluster. With predictable arrival times and periodic synchronization, the runtime needs less defensive buffering, and network latency can become a compile-time property instead of a variable discovered during serving.

The payoff is rack-scale determinism. NVIDIA says the LPX rack provides 128 GB of unified on-chip SRAM across the tensor-parallel domain and can partition trillion-parameter models across that pool with strategies such as layer-wise partitioning. For an agentic workload, that matters because the system is trying to preserve low per-token latency while avoiding cuts to context length, model quality, or tool-heavy interaction patterns. The difficult part is not only fitting the model; it is keeping the decode loop predictable when the session fans out and cache state grows.

NVIDIA does not position LPX as replacing the GPU pool. The post instead describes a heterogeneous serving path with Vera Rubin NVL72 and NVIDIA Dynamo. Vera Rubin NVL72 handles the throughput-heavy side: prefill, long-context decode attention, concurrent serving, and large KV-cache reads. NVIDIA gives rack-scale figures of up to 3,600 PFLOPS of NVFP4 compute, 20.7 TB of HBM4, and 1.6 PB/s of memory bandwidth. LPX handles the part where small batches and sequential token generation make micro-jitter expensive.

The serving architecture is Attention-FFN Disaggregation. In NVIDIA’s sketch, Rubin GPUs run decode attention over the accumulated KV cache, LPX accelerates feed-forward network execution, and Dynamo orchestrates low-overhead, KV-aware transfers of intermediate activations on every generated token. The division of labor is the important part. Attention over long context benefits from high bandwidth and amortization across cache reads. FFN decode is more latency-sensitive and less forgiving of scheduling noise. Splitting those phases lets the serving system use different hardware timing regimes inside one model-serving loop.

The headline claim is that this co-designed path can deliver 400 tokens per second per user on trillion-parameter MoE models with 400K-token context, with much higher throughput per megawatt than GB200 NVL72 for this specific class of agentic workloads. The exact economic comparison is NVIDIA’s claim, but the engineering argument is more broadly useful: once inference becomes a long-running, stateful, tool-using process, the bottleneck is not a single accelerator metric. It is the interaction among cache placement, expert routing, compiler scheduling, interconnect determinism, and request orchestration.

Why it matters

The post is valuable because it makes agentic inference look like a full-stack systems problem. In a simple chat workload, a serving team can often talk about latency in terms of prefill, decode throughput, batching, and model size. In an agentic workload, those terms are still necessary but incomplete. The system also has to preserve responsiveness across many dependent turns, tool calls, subagents, growing prompts, and reused KV state. Small per-token delays can multiply into a slow session.

The most interesting design choice is moving uncertainty out of the hot path. A conventional network makes many decisions dynamically: routes, queues, flow control, and buffering respond to current traffic. That can be efficient for aggregate throughput, but it makes worst-case latency harder to reason about. NVIDIA’s LPX approach moves those decisions into the compiler and timing protocol. Communication becomes scheduled work, not background infrastructure. That is a meaningful systems pattern for AI serving: if the workload is predictable enough at the execution-graph level, determinism can be more valuable than opportunistic flexibility.

The heterogeneous split also reflects where inference systems are headed. There is no single “best” accelerator behavior for all phases of a frontier model request. Long-context attention wants bandwidth, memory capacity, and high concurrency. Sequential decode wants low jitter and predictable handoff. Tool-using agents add another layer by making requests bursty, branching, and hard to batch cleanly. A serving platform that can route different phases of the same token loop to different hardware classes may be more realistic than expecting one rack architecture to optimize every point on the latency-throughput curve.

There is also a compiler lesson. As models span more chips, the compiler is no longer just producing efficient kernels inside an accelerator. It is scheduling a distributed execution surface. That includes communication, synchronization, and placement. The boundary between “model compiler” and “cluster scheduler” becomes less clean when a single token depends on thousands of coordinated chip-level operations.

Takeaway

NVIDIA’s Vera Rubin and LPX post is ultimately about making agentic inference deterministic enough to operate as an interactive product at frontier-model scale. The model session is long-running and stateful; the serving path has to move KV cache, activations, attention work, and feed-forward work through a heterogeneous system without letting jitter compound across turns.

The broader engineering takeaway is that agentic AI pushes inference infrastructure away from best-effort throughput alone and toward predictable end-to-end execution. Hardware interconnects, compiler scheduling, memory hierarchy, cache-aware routing, and serving orchestration all become part of the same latency budget. For teams building production agents, the relevant question is not only how many tokens a system can emit in aggregate. It is whether the entire stack can keep a multi-step, tool-heavy session responsive as context, parallelism, and model size grow together.