Generated by Codex with GPT-5

What happened

NVIDIA’s official technical blog published Streaming Tokens and Tools: Multi-Turn Agentic Harness Support in NVIDIA Dynamo, a post about making an inference server behave like a first-class backend for modern coding and agent harnesses rather than a plain text-completion endpoint.

The core point is that agentic inference has a richer contract than ordinary chat. A model turn may contain reasoning, tool calls, tool results, more reasoning, and more tool calls, all of which have to be preserved in the structure expected by the client. If the server streams tokens but reconstructs tool calls incorrectly, drops the reasoning that justified a tool call, or loses request metadata during an internal conversion, the model can receive a subtly different conversation on the next turn. The failure mode is not a visible HTTP error. It is a degraded agent that forgets why it called a tool, waits too long to execute tools, or runs with a different harness policy than intended.

Dynamo’s new harness-facing path addresses that contract at several layers. On the frontend, NVIDIA highlights an Anthropic-compatible API, stripping of unstable Anthropic preamble headers, and streaming tool dispatch. On the worker side, model-specific reasoning and tool-call parsers reconstruct the output into the content blocks expected by harnesses such as Claude Code, Codex, and OpenClaw. The framing is useful because it treats parser fidelity, prompt stability, and streaming semantics as serving-system responsibilities, not as incidental glue code outside the inference engine.

The simplest example is prompt-prefix caching. Claude Code sends a large reusable prompt scaffold, but the request can begin with a session-specific billing header. If that varying header is tokenized at position zero, the rest of the stable prompt no longer lines up for KV-cache reuse. Dynamo’s --strip-anthropic-preamble removes the unstable header before tokenization. In NVIDIA’s B200 experiment with a 52K-token prompt, the stable-prefix path reached roughly 168 ms time to first token, while leaving the varying header in place pushed that to about 912 ms. Removing the header restored cache reuse and brought the path back to roughly 169 ms.

That detail matters because many serving optimizations assume stable prefixes. For agent harnesses, the static prompt is often enormous: base instructions, tool definitions, safety policy, environment rules, and model-specific behavior all precede the user’s actual task. A single unstable line can turn a cheap cached prefix into repeated cold prefill. The post’s broader lesson is that billing, metering, routing, and observability metadata must be kept out of the token prefix unless the model is meant to see it.

The second major mechanism is reasoning and tool parsing. Modern reasoning models do not always emit a single block of hidden reasoning followed by a single tool call. They can interleave reasoning spans and tool calls inside the same assistant turn. In that shape, order is semantic. A reconstruction that groups all reasoning first and all tools second may contain the same tokens, but it destroys the relationship between each reasoning span and the tool call it explains. Dynamo now makes parser ownership explicit, uses template-native reasoning behavior when the model template supports it, and applies per-request thinking controls so ordinary turns can still truncate reasoning while tool-calling turns preserve the reasoning needed for the next step.

This is also a latency problem. If reasoning content mutates between turns, the next prompt prefix can miss cache even when the visible conversation looks nearly identical. NVIDIA reports a separate B200 experiment where an unchanged next-turn prefix with about 500 tokens of assistant thinking reached about 167 ms TTFT, while mutated thinking pushed TTFT to about 322 ms. Correctness and performance are coupled: preserving the right structured history avoids both semantic drift and avoidable prefill work.

The streaming tool-call change is the most operationally concrete. In older Dynamo behavior, reasoning tokens could stream, but tool calls were effectively held until the end of the turn. That made the interface feel responsive while still delaying the actual work. Dynamo’s dispatch mode emits a typed tool_call_dispatch event when a complete tool payload has been parsed. The harness no longer has to buffer deltas, guess whether JSON arguments are complete, or wait for final turn completion before starting the tool. Tool execution can begin as soon as the serving stack has a coherent call.

NVIDIA then shows that API compatibility is deeper than schema compatibility. Claude Code and OpenClaw need model metadata, slashed model IDs, valid token counts at stream start, and cache_control acceptance. Codex has its own version of the same issue through the OpenAI Responses API and local model-catalog metadata. The selected model profile controls base instructions, tool-output truncation, reasoning summaries, image support, verbosity controls, and parallel tool-call behavior before the request ever reaches Dynamo. In NVIDIA’s 50-task SWE-Bench Verified subset, using a fallback Codex model profile caused roughly half as many tool calls as the intended gpt-5.5 profile. Adding the proper model-catalog alias brought tool-call behavior much closer to the native profile.

Why it matters

The post is valuable because it treats agent serving as a distributed-systems interface problem. A fast decoder is necessary, but it is not sufficient. The serving stack has to preserve conversation state, tool structure, cache identity, model metadata, token accounting, and streaming event order across multiple turns. These details sit below the product UI but above raw model execution, which makes them easy to overlook and expensive to debug.

The design also clarifies where agent latency comes from. Some latency is raw compute. Some is avoidable prefill caused by unstable prompts or mutated reasoning history. Some is harness idle time caused by buffering tool calls until a stream ends. Dynamo’s changes attack the latter two categories by making prompt identity stable and by turning parsed tool calls into first-class streaming events. That is a different optimization mindset from simply making tokens faster: it reduces wasted work and lets external tools run earlier.

The Codex and Claude Code examples are a useful warning for teams building custom model gateways. Passing an HTTP compatibility test does not guarantee behavioral parity with a native backend. If the model ID maps to the wrong catalog profile, if token counts arrive too late for context compaction, or if the server collapses model-specific reasoning blocks into a generic representation, the agent may still run but with different policies and worse outcomes. Compatibility has to be measured at the harness-behavior level, not just the endpoint level.

There is a broader architecture lesson in the parser extraction work. Dynamo is moving protocol, parser, and tokenizer layers into reusable crates, which suggests that agent-serving systems need explicit, versioned components for the interface between model tokens and client-visible events. Treating parsers as ad hoc string handling inside each harness scales poorly because every model family and every API shape has slightly different reasoning, tool-call, and history-replay rules.

Takeaway

The engineering takeaway is that production agent systems need serving infrastructure that understands the shape of agent work. Tool calls, reasoning replay, request compression, token accounting, and model metadata are part of the execution contract. If they are handled as afterthoughts, the result is slower and less faithful even when the model and hardware are strong.

Dynamo’s pattern is to keep the model-facing prefix stable, parse model-specific reasoning and tools before the harness consumes them, dispatch complete tool calls as typed stream events, and verify compatibility using real agent clients rather than only API schemas. For teams running coding agents or long-running tool-using agents on custom inference backends, that is the practical bar: the backend must preserve not only tokens, but the structured state that lets the next turn mean the same thing as the previous one intended.