HeadsUpAI

NVIDIA Dynamo Rebuilds Inference Stack for Agentic Coding Workloads

· Updated

NVIDIA updated Dynamo, its open-source inference orchestration layer, with optimizations for agentic AI (systems that plan and act autonomously). The update introduces agent_hints, an API for passing metadata like task priority to the stack, plus native support for protocols that handle interleaved thinking and tool calls.
Throughput increase
Up to 7x more
Cache hit rate
85-97%
Memory hierarchy tiers
4 (GPU, CPU, NVMe, Remote)
Routing performance
170M ops/s
Read to write ratio
11.7x
Supported protocols
v1/chat, v1/responses, v1/messages

Traditional inference was built for chat, but agents make hundreds of API calls with 97% context overlap. This "write-once-read-many" pattern overwhelms standard caches. These optimizations follow NVIDIA Dynamo 1.0 to give self-hosted models the same cache-reuse efficiency found in frontier APIs.

You can now use the Dynamo router for KV-aware placement (routing based on stored model memory), sending requests to the GPU worker holding the relevant context. The system uses a four-tier memory hierarchy to pin system prompts while evicting ephemeral reasoning tokens. The agent_hints API is available on GitHub.

View the full update on developer.nvidia.com
NVIDIA AI
NVIDIA AI
@NVIDIAAI
X

Traditional inference wasn’t built for agentic coding. Agentic tools make hundreds of API calls per coding session, often with recomputed context, creating bottlenecks that drive up cost per token. NVIDIA Dynamo rebuilds the stack for agents with: → KV-aware routing → Agent-aware scheduling → Multi-tier caching → Unified orchestration The result: higher cache hit rates, lower latency, and up to 7× more throughput: https://t.co/E9tRgiLmar

49retweets478likes
View on X

Still wondering? A few quick answers below.

NVIDIA Dynamo is an open-source inference orchestration layer that acts as a distributed operating system for AI clusters. It coordinates between agent frameworks and inference engines like vLLM or TensorRT-LLM. It is designed to manage complex workloads by handling request routing, scheduling, and memory management across multiple GPU nodes to improve performance.

AI agents often reuse the same conversation history across hundreds of API calls. Dynamo optimizes this by using KV-aware routing, which sends requests to the specific worker that already has the relevant context in its memory. This reduces redundant recomputations, leading to lower latency and up to seven times higher throughput for agentic sessions.

Agent hints are a new API extension that allows agent frameworks to pass metadata to the inference stack. These hints include task priority, estimated output length, and speculative prefill signals. By sharing this context, the orchestrator can make better decisions about which requests to prioritize and when to warm up the cache before a tool call returns.

Dynamo manages the Key-Value cache across four storage layers: GPU high-bandwidth memory, CPU pinned DRAM, local NVMe storage, and remote cluster-wide storage. This hierarchy allows the system to offload and share cached context between different workers. It ensures that high-value data, like system prompts, is retained while ephemeral data, such as reasoning tokens, is evicted.

Yes, NVIDIA Dynamo is an open-source project. Developers can access the source code, documentation, and the new agent hints API through the official GitHub repository. It is designed to work with various open-source models and inference engines, providing a standardized infrastructure layer for teams running their own GPU-backed agentic workflows.

Share this update