Agents | NVIDIA Dynamo Documentation

This page is for version dev. For other versions, use one of these documentation indexes: - Latest (v1.6.1) (default): https://docs.nvidia.com/dynamo/latest/llms.txt - dev: https://docs.nvidia.com/dynamo/dev/llms.txt - v1.6.1: https://docs.nvidia.com/dynamo/v1.6.1/llms.txt - v1.6.0: https://docs.nvidia.com/dynamo/v1.6.0/llms.txt - v1.5.1: https://docs.nvidia.com/dynamo/v1.5.1/llms.txt - v1.5.0: https://docs.nvidia.com/dynamo/v1.5.0/llms.txt - v1.4.2: https://docs.nvidia.com/dynamo/v1.4.2/llms.txt - v1.4.1: https://docs.nvidia.com/dynamo/v1.4.1/llms.txt - v1.4.0: https://docs.nvidia.com/dynamo/v1.4.0/llms.txt - v1.3.0: https://docs.nvidia.com/dynamo/v1.3.0/llms.txt - v1.2.1: https://docs.nvidia.com/dynamo/v1.2.1/llms.txt - v1.2.0: https://docs.nvidia.com/dynamo/v1.2.0/llms.txt - v1.1.1: https://docs.nvidia.com/dynamo/v1.1.1/llms.txt - v1.1.0: https://docs.nvidia.com/dynamo/v1.1.0/llms.txt - v1.0.2: https://docs.nvidia.com/dynamo/v1.0.2/llms.txt - v1.0.1: https://docs.nvidia.com/dynamo/v1.0.1/llms.txt - v1.0.0: https://docs.nvidia.com/dynamo/v1.0.0/llms.txt - v0.9.1: https://docs.nvidia.com/dynamo/v-0-9-1/llms.txt - v0.9.0: https://docs.nvidia.com/dynamo/v-0-9-0/llms.txt - v0.8.1: https://docs.nvidia.com/dynamo/v-0-8-1/llms.txt - v0.8.0: https://docs.nvidia.com/dynamo/v-0-8-0/llms.txt - v0.7.1: https://docs.nvidia.com/dynamo/v-0-7-1/llms.txt - v0.7.0: https://docs.nvidia.com/dynamo/v-0-7-0/llms.txt

Agent traffic looks different from chat. Each turn repeats a growing prefix, tool calls leave the GPU idle between turns, and some requests (the one a user is waiting on) matter more than others (a background subagent). Dynamo does not run your agent loop. Your harness still owns prompts, tools, subagents, and reasoning state. Dynamo reads a little metadata on each request and uses it to route and schedule better, keep reusable KV cache, and record what happened.

What You Can Do

I want to…	Use	Guide
Point a coding agent at Dynamo	A local endpoint and a harness config	Run an agent on Dynamo
Tell Dynamo which requests belong to which agent	Session IDs	Identify agent sessions
Keep a session’s turns on the worker that holds its KV cache	Session affinity	Maintain session affinity
Let interactive requests go ahead of background work	Priority hints	Prioritize requests
See where time goes between LLM calls and tools	Request tracing	Trace an Agent Run
Compare serving configurations without rerunning the agent	Trace replay	Simulate an Agent Run
Free KV cache held by agents that are waiting on tools	ThunderAgent	Schedule Agents with ThunderAgent

Try It

Send a request that carries a session ID and a priority hint:

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -H 'x-dynamo-session-id: research-run-42:researcher' \
  -d '{
    "model": "my-model",
    "messages": [{"role": "user", "content": "..."}],
    "nvext": {
      "agent_hints": { "priority": 5, "osl": 1024 }
    }
  }'

The response is a normal chat completion. The session ID labels the request for tracing. The hints change queueing, scheduling, and cache eviction only under load, and only for the priority layers you turn on. With the launcher defaults they are metadata. See Prioritize requests for the switches. Neither one pins the session to a worker; for that, see Maintain session affinity.

How Dynamo Uses the Metadata

The agent harness sends each LLM request with a session ID and optional hints. The Dynamo frontend normalizes the metadata, the router uses it for placement, session binding, and queue order, and the backend engine uses priority for scheduling and eviction. The frontend also records a request trace.

Layer	Signal	What Dynamo does with it
Frontend API	Session headers and `nvext` request extensions	Normalizes agent identity and serving intent across the Chat Completions, Responses, and Messages APIs.
Router	Session ID, priority, expected output length, cache overlap	Places requests for KV reuse, binds sessions to workers when affinity is on, and orders queued work.
Backend engine	Priority forwarded by the router	Influences engine scheduling and cache eviction where the backend supports it.

Where to Start

Follow Run an agent on Dynamo to get an agent talking to a local deployment.
Add session IDs so every request is attributable.
Turn on session affinity if your agents run many turns on the same context.
Add priority once interactive and background requests compete for the same workers.
Use tracing to measure each change before you keep it.

For SGLang engine flags and subagent KV isolation, see Agents on SGLang.