docs(agents): document agent trace golden path by ishandhanani · Pull Request #10869 · ai-dynamo/dynamo

ishandhanani · 2026-06-23T14:55:15Z

Documents an end-to-end path from a live Dynamo agent workload to offline DynoSim and live AIPerf replay. The guide makes request-DAG construction, fixed-schedule gating, cache-hash scope, and replay fidelity limits explicit.

CLOSES: DYN-3248

How This Was Implemented

Added one capture-to-replay guide for DYN_REQUEST_TRACE, DynoSim, and AIPerf.
Documented AIPerf session headers, global Dynamo hash scope, trace block size, and fixed-schedule commands.
Explained sequence, spawn, join, and tool-delay graph construction for both replay paths.
Added preservation, drift, validation, and trace-sharing guidance.
Kept format schemas and constraints in a dedicated trace-format reference.

Walkthrough

flowchart LR
    E["Live Dynamo endpoint"] --> T["Dynamo request trace"]
    T --> D["DynoSim offline replay"]
    T --> W["AIPerf Weka conversion"]
    W --> L["Live Dynamo replay"]

Validation

pre-commit run --files docs/agents/agent-replay.md docs/dynosim/trace-formats.md docs/dynosim/runs.md
fern check (0 errors)
fern docs broken-links
Dynamo Agentic Mooncake smoke replay: 17/17 requests completed.
AIPerf global-hash live replay: 149/149 requests completed with zero errors.

Benchmark Results

75 sessions / 149 requests; 625,769 replay output tokens exactly matched the capture.
Replay input was 882,452 versus 881,659 source tokens (+0.090%).
Global hash scope improved actual cache reuse from 18.22% to 25.79%; theoretical reuse was 26.74%.
Replay wall time was 3,917.54 s versus a 3,887.31 s recorded schedule span (+30.24 s).

Summary

Adds one entry point for capturing and replaying agent workloads with offline DynoSim or live AIPerf. This keeps capture, format, and execution guidance separate while making the end-to-end path discoverable.

CLOSES: DYN-3248

make Agent Simulation the capture-to-replay tutorial
add a trace-format matrix and reference for Mooncake, Agentic Mooncake, Mooncake Delta, Applied Compute TRIE, audit records, and coding-agent exports
remove duplicated Agentic Mooncake detail from DynoSim Runs and replace the stale agent_trace_to_mooncake command
document current live AIPerf terminal-marker and version requirements

Dependency

Dynamo #10214 is merged and provides trajectory-derived backend session lifecycle.
The live replay commands require the AIPerf AgentX fork (AIPerf #3), or another AIPerf build with the same AgentX support.

Walkthrough

docs/agents/agent-replay.md: choose offline or live replay, capture a request trace, convert it, run a smoke test, and understand the inferred request DAG.
docs/dynosim/trace-formats.md: compare capture sources, consumers, schemas, block-size handling, and replay constraints.
DynoSim and Agents indexes now route readers to the canonical pages.

Validation

Targeted pre-commit hooks passed for all changed docs files.
fern check: 0 errors.
fern docs broken-links: passed.
Dynamo request trace conversion produced 17 Agentic Mooncake rows; offline replay completed 17/17 requests.
The Applied Compute example completed 3/3 replay requests.
AIPerf Dynamo-trace converter tests passed: 3/3.

Summary by CodeRabbit

Documentation
- Restructured agent replay documentation with clearer workflow guidance for capturing workloads and offline/live replay paths.
- Added comprehensive trace formats reference guide documenting supported formats and row schemas.
- Improved navigation with new entry points for agent simulation and trace schema selection.
- Refined agent simulation overview and consolidated agentic mooncake details into dedicated references.

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

github-actions · 2026-06-23T14:58:24Z

🌿 Fern Docs Preview: https://nvidia-preview-0bfc0def-144a-40a3-a596-03d0137dae1e.docs.buildwithfern.com/dynamo/dev

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

devin-ai-integration

Devin Review found 1 potential issue.

devin-ai-integration · 2026-06-23T15:46:46Z

+  jq -c '
+    select(.event.event_type == "request_end")
+    | {request_id: .event.request.request_id, agent_context: .event.agent_context}
+  ' | \
+  head


🟡 Incorrect jq field paths in example command - .event. prefix doesn't match flat JSONL record schema

The jq command in docs/agents/agent-replay.md uses .event.event_type, .event.request.request_id, and .event.agent_context, implying the JSON records have an event wrapper object. However, the actual RequestTraceRecord serialization format (confirmed by the test data in lib/bench/src/request_trace/load.rs:446) is flat: fields like event_type, agent_context, and request are top-level keys. The correct jq paths should be .event_type, .request.request_id, and .agent_context. As written, the select filter will never match anything because .event will be null, so the command silently produces no output.

Test data showing flat record format (load.rs:446)

{"schema":"dynamo.request.trace.v1","event_type":"request_end","event_time_unix_ms":1100,"event_source":"dynamo","agent_context":{"trajectory_id":"root"},"request":{"request_id":"req-1",...}}

Was this helpful? React with 👍 or 👎 to provide feedback.

coderabbitai · 2026-06-23T15:47:37Z

Walkthrough

Documentation restructuring: the agent-replay guide is substantially rewritten to cover both DynoSim offline and AIPerf live-endpoint replay paths. A new trace-formats.md reference document is added covering all --trace-format values with full JSONL schemas. Navigation entries, cross-links, and the condensed runs.md agentic section are updated to point to the new reference.

Changes

Agent Replay and Trace Format Documentation Restructure

Layer / File(s)	Summary
New trace-formats reference document `docs/dynosim/trace-formats.md`	New file documenting all supported `--trace-format` values (`mooncake`, `mooncake-delta`, `agentic_mooncake`, `applied_compute_agentic`) with JSONL row schemas, scheduling constraints, `cargo run` conversion commands, and reusable building blocks (`MooncakeRow`, `MooncakeJsonlWriter`, rolling-hash helpers).
Agent replay guide rewrite `docs/agents/agent-replay.md`	Page subtitle and intro updated; adds a "Choose a Replay Path" section with a goal-to-path table and mermaid diagram. Revises capture procedure with trajectory identity fields and tool event guidance. Replaces prior offline replay instructions with a DynoSim two-run (smoke + `kv_router`) flow. Adds an AIPerf live-endpoint section covering prerequisites, JSONL merge, `aiperf synthesize dynamo-trace`, Weka output, and runtime flag behavior. Replaces "Agentic Row Semantics" with "How the Request DAG Is Built."
Navigation and cross-link updates `docs/agents/README.md`, `docs/dynosim/README.md`, `docs/dynosim/runs.md`, `docs/index.yml`	`agents/README.md` description for Agent Simulation updated to reference DynoSim/AIPerf replay. `dynosim/README.md` gains two entry-point rows for agent DAG simulation and trace schema selection. `dynosim/runs.md` agentic_mooncake section condensed to cross-links. `docs/index.yml` adds the Trace Formats page under DynoSim navigation.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly reflects the main change: documenting the agent trace capture-to-replay path.
Description check	✅ Passed	The PR description covers overview, implementation details, reviewer guidance, validation, and links the work to DYN-3248.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

_{Comment @coderabbitai help to get the list of available commands.}

…ntic-simulation-docs Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com> # Conflicts: # docs/agents/agent-replay.md # docs/dynosim/runs.md

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

devin-ai-integration

Devin Review found 2 new potential issues.

devin-ai-integration · 2026-06-24T22:42:44Z

🚩 Pre-existing ../../ escape link in unchanged runs.md line 69

The link at docs/dynosim/runs.md:69 — [\lib/mocker/src/replay/offline/README.md`](../../lib/mocker/src/replay/offline/README.md)— uses a relative../../path that escapesdocs/. Per docs/AGENTS.mdand the documentation style guide, targets outsidedocs/` should use absolute GitHub URLs. This is pre-existing and not part of the PR diff, but it's in the same file that this PR edits.

(Refers to line 69)

Was this helpful? React with 👍 or 👎 to provide feedback.

devin-ai-integration · 2026-06-24T22:42:45Z

🚩 Pre-existing SPDX header uses single year 2026 instead of range 2025-2026 in runs.md

The SPDX header at docs/dynosim/runs.md:2 uses Copyright (c) 2026 rather than the required range Copyright (c) 2025-2026. The documentation style guide mandates the 2025-2026 range. This is pre-existing (not touched by this PR) but is in the same file that was modified.

(Refers to line 2)

Was this helpful? React with 👍 or 👎 to provide feedback.

ishandhanani · 2026-06-25T00:11:30Z

Closing in favor of a smaller replacement PR focused only on the replay guide.

docs(agents): add agent simulation replay guide

619d474

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

pull-request-size Bot added the size/L label Jun 23, 2026

github-actions Bot added docs documentation Improvements or additions to documentation labels Jun 23, 2026

docs(agents): remove final trajectory marker requirement

3fb0007

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

ishandhanani commented Jun 23, 2026

View reviewed changes

Comment thread docs/dynosim/trace-formats.md Resolved

Comment thread docs/dynosim/trace-formats.md Resolved

docs(dynosim): simplify trace format entry point

9885358

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

ishandhanani marked this pull request as ready for review June 23, 2026 15:42

ishandhanani requested a review from a team as a code owner June 23, 2026 15:42

devin-ai-integration Bot reviewed Jun 23, 2026

View reviewed changes

ishandhanani changed the title ~~docs(agents): add agent simulation replay guide~~ docs(agents): document agent trace golden path Jun 24, 2026

ishandhanani added 2 commits June 24, 2026 22:37

Merge remote-tracking branch 'origin/main' into idhanani/dyn-3248-age…

067f73c

…ntic-simulation-docs Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com> # Conflicts: # docs/agents/agent-replay.md # docs/dynosim/runs.md

docs(agents): document live trace replay golden path

f6de294

Signed-off-by: Ishan Dhanani <ishandhanani@gmail.com>

ishandhanani temporarily deployed to external_collaborator June 24, 2026 22:39 — with GitHub Actions Inactive

copy-pr-bot Bot temporarily deployed to GITLAB June 24, 2026 22:39 Inactive

devin-ai-integration Bot reviewed Jun 24, 2026

View reviewed changes

copy-pr-bot Bot temporarily deployed to GITLAB June 24, 2026 22:55 Inactive

ishandhanani closed this Jun 25, 2026

Navigation Menu

Repository navigation

docs(agents): document agent trace golden path - #10869

docs(agents): document agent trace golden path#10869
ishandhanani wants to merge 5 commits into
mainfrom
idhanani/dyn-3248-agentic-simulation-docs

ishandhanani commented Jun 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Jun 23, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jun 23, 2026 •

edited

Loading

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Jun 24, 2026

Uh oh!

devin-ai-integration Bot Jun 24, 2026

Uh oh!

ishandhanani commented Jun 25, 2026

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ishandhanani commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How This Was Implemented

Validation

Benchmark Results

Summary

Dependency

Validation

Summary by CodeRabbit

Uh oh!

github-actions Bot commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Jun 24, 2026

Choose a reason for hiding this comment

Uh oh!

ishandhanani commented Jun 25, 2026

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ishandhanani commented Jun 23, 2026 •

edited

Loading

github-actions Bot commented Jun 23, 2026 •

edited

Loading

devin-ai-integration Bot Jun 23, 2026 •

edited

Loading

coderabbitai Bot commented Jun 23, 2026 •

edited

Loading