Repository navigation
perf(agentx): 50k runs a 600s window; memory 818.6 -> 219-295 GiB, warmup 36.5 -> ~6,000 req/s - #8
ajcasagrande wants to merge 6 commits into
Conversation
Try out this PRQuick install: pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@305a9963c79aac6cafc3b15038a56d81e370038eRecommended with virtual environment (using uv): uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@305a9963c79aac6cafc3b15038a56d81e370038eLast updated for commit: |
Three scans ran once per credit at 50,000 lanes: - warmup_credit_count rebuilt an O(lanes) tuple on every warmup credit return (53,740 returns x 50,000 lanes), pinning the single timing-manager thread at a full core while the pod used ~1.2 of 192 cores. Memoize the per-lane primer counts and their total, invalidated whenever trajectories are replaced. - The pending-timer idle-cap pass advanced one replay root's timers but scanned every pending timer in the run. At trace_idle_gap_cap_seconds=300 all ~50k roots fire in one batch, making it O(roots x timers) in one thread; the run hung permanently once it crossed 300 s. Index pending handles per group. - The join map answered "does this parent still have tracked children?" by walking every entry of every child's list, once per credit. Count per parent. - The cache-warmup admission check scanned every lane per admitted warmup turn (500k admits at 50k lanes). Count saturated lanes instead. Also makes two per-lane scans O(1) in close_replay_root and the cache-warmup admission path. Measured at 50k lanes: warmup 36.5 req/s -> ~6,000 req/s average (~9,000 req/s mid-phase), and the 600 s window completes instead of hanging at 300 s. Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com> (cherry picked from commit cdd5fda)
AgentX trajectories re-send the same ~230 KB first user message after each context reset, up to 26 times per conversation, and a few non-ASCII characters force Python's 2-bytes-per-character storage: 522 MiB of Python strings for 261 MiB of UTF-8. Collapse repeated strings of 256+ characters at decode time so every occurrence points at one copy. Also serialize the dataset-configured notification once per publish and splice the cached bytes into status replies, instead of re-dumping the whole multi-MB metadata through pydantic on every worker poll (193.5 ms -> 0.65 ms per reply). Measured at 50k lanes: removing the collapse costs ~140 GiB. Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com> (cherry picked from commit a00d33d)
One record per request carries ~200 streamed chunks, so ~1.9M chunks/s paid pydantic model_dump + orjson on send and orjson + union validation on receive. Encode with a msgspec encoder and decode with a typed decoder: 4,474 -> 626 ns per chunk. Client -> server HTTP bodies are unchanged; this is the internal channel only. Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com> (cherry picked from commit 489015b)
…ache-bust marker at wire build Two changes to the per-request path at 50,000 lanes over 9,843 conversations: Decode each conversation once per worker and give each session a shallow clone that copies only the containers mutators write to, keyed on the conversation id. The key is the id alone; the PAYLOAD_BYTES reconstruction loops range(num_turns), so a cached conversation is exactly as long as the count it was built with. That is asserted on the hit path instead of encoded in the key: a credit declaring a different count is served an uncached reconstruction and the disagreement is reported once, so one conversation occupies one entry and a dataset that varies the count is visible rather than silently multiplied. Apply the cache-bust marker when the endpoint builds the wire messages rather than by mutating the session's Turn. The old path stored marker + content - a full copy of the message, ~230 KB for an AgentX first turn - for the life of the session, and a trace re-sends the same first-user message after each context reset, so a lane retained many copies. Wire bytes are unchanged: same marker, same message, same position. SYSTEM_* targets still mark the per-request system message, which was already transient. Measured at 50k lanes / 600 s: 311-319 GiB -> 219 GiB at window end. Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com> (cherry picked from commit 4b2b463)
Least-loaded placement sends every new session to whichever worker is idlest, so with L lanes drawn from C conversations each worker decodes nearly all C conversations. Workers are separate processes, so the decoded set is duplicated per worker: at 50k lanes each of 100 workers held ~3.7 GiB of private heap. Assign a conversation to the worker carrying the least accumulated weight, weighted by its turn count, and remember the assignment, so a worker's working set is C/workers instead of C. Weighting rather than hashing matters because conversations differ enormously in size: hashing balanced the count and left a worker holding heavy conversations as the bottleneck, which measured a ~0.9% throughput cost that weighting recovers. Multi-turn sessions are unchanged: the first turn still creates the sticky entry that binds later turns. DAG children stay least-loaded - pinning them to their tree's worker was measured worse (238 vs 219 GiB), because grouping a tree lowers the number of decodes but raises that worker's peak, and peak is what binds. Measured at 50k lanes / 600 s: 487 GiB -> ~219-295 GiB across repeats, warmup 94 s vs 103 s, throughput at parity. Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com> (cherry picked from commit dc94d28)
…CONVERSATION_CACHE_MAX_ENTRIES The bound was a module constant. Expose it so it can be tuned without a code change, and record what was measured: at 50,000 lanes over 600 s, bound 1 -> 388 GiB, 16 -> 219 GiB, 128 -> 272 GiB, 393 -> 281 GiB, throughput flat. The deltas are NOT resolved - run-to-run variance on that setup is ~25% on memory, larger than the gaps between neighbouring settings - so the description says to treat 16 as the untuned default rather than a measured optimum. Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
5c7c6e2 to
305a996
Compare
FrankD412
left a comment
There was a problem hiding this comment.
A bot checked the memo invalidation raised on #7: the stale-value probe from that thread now passes, and the new regression test passes (measured). The new per-group, per-parent and saturated-lane counters, and the per-session conversation clone, look consistent from reading. The loop_scheduler, branch_orchestrator and worker cache-bust suites pass locally (measured).
One gap in "wire bytes are identical" for d849b93, inline below.
| CacheBustTarget.FIRST_TURN_PREFIX, | ||
| CacheBustTarget.WARMUP_ISOLATION_FIRST_TURN, | ||
| ) | ||
| for index, message in enumerate(messages): |
There was a problem hiding this comment.
A bot compared request bodies built at 79994bb (before this commit) and at this head, for chat, responses and messages × FIRST_TURN_PREFIX / FIRST_TURN_SUFFIX / WARMUP_ISOLATION_FIRST_TURN / SYSTEM_PREFIX × 12 conversation shapes: 114 of 144 byte-identical, 30 differ (measured). All raw_messages shapes match, including deltas, mid-trajectory resume and a buried reset, as do single-text synthetic turns and every SYSTEM_* case (measured). The differences fall into two groups:
1. user_context_message (chat, messages). The preamble is prepended as a user message before this function runs, so the first-user scan now marks the preamble instead of the trace's first user message (measured):
before: [user "CTX"] [system "sys"] [user "[rid:…]\n\nhello…"]
after: [user "[rid:…]\n\nCTX"] [system "sys"] [user "hello…"]
If the preamble is shared across conversations, the trace's own first message is no longer unique per lane, which may defeat the cache-bust on those datasets (reading; not tested against a server). Responses is unaffected because the marker is applied before its context item is added (reading, measured).
2. Synthetic multi-part turns (all three). For a turn with two texts, or text plus an image, the old path spliced the marker into the first text; this path appends a separate part with the marker stripped of its \n\n (measured):
before: [{"type":"text","text":"[rid:…]\n\nlook…"}, {"type":"image_url",…}]
after: [{"type":"text","text":"[rid:…]"}, {"type":"text","text":"look…"}, {"type":"image_url",…}]
On responses the added part is "type": "text" next to "input_text" parts (measured); whether servers accept that mix was not checked.
These were hand-built shapes, not AgentX traces. AgentX's raw_messages paths were all identical, so the 50k numbers are likely unaffected (inference, not measured). Suggest either scoping "identical" in the commit/PR text, or applying the marker after skipping a leading user_context_message and, for list content, splicing into the first text part to match the old path; a before/after byte test over these shapes would pin it.
What this does
Make the AgentX 50,000-concurrency scenario actually runnable at a 600 s profiling
window, then cut its memory roughly 3x and its warmup time ~30x.
Like-for-like against master, both cold-started with warmup's wire phase skipped,
240 s window, identical socket settings:
240 s rather than 600 s because master hangs at exactly 300 s into profiling
without the per-group timer index (commit 1).
Why each change
Remove quadratic per-credit scans from the credit path. Three scans ran once
per credit at 50k lanes:
warmup_credit_countrebuilt an O(lanes) tuple on everywarmup return (53,740 x 50,000), pinning the timing-manager thread at a full core
while the pod used 1.2 of 192; the idle-cap pass scanned every pending timer per
replay root, and at
trace_idle_gap_cap_seconds=300all ~50k roots fire at once,which hung the run permanently past 300 s; and the join map walked every child's
list to answer "does this parent still have children?". All three are indexed or
memoized.
Stop duplicating repeated strings. AgentX re-sends the same ~230 KB first
user message after each context reset, up to 26 times per conversation, and a few
non-ASCII characters force 2-bytes-per-character storage. Collapsing repeated
256+ character strings at decode gives 522 MiB of strings for 261 MiB of UTF-8.
Typed msgpack on the worker -> record-processor channel. ~1.9M chunks/s
passed through pydantic
model_dump+ orjson on send and orjson + unionvalidation on receive: 4,474 -> 626 ns per chunk. Client -> server HTTP bytes are
unchanged.
Decode each conversation once, and apply the cache-bust marker at wire build.
The marker used to be written into the session's
Turnasmarker + content— afull copy of the message, retained for the session's life — and a trace re-sends
that same message after each reset, so a lane held many copies. Applying it when
the endpoint builds the wire keeps the session on the original shared content.
Wire bytes are identical. Separately, the conversation cache is keyed on
conversation id with the turn count asserted on the hit path.
Place a root's first turn on its conversation's worker. Least-loaded
placement made every worker decode nearly the whole corpus; workers are separate
processes, so at 50k lanes each held ~3.7 GiB of private heap. Assignment is
weighted by turn count rather than hashed, because hashing balanced the count and
left workers holding heavy conversations as bottlenecks.
Measurement caveat
Run-to-run variance on this setup is ~25% for memory (two runs of identical
code measured 237 and 295 GiB at window end) and ~1% for throughput.
Differences below ~60 GiB or ~2% in this branch's notes are therefore not
resolved. The headline numbers above are far outside that band; finer deltas
recorded in commit messages (for example the cache-bound sweep) are not, and are
labelled accordingly there.
Testing
tests/unit/{credit,workers,endpoints,dataset,zmq,transports}: 5,493 passed, 9 skipped, 2 failed. Both failures are environmental, not code:test_socket_defaultsreads the process environment, and this host setsAIPERF_HTTP_SO_SNDBUF=0globally; passes with those unset.PermissionError, which cannot occur whenthe suite runs as root.