Skip to content

fix(agentx): preserve trace timing and joins - #31

Open
cquil11 wants to merge 7 commits into
mainfrom
cquil11/allow-agentx-trace-idle-cap
Open

fix(agentx): preserve trace timing and joins#31
cquil11 wants to merge 7 commits into
mainfrom
cquil11/allow-agentx-trace-idle-cap

Conversation

@cquil11

@cquil11 cquil11 commented Jul 30, 2026

Copy link
Copy Markdown

Summary

  • enforce --trace-idle-gap-cap-seconds only as an observed runtime idle watchdog per complete trajectory tree
  • preserve source dataset timestamps and end-to-start replay delays without load-time or phase-start rewriting
  • include the root, subagents, and flattened-agent streams under one runtime-root watchdog
  • preserve deferred child and join state across accelerated warmup handoff
  • release profiling joins only after both the recorded replay deadline and all required child completions
  • retain the separate whole-system idle guard without bypassing replay barriers or spawn/join dependencies

Runtime semantics

The trace watchdog starts for initial future profiling work and restarts whenever the final in-flight request across a root and all descendants completes. If no request from that tree reaches the wire before the configured cap, AIPerf uniformly advances only that tree’s pending scheduler timers. Other roots are untouched, relative timer spacing is preserved, and dependency gates remain authoritative.

The dataset loader no longer interprets this flag. Recorded timestamps, api_time, and derived end-to-start delays remain unchanged. Actual server completion behavior determines when the runtime tree becomes idle.

Validation

  • warmup handoff tested with 1, 10, and 100 additional requests per lane
  • exact root/subagent ordering and SPAWN_JOIN completion gates verified through the HTTP mock server
  • issue agentic_replay drops post-child delay on parent resume ai-dynamo/aiperf#1231 root-child-root timing shape verified with and without the separate global idle guard
  • independent sibling-stream drift verified against the runtime whole-tree idle cap
  • initial profiling idle verified with real grouped scheduler timers
  • 1,807 focused loader, scheduler, replay, and regression tests passed
  • 7 focused end-to-end HTTP mock-server tests passed
  • all pre-commit hooks passed

@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

Try out this PR

Quick install:

pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@ed057829b78d25d79ce6f3b87763d48fe50363f5

Recommended with virtual environment (using uv):

uv venv --python 3.12 && source .venv/bin/activate
uv pip install --upgrade --force-reinstall git+https://github.com/ai-dynamo/aiperf.git@ed057829b78d25d79ce6f3b87763d48fe50363f5

Last updated for commit: ed05782Browse code

@github-actions

github-actions Bot commented Jul 30, 2026

Copy link
Copy Markdown

Fern Docs Preview: generation failed — see the Actions log for details. This does not block merge; ask a maintainer to retry if needed.

@cquil11
cquil11 marked this pull request as ready for review July 30, 2026 16:49
@cquil11 cquil11 changed the title fix(agentx): allow configurable per-trace idle-gap cap fix(agentx): allow configurable trace idle-gap cap Jul 30, 2026
@cquil11 cquil11 changed the title fix(agentx): allow configurable trace idle-gap cap fix(agentx): preserve trace timing and joins Jul 31, 2026
cquil11 added 4 commits July 31, 2026 11:47
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
@cquil11
cquil11 force-pushed the cquil11/allow-agentx-trace-idle-cap branch from 1c315af to deb5421 Compare July 31, 2026 16:47
ajcasagrande and others added 3 commits July 31, 2026 14:07
…ynamo#1228)

When cache-bust owns trajectory prefix isolation, preserve the same prefix across warmup and profiling so warmup primes the cache used by profiling.

Upstream: 9b5f5f7

中文:启用 cache-bust 时,由其负责轨迹前缀隔离,并在预热与性能测试阶段保持相同前缀,使预热能够填充性能测试实际复用的缓存。

Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Keep the default profiling grace period unless the user explicitly overrides it.

Upstream: 5ad0816

中文:除非用户显式覆盖,否则保留默认的性能测试宽限期。

Signed-off-by: Anthony Casagrande <acasagrande@nvidia.com>
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Carry each stream's full recorded delay across the warmup barrier, then subtract one phase-wide minimum so the earliest profiling request starts immediately without changing relative timing, order, or join gates.

中文:在 warmup 屏障后保留每个流完整的记录延迟,再统一减去 profiling 阶段的全局最小值,使最早请求立即启动,同时保持相对时序、请求顺序和 join 门控不变。

Signed-off-by: Cam Quilici <cjquilici@gmail.com>

@ajcasagrande ajcasagrande left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants