Optimize agentic trace ingestion without changing payloads

[cquil11]

What changed

• Index complete agentic_<suffix>/conc_<N>/aiperf_artifacts uploads before considering their duplicate multinode_server_logs tarballs.
• Skip nested extraction only for an exact per-concurrency match with all three trace inputs present; incomplete and legacy layouts retain the existing archive fallback.
• Parse oversized server-metrics JSON once and share the resulting profiling/warmup metric maps between aggregate-stat and chart-series generation.
• Preserve the existing bounded-vs-streaming metric selection behavior and all malformed-input fallbacks.

Why

Run 30781275770 expanded approximately 92.8 GiB of trace data a second time from nested server-log archives, consuming roughly 56 minutes and causing smaller runners to run out of disk. Each oversized server-metrics document was then independently decompressed and tokenized three times: aggregate metrics, chart profiling metrics, and chart warmup metrics.

The producer copies the same completed logs directory into both artifact families. As an external verification, the direct and nested copies from run 30781275770, concurrency 1, had identical SHA-256 hashes for profile_export.jsonl, server_metrics_export.csv, and server_metrics_export.json.

Functional-safety checks

• Added byte-for-byte JSON equivalence coverage for the uploaded aggregate_stats, chart_series, and request_timeline payloads.
• Exercised both the bounded and forced-streaming paths.
• Verified malformed-input behavior remains isolated by payload.
• Verified incomplete direct artifacts still extract and use the nested archive.
• Raw profile, CSV, and server-metrics compression/upload paths are unchanged.

Validation

• bun run test:unit — 3,574 tests passed across all workspaces
• bun run fmt
• bun run lint
• bun run typecheck
• git diff --check

---

Note

Medium Risk
ETL behavior changes could alter discovery or derived JSON if equivalence assumptions fail, but extensive byte-for-byte tests and preserved fallback paths mitigate regressions on ingest-critical paths.

Overview
Speeds up agentic trace ingest by avoiding duplicate disk expansion and repeated parsing of huge server-metrics JSON, while keeping stored aggregate_stats, chart_series, and request_timeline byte-equivalent to the old path.

Artifact discovery now indexes complete direct uploads under agentic_<suffix>/conc_<N>/ before touching multinode_server_logs_*.tar.gz, and skips tarball extraction when that concurrency already has all three trace files. Incomplete direct layouts still use the archive fallback.

Derived payload computation routes insertTraceReplay() through new computeTraceDerivedPayloads(), which parses server metrics once via collectMetricPhases() (single gunzip + token stream tee’d to metrics and warmup_metrics). Shared helpers withServerMetricAggregateStats() and computeChartSeriesFromMetricPhases() feed aggregate and chart generation from the same parse; bounded vs streaming metric key selection and per-payload error isolation match the previous three independent calls.

Tests assert JSON equivalence (including forced streaming), malformed-input behavior, and discovery skip/fallback rules.

^{Reviewed by Cursor Bugbot for commit 1327c67. Bugbot is set up for automated code reviews on this repo. Configure here.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
inferencemax-app	Ready	Preview	Aug 4, 2026 10:12pm