perf(db): accelerate parallel trace ingestion / 加速并行 trace 摄取

[cquil11]

Summary

• replace the dual stream-json token fan-out with one packed-token consumer that materializes only selected metrics
• split trace preparation from persistence and execute CPU-heavy preparation in a bounded worker pool derived from os.availableParallelism()
• use four workers by default on the 16-vCPU ingest runner, with a conservative four-worker default cap and an INGEST_TRACE_WORKERS override
• cap database uploads at two concurrent transactions and retain each worker slot until its prepared payload has been consumed, bounding peak buffered data
• recheck and lock unlinked benchmark rows inside the persistence transaction so concurrent retries cannot create orphaned trace rows
• preserve the existing sequential insertTraceReplay() API for backfills and other callers

This follows merged PR #672 and addresses the parser slowdown observed while staging InferenceX run 30781275770.

Functional-safety validation

• existing byte-for-byte aggregate/chart/timeline equivalence tests remain in place
• new parser tests cover both phases, nested lookalikes, scalar and compound values, duplicate keys, and malformed input
• new persistence tests verify all six payloads are uploaded and that a concurrent-link race exits before inserting a blob
• new worker tests verify vCPU scaling, upload concurrency limits, bounded completed buffers, exact payload transfer, and recovery after a failed job
• exact 2.6 GiB production artifact benchmark: 28.1s with the new parser vs 209.1s with the old fan-out locally (7.4x faster), with identical selected-map SHA-256 digests
• exact production artifact worker smoke: direct preparation and all three worker preparations produced identical hashes for compressed raw blobs, CSV, aggregate stats, chart series, request timeline, counts, and cache-hit rates
• formatting, lint, TypeScript, and targeted tests pass
• full unit suite passes: 202 files and 3,586 tests; Cypress component suite passes: 27 specs and 195 tests
• local Cypress integration run passes 44/45 specs and 577/578 tests; the sole failure is an untouched overview-page pixel-geometry assertion (3332.5px vs 3334px ± 1px), with the authoritative sharded CI run triggered when this PR leaves draft

中文说明

• 将双路 stream-json token 分发改为单一 packed-token 消费器，只实例化实际需要的指标
• 将 trace 预处理与持久化拆分，并根据 os.availableParallelism() 使用有界 worker 池执行 CPU 密集型预处理
• 当前 16 vCPU 摄取 runner 默认使用 4 个 worker；默认上限为 4，也可通过 INGEST_TRACE_WORKERS 覆盖
• 数据库上传最多并发 2 个事务；每个 worker 槽位会保留到对应产物消费完成，从而限制内存中待上传数据的峰值
• 在持久化事务中重新检查并锁定尚未关联的基准测试行，避免并发重试生成孤立 trace 记录
• 保留现有顺序执行的 insertTraceReplay() API，兼容 backfill 和其他调用方

这是已合并 PR #672 的后续优化，用于解决暂存 InferenceX 运行 30781275770 时观察到的解析性能下降。

功能安全验证

• 保留 aggregate/chart/timeline 的逐字节等价测试
• 新增解析器测试，覆盖两个阶段、嵌套同名字段、标量与复合值、重复键及异常输入
• 新增持久化测试，确认六类 payload 全部上传，并验证并发关联竞态会在插入 blob 前安全退出
• 新增 worker 测试，覆盖 vCPU 伸缩、上传并发限制、完成产物的内存上限、精确数据传输及失败后的恢复能力
• 使用真实 2.6 GiB 生产产物测试：新解析器本地耗时 28.1 秒，旧分发实现耗时 209.1 秒，提速 7.4 倍；所选指标映射的 SHA-256 完全一致
• 使用真实生产产物进行 worker 冒烟测试：直接预处理与三次 worker 预处理在压缩原始 blob、CSV、aggregate stats、chart series、request timeline、计数和缓存命中率上的哈希完全一致
• 格式、lint、TypeScript 和定向测试均已通过
• 完整 unit 测试通过：202 个文件、3,586 个测试；Cypress component 测试通过：27 个 spec、195 个测试
• 本地 Cypress integration 测试通过 45 个 spec 中的 44 个、578 个测试中的 577 个；唯一失败项是本 PR 未修改的 overview 页面像素几何断言（3332.5px，预期 3334px ± 1px），PR 退出 draft 后将以分片 CI 结果为准

---

Note

Medium Risk
Touches ETL persistence (row locking, concurrent trace linking) and large-document JSON parsing; behavior is heavily tested but ingest correctness and memory bounds matter in production.

Overview
Speeds up CI trace ingestion for large agentic runs by replacing the dual stream-json token tee with a single packed-token pass that only assembles selected metrics / warmup_metrics keys (fewer tokens on multi‑GiB blobs), plus new parser tests for nested lookalikes, duplicate keys, and edge cases.

Splits trace work into prepare vs persist: prepareTraceReplay does gzip/derivation (usable from worker threads); persistPreparedTraceReplay uploads chunks and links rows. Persistence now SELECT … FOR UPDATE and rechecks trace_replay_id inside the transaction so concurrent ingests cannot insert orphan blobs. Cache-hit metrics are computed during prepare and applied in the same transaction.

CI path (ingest-ci-run) queues trace jobs on a TraceReplayWorkerPool sized from vCPUs / INGEST_TRACE_WORKERS, holds each worker slot until the upload callback finishes (bounded memory), and limits DB uploads with an AsyncSemaphore (max 2 concurrent). Sequential insertTraceReplay() remains for other callers.

^{Reviewed by Cursor Bugbot for commit f4e4518. Bugbot is set up for automated code reviews on this repo. Configure here.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
inferencemax-app	Ready	Preview	Aug 5, 2026 3:19pm