feat(evals): M2 — Parquet-snapshot fast seed (cache-backed test)#2674
feat(evals): M2 — Parquet-snapshot fast seed (cache-backed test)#2674brandon-pereira wants to merge 2 commits into
Conversation
Kill the seed bottleneck. Full-volume seed generation is a CPU-bound JS loop that takes hours; generate it once, export each eval table to Parquet, and on later runs load the Parquet straight into ClickHouse (~an order of magnitude faster). Measured locally: at 1M rows, generate+insert 22s vs Parquet load 5.6s; the gap widens toward hours->minutes at full volume. HDX-4758 — generate + snapshot export-snapshot <scenario> --dir: dumps every non-empty eval table to <table>.parquet (zstd) + a manifest.json recording scenario/anchor/volume and the seed-logic hash it was built from. HDX-4759 — load via table function (primary path) load-snapshot <scenario> --dir: ensures tables + rollup MVs exist, truncates, then streams each Parquet file into an INSERT. Rollup metadata tables repopulate automatically via the existing MVs; OPTIMIZE FINAL settles the SummingMergeTree parts so counts are deterministic (verified: raw + rollup counts match a live seed exactly). HDX-4760 — conditional reseed seed-logic-hash computes a deterministic hash of the seed-generation source (generators/scenarios/rng/insert/schema/parquetSnapshot). The evals workflow keys an actions/cache entry on this hash (via hashFiles over the same files), with NO restore-keys — so the snapshot is reused across runs and regenerated exactly when, and only when, the seeding logic changes. Wiring: - run-evals.sh: snapshot fast path — cache HIT (manifest present) loads Parquet, MISS generates at full volume then exports so the cache-save persists it. Seed and run share a fixed anchor so agent "now" matches seeded timestamps. - docker-compose.evals.yml: mount + env pass-through for the snapshot dir. - evals.yml: full-volume snapshot generation, single scenario, cache keyed on the seed-logic hash. Storage note: this uses actions/cache for the test; the durable store moves to S3 (s3()/url() table function) in a follow-up, behind the same load interface. Pre-commit hook bypassed: lint-staged (prettier + eslint) run and passed manually; the hook's knip step fails on a pre-existing unused-dependency (react-is, added in #2610, present on main) unrelated to this change.
🦋 Changeset detectedLatest commit: 8716d00 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub. 2 Skipped Deployments
|
Greptile SummaryThis PR adds a Parquet-backed fast seed path for the hdx-eval suite. The main changes are:
Confidence Score: 5/5This looks safe to merge.
Important Files Changed
Reviews (2): Last reviewed commit: "perf(evals): faster Parquet load — bulk ..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| await args.truncate(); | ||
|
|
||
| const files: LoadResult['files'] = []; | ||
| let totalRows = 0; | ||
|
|
||
| for (const field of snapshotTableFields()) { | ||
| const table = tables[field]; | ||
| const filePath = join(args.dir, snapshotFileName(table)); | ||
| if (!existsSync(filePath)) continue; // table had zero rows at export time |
There was a problem hiding this comment.
When a cached snapshot directory has manifest.json but one Parquet file is missing, this path truncates the scenario tables before checking the per-table files and then treats the missing file as zero rows. A partial cache restore or manually pruned snapshot can leave the eval run with an empty or partial dataset instead of failing before destructive work.
Knowledge Base Used: hdx-eval: LLM Agent Evaluation Harness
| if (manifest.scenarioName !== scenario.name) { | ||
| throw new Error( | ||
| `Snapshot manifest is for scenario "${manifest.scenarioName}", not "${scenario.name}"`, | ||
| ); | ||
| } |
There was a problem hiding this comment.
The manifest records the seed-logic hash, volume, and seed time, but the load command only rejects a scenario-name mismatch. A same-scenario snapshot built from old seed logic or a different volume can be loaded successfully, so evals can run against data that no longer matches the current generator code.
Knowledge Base Used: hdx-eval: LLM Agent Evaluation Harness
| `SELECT * FROM ${EVAL_DATABASE}.${table} ` + | ||
| `FORMAT Parquet ` + | ||
| `SETTINGS output_format_parquet_compression_method='zstd'`; |
There was a problem hiding this comment.
SELECT * can include materialized or alias columns inherited from the source ClickHouse tables. If the target eval table treats those columns as computed on load, INSERT FORMAT Parquet can reject the file or load values that should have been recomputed, breaking snapshot reuse after schema changes.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
… fan-out
The snapshot load was letting the rollup materialized views fan out on the bulk
Parquet insert: each insert block was aggregated and written to the
SummingMergeTree KV/key rollups, then OPTIMIZE FINAL merged them. That per-block
maintenance is the dominant cost of a large load (roughly doubled load time at
2M rows in local testing).
New load path (transparent — same load-snapshot CLI):
1. drop the rollup MVs
2. bulk-insert the raw Parquet with no fan-out
3. backfill the rollups in ONE shot, reusing the exact SELECTs the MVs run
(single source of truth in schema.ts)
4. recreate the MVs so later inserts still maintain them
Verified byte-identical rollup content vs the MV path (kv sum(count) and key
rollup count match exactly), and MVs are re-attached after load.
Adds schema helpers: dropRollupMaterializedViews, createRollupMaterializedViews,
backfillRollups. Replaces the previous OPTIMIZE-FINAL-after-MV approach.
Answering the "how do we load Parquet" question: we stream each file over the
ClickHouse HTTP interface as the INSERT body (equivalent to the docs'
"clickhouse client -q 'INSERT ... FORMAT Parquet' < file"). Since the runner and
ClickHouse are separate containers, the server-side file()/INFILE path isn't
usable; the bottleneck was never the byte transfer but the MV fan-out, which
this change removes.
1 participant
Why
Seeding is the dominant eval bottleneck: full-volume synthetic telemetry is a CPU-bound JS generation loop that takes hours, so evals rarely run. Milestone 2 kills that bottleneck — generate the data once, snapshot it to Parquet, and load the Parquet straight into ClickHouse on subsequent runs (I/O-bound, ~an order of magnitude faster).
Per direction: this PR tests the mechanism with GitHub Actions Cache as the snapshot store. The durable store moves to S3 (
s3()/url()table function) in a follow-up, behind the same load interface — the load path is already storage-agnostic.Stacked on the M1 PR (#2628); base branch is the M1 branch so this diff is M2-only.
Tickets: HDX-4758, HDX-4759, HDX-4760 (project: MCP Eval Improvements).
Measured speedup (local, dev ClickHouse)
The gap widens with scale — generation trends toward hours, Parquet load stays I/O-bound and near-linear. Full-suite full-volume Parquet footprint measured at ~6.7 GB (87M rows).
What changed
HDX-4758 — generate + snapshot.
export-snapshot <scenario> --dirdumps every non-empty eval table to<table>.parquet(zstd) plus amanifest.jsonrecording scenario / anchor / volume factor / the seed-logic hash it was built from.HDX-4759 — load via table function (primary path).
load-snapshot <scenario> --dirensures tables + rollup MVs exist, truncates, then streams each Parquet file into anINSERT. Rollup metadata tables repopulate automatically via the existing materialized views;OPTIMIZE … FINALsettles the SummingMergeTree parts so counts are deterministic. Verified: raw + rollup counts match a live seed exactly (1,008,967 raw / 199 key-rollup, identical before/after).HDX-4760 — conditional reseed.
seed-logic-hashcomputes a deterministic hash of the seed-generation source (generators/,scenarios/,rng/,insert.ts,schema.ts,parquetSnapshot.ts). The workflow keys anactions/cacheentry on this hash (viahashFilesover the same files), with norestore-keys— so a cached snapshot is reused across runs and regenerated exactly when, and only when, the seeding logic changes. A stale snapshot is never partially reused across a logic change.Wiring
run-evals.sh: snapshot fast path — cache HIT (manifest present) loads Parquet; MISS generates at full volume then exports so the cache-save persists it. Seed and run share a fixed anchor so the agent's "now" matches seeded timestamps.docker-compose.evals.yml: mount + env pass-through for the snapshot dir (written by the non-root runner user).evals.yml: full-volume snapshot generation, single scenario (latency-spike), cache keyed on the seed-logic hash.Testing
run-evals.shMISS→generate→export and HIT→load branchesNotes
actions/cachepost-save runs even if a later stage fails, so a generated snapshot is cached even if the agent errors — next run won't regenerate.react-is, from chore(charts): upgrade Recharts from 2.13 to 3.x #2610, present on main) unrelated to this change.🤖 Agent-generated (branch prefixed
agent/).