Add declarative SQL benchmark matrix profiles and vx-bench matrix CLI#8801
Add declarative SQL benchmark matrix profiles and vx-bench matrix CLI#8801connortsui20 wants to merge 1 commit into
vx-bench matrix CLI#8801Conversation
### Motivation - Centralize CI benchmark coverage in Python so workflow YAMLs do not embed large, duplicated JSON matrices. - Provide a reproducible way to emit GitHub Actions `include` arrays for named profiles to keep coverage and workflow shape consistent. ### Description - Add `bench_orchestrator/matrix.py` which implements `TargetSet`, `BenchmarkDef`, `Profile`, storage handling, and `resolve_matrix` to render profile entries into GitHub Actions matrix entries. - Add `bench_orchestrator/benchmarks.py` to declare `BENCHMARKS` and `PROFILES` as the canonical source of benchmark definitions and CI profiles. - Add a new CLI command `matrix` in `bench_orchestrator/cli.py` to list profiles or emit JSON with `resolve_matrix`, and wire in the new modules. - Update `README.md` to document the `matrix` command and the declarative benchmark matrix approach. ### Testing - Added unit tests in `bench-orchestrator/tests/test_matrix.py` covering policy narrowing, resolver output shape, profile roles, preserved display/scale values, and CLI behavior, and these tests were run with `pytest` and passed. - Exercised the CLI behavior using the `typer` test runner to assert `vx-bench matrix develop` emits JSON and unknown profiles return a failing exit code, and these checks succeeded. Signed-off-by: Codex <codex@openai.com>
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7ee63eed84
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "id": benchmark.id, | ||
| "subcommand": benchmark.subcommand, | ||
| "name": benchmark.name, | ||
| "targets": [target.to_dict() for target in run_targets], |
There was a problem hiding this comment.
When the output of vx-bench matrix ... is used as benchmark_matrix for the current reusable SQL workflow, the Resolve benchmark targets step in .github/workflows/sql-benchmarks.yml:562-565 reads matrix.pr_targets/matrix.develop_targets, not matrix.targets. With only targets here, that step emits JSON null; the subsequent vx-bench run --targets-json 'null' fails in parse_targets_json because targets must be an array. Either keep emitting the workflow keys or update the workflow to consume targets before treating this output as workflow-ready.
Useful? React with 👍 / 👎.
Benchmarks: Vortex queriesVerdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.011x ➖, 0↑ 0↓)
datafusion / parquet (0.924x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (0.967x ➖, 0↑ 0↓)
duckdb / parquet (0.991x ➖, 0↑ 0↓)
No file size changes detected. |
Codecov Report✅ All modified and coverable lines are covered by tests. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
1 participant
Motivation
includearrays for named CI profiles (develop,pr,nightly) and keep workflow-facing fields explicit.Description
bench_orchestrator/bench_orchestrator/matrix.pywhich implementsTargetSet,BenchmarkDef,Profile, storage handling,_data_formats, andresolve_matrixto render profile entries into GitHub Actionsincludeentries.bench_orchestrator/bench_orchestrator/benchmarks.pydeclaringBENCHMARKSandPROFILESas the canonical coverage source and wire in storage-aware remote dataset paths (workflow-facingremote_storage).matrixsubcommand to thevx-benchCLI inbench_orchestrator/bench_orchestrator/cli.pyto list profiles or emit resolved JSON, and updateREADME.mdto document the command and the declaration/profile/rendering model.bench-orchestrator/tests/test_matrix.pythat assert target-narrowing behavior, resolver output shape, profile role separation, preserved display/scale values, and CLI success/failure behavior, and update code to useremote_storageinstead ofremote_keyfor workflow compatibility.Testing
ruff checkandpython -m py_compileagainst the new modules, and both checks passed.git diff --checkand it passed with no whitespace/format errors.uv run --project bench-orchestrator pytest bench-orchestrator/tests/test_matrix.pybut it was blocked because the environment hasuv0.7.22while the project requiresuv >=0.8.0.PYTHONPATH=bench-orchestrator python -m pytest bench-orchestrator/tests/test_matrix.pybut collection failed due to missing runtime dependencies (for examplepandas) in the sandbox environment.Codex Task