Add declarative benchmark matrices and shared workflows

[connortsui20]

Rationale

Benchmark coverage was duplicated across hand-maintained SQL matrices and two nearly identical Rust micro-benchmark jobs. This made it difficult to see what CI runs and allowed the workflows to drift.

This PR moves all benchmark declarations into one deliberately obvious place: bench_orchestrator/benchmarks.py. BENCHMARKS declares the SQL suites and supported targets, PROFILES declares the SQL coverage used by each workflow, and MICRO_BENCHMARKS lists the Rust benchmark binaries. Matrix rendering remains separate in matrix.py.

The implementation remains deliberately concrete: there is no generic predicate or tag language.

Commit structure

1. feat(bench-orchestrator): declare benchmark matrix profiles
   • Adds the declaration model, CLI command, documentation, and focused contract tests.
2. refactor(ci): resolve SQL benchmark matrices from profiles
   • Replaces the duplicated SQL YAML matrices with the declarative profiles.
3. refactor(ci): share Rust micro-benchmark workflow
   • Replaces the duplicated develop/PR Rust jobs with one reusable workflow.

Behavior

A semantic comparison against the removed YAML confirms that the generated develop and nightly SQL matrices are exact matches, including IDs, names, target ordering, scale-factor strings, and remote paths.

The default PR SQL profile intentionally fixes the inconsistencies described in #4935:

• adds Appian and TPC-H SF=10 on S3;
• removes Arrow from the two NVMe TPC-H jobs so every default PR job uses DataFusion/DuckDB × Parquet/Vortex.

The explicit full-PR action uses the complete declared target set. The Rust micro-benchmark consolidation preserves the existing mode-specific comparison, upload, ingest, profiling, and failure-reporting behavior.

API impact

No Vortex library APIs change. vx-bench gains the matrix subcommand; its existing commands are unchanged.

[joseph-isaacs]

Do we not use yml for the ci ones? Is this hook into there?

[connortsui20]

@joseph-isaacs Yes — it's now hooked into the workflow YAML, not just a standalone module. The benchmark workflows resolve their matrix from a plan job that runs vx-bench matrix <profile> and feed it into the bench job via fromJSON:

• sql-benchmarks.yml — the two inline full/base JSON matrices are deleted; it now takes a profile input and resolves it.
• bench-core.yml (new) — reusable micro-benchmark workflow, replacing the duplicated steps that were copy-pasted across bench.yml/bench-pr.yml.
• bench.yml, bench-pr.yml, sql-pr.yml, nightly-bench.yml — thin callers passing a profile (develop/pr/nightly/micro).

Net −549 lines across the workflows. develop/nightly resolve to the same target sets as before; the source of truth is bench-orchestrator/bench_orchestrator/matrix.py.

---

Generated by Claude Code

[codspeed-hq]

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 1698 untouched benchmarks
⏩ 9 skipped benchmarks¹

---

_{Comparing ct/bench-declarative (51db3b2) with develop (cdaa5a5)}

Footnotes

1. 9 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩