Skip to content

Own SQL benchmark suite#8719

Merged
myrrc merged 1 commit into
developfrom
myrrc/sql-bench
Jul 16, 2026
Merged

Own SQL benchmark suite#8719
myrrc merged 1 commit into
developfrom
myrrc/sql-bench

Conversation

@myrrc

@myrrc myrrc commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Testing SQL queries' correctness is done in vortex-sqllogictest, but we also
want to test relative performance of queries not covered in existing benchmarks.
One example is aggregation with filter which should be much more performant
once we use zone maps statistics, but this type of query isn't present is
existing benchmarks.

This PR introduces a new query-per-file dataset and support for running it in CI

@myrrc myrrc requested a review from AdamGS July 10, 2026 15:55
@myrrc myrrc force-pushed the myrrc/sql-bench branch from c16907b to 728668e Compare July 10, 2026 16:14
@codspeed-hq

codspeed-hq Bot commented Jul 10, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

✅ 1698 untouched benchmarks
⏩ 9 skipped benchmarks1


Comparing myrrc/sql-bench (c4e8163) with develop (198cbd5)2

Open in CodSpeed

Footnotes

  1. 9 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

  2. No successful run was found on develop (d54f70d) during the generation of this report, so 198cbd5 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

Comment thread vortex-bench/.gitignore Outdated
Comment thread vortex-bench/src/vortex_queries.rs
@myrrc myrrc force-pushed the myrrc/sql-bench branch from 728668e to 2c9252a Compare July 15, 2026 15:25
@myrrc myrrc requested a review from AdamGS July 15, 2026 15:25
@myrrc myrrc enabled auto-merge (squash) July 15, 2026 15:25
@myrrc

myrrc commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

As this is fast we should probably run this on every commit @AdamGS WDYT?

@myrrc myrrc force-pushed the myrrc/sql-bench branch from 2c9252a to f2e8697 Compare July 16, 2026 09:43
@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Polar Signals Profiling Results

Latest Run

Status Commit Job Attempt Link
🟢 Done c4e8163 1 Explore Profiling Data
Previous Runs (1)
Status Commit Job Attempt Link
🟢 Done f2e8697 1 Explore Profiling Data

Powered by Polar Signals Cloud

@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

🚨🚨🚨❌❌❌ SQL BENCHMARK FAILED ❌❌❌🚨🚨🚨

Benchmark Vortex queries (full) failed! Check the workflow run for details.

Signed-off-by: Mikhail Kot <mikhail@spiraldb.com>
@myrrc myrrc force-pushed the myrrc/sql-bench branch from f2e8697 to c4e8163 Compare July 16, 2026 10:39
@myrrc myrrc requested a review from AdamGS July 16, 2026 10:45
@myrrc myrrc merged commit 1c419e5 into develop Jul 16, 2026
76 of 78 checks passed
@myrrc myrrc deleted the myrrc/sql-bench branch July 16, 2026 10:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants