Skip to content

docs: re-measure the cross-engine table on current main (#348) - #356

Merged
jdatcmd merged 1 commit into
mainfrom
docs/348-rerun-cross-engine
Aug 3, 2026
Merged

docs: re-measure the cross-engine table on current main (#348)#356
jdatcmd merged 1 commit into
mainfrom
docs/348-rerun-cross-engine

Conversation

@jdatcmd

@jdatcmd jdatcmd commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Fixes #348.

Why

The published cross-engine table was committed 2026-08-02 12:53. Column projection merged at 17:33 the same day. Every figure in it came from a build where the reader decoded every column of every row group regardless of the query (#338).

It understated the project, which is the direction nobody audits:

published measured today
q6 pgColumnar 81,966 ms 1,592 ms
q6 TimescaleDB 1,771 ms 6,956 ms
q4, q5 pgColumnar loses to heap does not reproduce

Why it is a new run, not a line-by-line correction

Two things make a row-by-row edit misleading:

So the method is stated and the SQL is published with the table, which is the main thing the old one lacked.

q1 to q3 ship with their cause attached

They cost ~28.7 ms per row returned, flat across a twelve-fold change in row count, on the same Index Scan plan heap uses. That is #353 (default stripe_row_limit over the 32 MB fetch cache), compounded by #355 (planner takes an index scan for ordering without modelling fetch cost).

Publishing q2 at 123 seconds without that would read as a property of columnar storage. It is one default and one cost model, and the doc says so, with the workaround.

TimescaleDB is serial and labelled

Its parallel path fails on the bench host with could not read blocks 0..0. Not caused by pgColumnar: it persists with every pgColumnar planner hook disabled, and the chunks use the heap AM. Neither workaround is usable, one still errors and the other silently returns zero rows.

Citus is omitted rather than carried forward from an unknown configuration.

Storage table unchanged

Measured 6,587 MB / 7,975 MB / 22 GB against its published 6.4 / 7.8 / 22 GB. It was correct. #348 originally suspected it; that suspicion is withdrawn. The 2.67 GB figure in #289's body was the wrong one.

Correctness

Verified per query against heap. Counts and max match exactly. Averages agree to 2.9e-15 relative difference, the residual being float reassociation across workers. No group appears in one engine and not the other.

An earlier md5-based check appeared to show q4, q5 and q6 mismatching. That was the instrument: md5 over the text of avg() output changes on the last digit. Recorded so it is not rediscovered.

Gate

docs_style passes. The STE line-length rule required splitting several sentences.

🤖 Generated with Claude Code

https://claude.ai/code/session_011miCFRSatixeNRw3w5yNq8

The published cross-engine query latency table was measured on 2026-08-02 at
12:53. Column projection merged at 17:33 the same day, 4.7 hours later. Every
figure in it therefore came from a build where the reader loaded and decoded
every column of every row group it visited, whatever the query referenced
(#338, fixed in #339).

It understated the project, which is the direction nobody checks. It published
q6 at 81966 ms against TimescaleDB's 1771. Measured today the same shape is
1592 ms against 6956. It also showed pgColumnar losing to heap on q4 and q5,
which does not reproduce.

This is a new run rather than a row-by-row correction, because the two are not
the same experiment. The fixture now carries indexes the earlier one did not,
heap moving from 11045 ms to 6 ms on q1, and the earlier queries were never
recorded. bench-tsdb/BENCHMARK_RESULTS.md, cited by #289 as the full data and
method, has never existed in this repository. So the method is stated here and
the SQL is published with the table.

TimescaleDB is reported serial and labelled as such. Its parallel path fails on
the bench host with "could not read blocks 0..0". That is not caused by
pgColumnar: it persists with every pgColumnar planner hook disabled, and the
chunks use the heap access method. Neither available workaround is usable, one
still errors and the other silently returns zero rows.

Citus is omitted rather than carried forward from a run whose configuration is
unknown.

q1 to q3 are published with their cause attached. They cost about 28.7 ms per
row returned, flat across a twelve-fold change in row count, on the same index
scan plan heap uses. That is #353, the default stripe_row_limit putting a table
this wide over the 32 MB fetch cache, compounded by #355, the planner choosing
an index scan for ordering without modelling the fetch cost. Publishing 123
seconds without that would read as a property of columnar storage rather than
as one default and one cost model.

The storage table is unchanged. Measured 6587 MB, 7975 MB and 22 GB against its
published 6.4 GB, 7.8 GB and 22 GB. It was correct. The 2.67 GB figure in #289's
body was the wrong one.

Results were verified against heap per query. Counts and max match exactly,
averages agree to 2.9e-15 relative difference, and no group appears in one
engine and not the other.

docs_style passes, and the STE line-length rule required splitting several
sentences.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011miCFRSatixeNRw3w5yNq8

@ChronicallyJD ChronicallyJD left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving — a correct, honest re-measurement and a clear improvement on the stale table.

Verified:

  • Method is stated and reproducible, and the SQL is published — the old table's key omission.
  • Internal consistency holds: q1/q2 are 28.7 ms/row × 360/4,320 rows; the q6 projection ablation (45,094 → 6,491) matches the #289/#348 numbers; the correctness check (counts and max exact, avg within 2.9e-15) is the expected cross-worker float reassociation.
  • q1–q3 are correctly framed as the #353 fetch-cliff defect (compounded by #355), not a storage property, with the smaller-stripe_row_limit workaround stated — so the 123 s doesn't misread as columnar being unfit for point queries.
  • The q5 vectorization regression is disclosed and tracked to #349; the GUC-defaults-off caveat is stated; Citus is honestly omitted rather than carried forward from an unknown config.
  • Cited commit 00290d7 (#350 merge) is valid; #354 post-dates it but doesn't touch these query shapes.

One follow-up, not a blocker: q1–q3 are exactly the shape my #353 fix (#357) addresses — once #357 lands, those rows should be re-measured (they drop ~200×) and the "until that is fixed" note updated. Fine to merge now with the caveat attached; the re-measure is a small follow-up.

@jdatcmd
jdatcmd merged commit 2b75ffb into main Aug 3, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docs/benchmarks.md cross-engine table was measured before column projection landed and understates pgColumnar by ~50x on q6

2 participants