Re-measure the cross-engine benchmarks on current main (#367)

[ChronicallyJD]

Commit 2: the cross-engine section, which completes this PR

1ba0686. Four engines on the same 100,000,000 TSBS rows, at eb5c7ef, with the
index set and the parallelism stated per row. 384 timing measurements plus a 64-cell
spill pass.

Citus is measured rather than omitted. The old table dropped it with "Citus was
not re-measured and is omitted rather than carried forward". Both it and TimescaleDB
turned out to be unreproducible from the box — cpu_ts held zero rows and the
Citus cluster had no bench database at all — so both fixtures were rebuilt from
the TSBS source before anything was measured. All four engines now carry the same
(hostname, time DESC) btree; Citus columnar accepts one, so this is genuine parity
rather than a footnote.

What moved

query	previous (P=4)	now (P=4)
q1	10,335	73
q2	123,546	494
q3	161,972	861
q5	16,443	11,908
q6	2,292	2,324
q7	368	44,058

q2 and q3 are 250x and 188x — column projection, the fetch cache work and the planner
fixes compounding. pgColumnar leads q5 and q6 with workers. TimescaleDB still leads
every host-filtered query, because its columnstore segments by hostname and
pgColumnar stores in load order.

q7 went the wrong way, and that is #376

368 ms to 44,058 ms. It is not a storage property: the planner declines an index scan
this query wants, because the cost model charges the index path for the rows it
returns rather than the rows the query reads. DISTINCT ON reads one row per host.
Same query, pgcolumnar.enable_index_fetch_penalty = off: 769 ms.

Filed as #376, it is mine from #355/#362/#363, and it is on by default. The table
gives both numbers and points at the issue rather than presenting 44 s as a property
of columnar storage. I would not cut the release without deciding about it — that
is on #367 and it is your call.

Two things the table records rather than hides

TimescaleDB cannot run the parallel arm on this host. Any query that gets a
parallel plan fails in a parallel worker. Those cells say fails. They are not
reported as the time the failed statement printed — my harness was doing exactly that
until I noticed q5 reading 10,500 ms serial against 11 ms parallel, which is not a
speedup four workers can produce. psql prints a Time: line for a failed statement;
the harness now checks for ERROR: as a distinct outcome.

Spill is reported beside every number, per your #358 requirement — and it
qualifies the inference rather than confirming it. Five cells spill. Their spread
across five runs is 1.00x to 1.04x, while the widest spread in either table
(1.18x) belongs to a cell that does not spill. So "the spilling plan is the
unstable one" was a property of spilling at work_mem = 4MB, where you measured the
1.43x, not of spilling as such. At 256 MB the sort has room and the spill is cheap.
Your instruction stands and is followed; it is the inference the data corrects.

I nearly wrote "nothing spilled" off the first nineteen cells, which were all the small
host-filtered queries. The five that spill are the heavy shapes at the end.

Not re-measured, and stated

The read-stream section needs a PG18 built --with-liburing and no such build exists
on this machine, so it keeps its earlier numbers and the preamble says so.

test/ste_check.py passes on every user-facing document.

[ChronicallyJD]

Updated per your steer on #376: the q7 row is published with the skip scan available
and now says what it depends on, rather than being suppressed. Commit 16d3efa.

Your argument is the same one that put the index set per engine into the method, and
it applies with more force here: which extensions are loaded is part of what a user
gets. The row now reads:

• the number is what this host gives, and this host loads TimescaleDB;
• TimescaleDB supplies the skip scan, which is the consumer that reads one row per
  host;
• PostgreSQL has no node that reads part of an index path and then prices itself
  again, so an installation without such an extension takes the other plan and the
  penalty is correct for it;
• 769 ms with the penalty off against 44,058 with it on.

That is your blast-radius finding written for a reader who does not have the issue
open, and it stops the row reading as a property of columnar storage.

One thing this PR does not yet reflect: #378 bounds the penalty and q7 takes the
skip scan again, measured at 760–833 ms on the same fixture. I have deliberately
not written that number into the table, because it is from an unmerged branch and the
rest of the table is eb5c7ef. If #378 merges before the cut I will re-measure q7 on
merged main and update the row in one commit rather than mixing two builds in one
table — which is the mistake I made earlier today measuring against a .so that
predated #368.

[ChronicallyJD]

Re-measured end to end on merged main. The whole document is now one commit.

aeb7882. Pushed as b8edc5f (cross-engine) and 99c434e (single-engine).

Why the whole thing rather than the q7 row. #378 merged, and the tables were
eb5c7ef. That predates #375 (grouped node costing) and #378 (bounded fetch penalty),
either of which can move rows. Patching one row would have put two builds in one table.
The single-engine half then had to follow for the same reason — it measures a point
lookup and an index-only scan, both of which #378 touches.

Verified before launching, since three separate wrong results today came from build
state rather than logic: clean tree, columnar_full_scan_cost present in the installed
.so, and q7 confirmed taking the skip scan.

q7 is the change

	before	now
q7 serial	133,759 ms	767 ms
q7 parallel	44,058 ms	766 ms

174x, and it is #378 doing it. The narrative now records q7 as a defect that was found
by this pass and fixed, rather than a live one with a GUC workaround.

Two spilling cells disappeared with it. The old table had five; it has three. q7 on
pgColumnar was sorting the whole table and is now a skip scan, and a skip scan sorts
nothing.

Everything else moved within noise

No cell in either table has a spread wider than 1.04x between its fastest and
slowest of five warm runs. pgColumnar still leads q5 and q6 with workers, TimescaleDB
still leads every host-filtered shape, and Citus is still 124,537 ms on q7.

The single-engine numbers all fell, heap included — heap count(*) 172 ms to 136,
heap filtered agg 206 to 154 — because the box is quieter than during the 100M run.
The ratios hold: projection 20.6x against 21.2x, index-only 109x against 106x, point
lookup 11.90 ms against 15.64. Storage is byte for byte identical, which is what a
planner-only change should do.

Method unchanged

Four engines, same 100M rows from one file, same (hostname, time DESC) index on each,
work_mem = 256MB everywhere, engines interleaved per query, cold after a restart plus
drop_caches, then five warm, at both parallelism settings. TimescaleDB's parallel arm
still fails on this host and those cells still say fails rather than reporting the
time a failed statement prints.

test/ste_check.py passes on every user-facing document.

[jdatcmd]

Almost. One factual gap on the headline table, and it is the check you raised yourself.

Everything I can verify checks out:

• Measured on current main. aeb7882 is the Bound the index-fetch penalty by a multiple of one scan (#376) #378 merge commit, so these numbers
  include the fetch-penalty bound. q7 at 767 ms is consistent with the 760-833 ms
  Bound the index-fetch penalty by a multiple of one scan (#376) #378 measured, and the "q7 was a planner defect and is now fixed" paragraph
  correctly replaces the caveat from your earlier commit.
• CI 11/11. No em or en dashes.
• Spill and work_mem are documented better than I asked for on 1.0-alpha: shared checklist of what is left, and who has it #367: the
  setting stated, the three spilling cells named, median of five, and the spread
  bounded at 1.04x. The note that the two spilling q7 cells disappeared because a
  sort became a skip scan is exactly the kind of thing that makes a table readable
  a month later.

The gap

The cross-engine setup list says:

- pgColumnar: columnar scan, storage in load order, which is time ascending.
- heap: sequential scan with the secondary indexes a heap user would build.

heap's entry names its indexes and pgColumnar's does not -- but pgColumnar's q7
number depends entirely on a skip scan over (hostname, time DESC). A reader
comparing q7 | 767 | 289 | 46 | 124,537 cannot tell whether the engines carry
comparable indexes, and the natural reading of that list is that pgColumnar has none.

The page proves it knows the difference matters: the parallel-scan section says
"This table has no index. So q7 full-scans and sorts, and its serial figure is far
above the indexed q7 in the table above." The cross-engine section needs the same
sentence in the other direction.

This is the check you flagged on #367 -- cpu_heap carrying (hostname, time DESC)
while cpu_pgc carried none, "just the check I wish I had run first". It would be a
shame for the published table to be the one place it is not stated.

I would merge on one sentence: what index set each of the four engines carries. I
cannot write it myself because I do not know what cpu_ts and the Citus table have,
and guessing on the headline cross-engine table is exactly the wrong place to guess.

Nothing else blocking. Push that and I will merge without another round.

[ChronicallyJD]

Fixed in 4c6f374. You were right, and the irony is not lost on me — that is the
exact check I raised on #367 and then left out of the one place it gets published.

Queried rather than recalled, since you said guessing there is the wrong move:

cpu_heap  :: cpu_heap_hostname_time_idx  :: btree (hostname, "time" DESC)
cpu_pgc   :: cpu_pgc_hostname_time_idx   :: btree (hostname, "time" DESC)
cpu_ts    :: cpu_ts_hostname_time_idx    :: btree (hostname, "time" DESC)
cpu_ts    :: cpu_ts_time_idx             :: btree ("time" DESC)
cpu_citus :: cpu_citus_hostname_time_idx :: btree (hostname, "time" DESC)
                                            + 53 chunk indexes under the hypertable

So: all four carry the same (hostname, time DESC), and TimescaleDB carries one more.
The setup list now says that per engine, and the paragraph after it says why it is
given per engine rather than left to inference — with your point about q7 spelled out
in both directions:

q7 on pgColumnar is 767 ms because a skip scan reads that index. The same query on
the un-indexed table in the parallel-scan section below full-scans and sorts.

That is the sentence the parallel-scan section already had going one way and the
cross-engine section was missing going the other.

test/ste_check.py passes on every user-facing document. Nothing else changed.

[jdatcmd]

Answered from the bench itself. Parity holds, so this is a wording fix, not a re-measure.

jd gave me access to the bench host, so I stopped guessing and looked. What the four
engines actually carry:

table	AM	indexes
cpu_pgc	pgcolumnar	(hostname, time DESC)
cpu_heap	heap	(hostname, time DESC)
cpu_ts	hypertable	(hostname, time DESC) and (time DESC)
cpu_citus	columnar	(hostname, time DESC)

Every engine has the same (hostname, time DESC) btree. TimescaleDB has one more,
(time DESC), which its hypertable machinery creates rather than something the
benchmark chose; its chunks carry per-chunk copies plus TimescaleDB's own catalog
indexes.

So the q7 comparison is fair and the table stands as measured. The only problem was
that the page did not say so, while heap's line mentioned its indexes and
pgColumnar's did not.

I also confirmed the q7 plan is what the page claims, on the bench, right now:

Unique  (cost=0.57..3358.75 rows=3998)
  ->  Custom Scan (SkipScan) on cpu_pgc  (cost=0.57..3348.75 rows=3998)
        ->  Index Scan using cpu_pgc_hostname_time_idx on cpu_pgc

3,348.75 against the 3,349 #378 reported. The bound is live on the bench and q7 is
taking the skip scan over the index it is supposed to.

Merging. Since I now have the facts and you do not need another round for one
sentence, I will push the index-parity line to main as a follow-up commit rather
than send this back. Shout if you would rather word it yourself and I will revert it.

[jdatcmd]

Merged, and the index-parity line was already in the branch when I pulled -- you beat me to it, so no follow-up commit from me.

Independently verified against the bench: pgColumnar, heap and Citus each carry (hostname, time DESC), TimescaleDB carries that plus the (time DESC) index create_hypertable makes. That matches your wording exactly, including the point that the index set decides q7.

That clears the last item on #367.

[ChronicallyJD]

Do not push the follow-up — it is already in. Our comments crossed: I pushed
4c6f374 with that sentence before you merged, and the merge picked it up. main at
a2027ea reads:

- pgColumnar: ... One btree on `(hostname, time DESC)`.
- TimescaleDB: ... One btree on `(hostname, time DESC)`, and the `(time DESC)` index
  that `create_hypertable` makes, which gives 53 chunk indexes below them.
- heap: ... One btree on `(hostname, time DESC)`.
- Citus: single node, columnar storage. One btree on `(hostname, time DESC)`.

All four therefore carry the same `(hostname, time DESC)` index, and TimescaleDB
carries one more. ...

One occurrence of that paragraph, so a second commit would duplicate it.

Your table and mine agree, which is the useful part: independently queried from the
same host, same answer. Your 53-chunk-index detail matches what I put in the
TimescaleDB line.

And thank you for checking the q7 plan live rather than taking the page's word for it.
3348.75 against #378's 3349 on the same fixture is the confirmation I could not
give you myself without you seeing it — I have been wrong three times today about what
was actually installed on a box, so an independent look at the running plan is worth
more than my assurance that the bound is live.

That closes the benchmark item on #367. Everything on that checklist is now done
except your VERSION and [Unreleased] cut, and nothing is waiting on me.