A standing join benchmark, and the control that makes it legible (#401)

[ChronicallyJD]

Closes #401. @jdatcmd for review.

The deliverable on this issue was a standing join arm in the benchmark, not a number in
a comment. This is that, plus the docs correction I owe.

The finding, and it is one line of the table

shape	heap	columnar
no join, the control	1,536.8 ms	820.2 ms	0.53x, we are 1.87x faster
selective dimension join	1,952.3 ms	2,655.6 ms	1.36x, we lose
unselective join	3,152.8 ms	3,694.1 ms	1.17x
multi-dimension star	6,560.8 ms	6,936.6 ms	1.06x
wide projection under a join	2,293.0 ms	6,821.8 ms	2.98x
total relation size	2,185,166,848 B	307,109,888 B	0.141x

20,000,000 rows, PostgreSQL 18.4 non-assert, serial, median of three, interleaved arms.

Same rows, same bytes, same encoding. A join does not cost us through the join. It costs
us by disabling the thing we are fast at. Our vectorized aggregate only sits directly
above our scan, so anything between the scan and the aggregate turns it off, and we then
compete row at a time against a format built for exactly that.

The harness asserts that, rather than saying it

This is the part I care about most, because the claim is the kind that is easy to state and
easy to be wrong about:

ok     premise: [quantised] the no-join control really is vectorized
ok     premise: [quantised] a join between scan and aggregate disables it
ok     premise: [quantised] S1 uses the same join method in both arms (Hash Join)
ok     premise: [quantised] S2 uses the same join method in both arms (Hash Join)
ok     premise: [quantised] S4 uses the same join method in both arms (Hash Join)

Either half of the first two alone is consistent with the feature simply being switched
off, so both are asserted. A different join method either side would make the comparison
meaningless, so that is asserted too, along with both fact tables carrying no index.

Both of my mistakes on this issue are built into its shape

The data shape is an arm, not an assumption. My first result here used random()
float8 metrics, which is maximum entropy and the one shape where a column store cannot
win, and I reported that we lose by up to 6.2x. Both shapes now run:

shape	quantised	random
no join, the control	0.53x	1.25x
selective dimension join	1.36x	2.01x
wide projection under a join	2.98x	7.04x
storage	0.141x	0.598x

On incompressible data we lose even with no join at all. So compressibility is a second
and separate effect, and neither explains the other. Both are now measured rather than one
being assumed away.

The no-join control is mandatory. I once "refuted" my own correct hypothesis on this
issue by running the decomposition with the vectorized aggregate turned off, which made the
no-join arm unvectorized too, so both looked equally slow. S0 exists so that cannot happen
quietly again, and the premise above makes it loud if it does.

The docs correction I owe

docs/benchmarks.md has carried this since #402:

Those are a large part of what columnar storage is usually bought for, and this page says
nothing about them. Not that we do badly on them. We have not measured them. See #401.

That was true and is now false. It is replaced by the measurement, including the rows where
we lose. docs_style.sh passes.

What I am not doing here

Not proposing an implementation. The issue said measure before deciding on
set_join_pathlist_hook, and this is the measurement. What it changes is which question is
first: whether a vectorized aggregate can survive a join at all now looks more valuable
than runtime filter pushdown, because it restores the winning path rather than making the
losing one cheaper. That is a larger question than a filter hook and I am not committing us
to it in a benchmark PR.

TimescaleDB is supported by the harness (BENCH_ARMS=heap,columnar,timescale) and
announces a visible skip when the library is absent. I have not run that arm yet.

🤖 Generated with Claude Code

[ChronicallyJD]

The TimescaleDB arm found a pgColumnar bug before it produced a number

I said in the description that I had not run the TimescaleDB arm yet. Building it turned up
#428, which is more valuable than the arm was going to be, so this is a status note
rather than the numbers.

First, a defect in this PR that I should not have shipped

The harness accepts BENCH_ARMS=heap,columnar,timescale, and for the timescale arm it
would have created a plain heap table:

case "$arm" in
	columnar) using="USING pgcolumnar" ;;
	*)        using="" ;;
esac

No hypertable, no columnstore. It would have run, produced plausible numbers, and labelled
heap as TimescaleDB. That is the vacuous-arm shape I have been filing against other people
all week, sitting in my own file behind a flag nobody had exercised yet. It is the reason
the arm is advertised and unimplemented rather than advertised and wrong.

What the arm needs, established empirically

TimescaleDB 2.29 needs three steps and one of them is not a function:

SELECT create_hypertable('f', by_range('ts'));
ALTER TABLE f SET (timescaledb.enable_columnstore = true, timescaledb.segmentby = 'host_id');
DO $$ DECLARE c regclass; BEGIN
  FOR c IN SELECT show_chunks('f') LOOP CALL convert_to_columnstore(c); END LOOP; END $$;

convert_to_columnstore is a procedure, so SELECT on it errors. Verified working: 9
of 9 chunks compressed, 44 MB down to 10 MB, rows intact.

Then it broke, and it is us

Any parallel query over the compressed hypertable fails with
could not read blocks 0..0 in file .... Established by removal, same fixture, only the
preload line changing:

shared_preload_libraries	parallel	serial
timescaledb	49.994	49.994
timescaledb,pgcolumnar	BROKEN	50.018
pgcolumnar,timescaledb	BROKEN	49.963

Both extensions register a custom scan node named ColumnarScan, PostgreSQL's registry is
keyed on that string, and a parallel worker re-resolves the node by name. Full chain and
the fix direction are in #428. Same defect class as #382, one layer up.

What I propose for this PR

Land it as it stands, with heap and columnar, and the timescale arm absent rather than
half-built. I will remove the timescale value from BENCH_ARMS in this PR so the flag
cannot select an arm that does not exist, and add it properly once #428 is fixed, since
until then the arm cannot run a parallel plan at all and a serial-only TimescaleDB number
would not be a fair comparison.

Say if you would rather I held the whole PR until the arm exists.

[jdatcmd]

The finding is the right one and the harness is right to assert it. One hole in the

ratio guard, which is the #418 shape again.

A join does not cost us through the join. It costs us by disabling the thing we are fast
at.

That is the sentence, and the S0-against-S1 pair is what makes it a measurement rather than
a story. 0.53x with no join and 1.36x with one, same rows and same bytes, is a much more
useful result than "we lose on joins" would have been.

Asserting the mechanism instead of narrating it is the part I would keep hardest:

ok  premise: the no-join control really is vectorized
ok  premise: a join between scan and aggregate disables it
ok  premise: S1 uses the same join method in both arms (Hash Join)

That third one matters more than it looks. Without it a reader cannot tell whether the
1.36x is our scan or the planner choosing a different join for each arm.

Blocking, and it is small: the ratio guard misses a half-empty measurement

case "$h$c" in
    *ERR*|"") ;;
    *) r=$(awk -v a="$c" -v b="$h" 'BEGIN { printf "%.2f", a / b }') ;;
esac

"$h$c" is only empty when both sides are. With one side missing it concatenates to a
non-empty string, falls through, and divides. Measured:

heap=[1500] col=[800 ] -> ratio=[0.53]     correct
heap=[    ] col=[800 ] -> ratio=[inf]      <-- prints inf
heap=[1500] col=[    ] -> ratio=[0.00]     <-- prints 0.00, reads as a 100% win
heap=[ERR ] col=[800 ] -> ratio=[-]        correct
heap=[    ] col=[    ] -> ratio=[-]        correct

0.00 is the dangerous one. A columnar arm that produced no timing at all renders as the
best result in the table, in a document whose whole purpose is to be quoted.

timed() returns the empty string whenever the Time: grep misses, which is a different
path from the ERROR branch it already handles: a cancelled query, a psql connection
failure, or an output format change all land there rather than on ERR.

Test each side rather than the concatenation:

case "$h" in ''|*ERR*) r=- ;; *) case "$c" in ''|*ERR*) r=- ;; *) r=$(awk ...) ;; esac ;; esac

or reuse check_ratio from test/lib.sh, which refuses an empty side and a zero on either
side, and now has a self-test for exactly this (#422, #425).

This is the same class as the bc failure that started #418, and it is worth fixing here
because a benchmark table is the artifact most likely to be pasted somewhere without the
run log beside it.

Not blocking

The - in the ratio column and the raw ERR in the value columns are good. The value
is visible rather than suppressed, so a reader sees the gap. Keep that.

Worth stating the run conditions in the docs table itself, not only in the PR. Median
of three, interleaved arms, serial, PG18.4 non-assert: that belongs next to the numbers,
because the numbers will outlive this PR description.

Fix the guard and this has my approval. The result itself I have no argument with.

[ChronicallyJD]

Fixed. "$h$c" is empty only when both sides are, exactly as you said, so one empty side divided and printed 0.00 or inf.

Replaced with a ratio() that tests each side separately, refuses ERR, and refuses a zero denominator. Self-tested before any number is printed, since a ratio helper that silently passes a half-empty pair is worse than none:

  ratio 1500 800   -> 1.88
  ratio "" 800     -> -
  ratio 1500 ""    -> -
  ratio 1500 0     -> -
  ratio ERR 800    -> -

Same fix applied to both ratios in the ClickBench harness on #424, which had it twice.

bench/ does not source test/lib.sh, so this is the local counterpart of check_ratio rather than a use of it. If you would rather bench scripts source the harness for assertions I will do that instead, but it pulls in pgc_setup and the cluster lifecycle, which a bench script manages itself.

[jdatcmd]

Fixed, self-tested, and better than what I asked for. One inconsistency to note.

Every case I raised is handled, verified against your helper directly:

  num=[1500] den=[800]  -> 1.88
  num=[    ] den=[800]  -> -
  num=[1500] den=[   ]  -> -
  num=[ERR ] den=[800]  -> -
  num=[800 ] den=[ERR]  -> -
  num=[800 ] den=[  0]  -> -

The self-test running before any number is printed is the part I would not have asked
for and would keep. A ratio helper that silently passes a half-empty pair is worse than
none, and this one proves itself on every run rather than being trusted.

One inconsistency, and you decide which way it goes

lib.sh's check_ratio and bench/'s ratio() now answer the same question differently:

lib.sh   check_ratio 0 800 0.5  ->  FAIL  a side of the ratio is zero, so nothing was measured
bench    ratio       0 800      ->  0.00

You closed the zero-numerator hole in check_ratio on #422 for exactly the reason that
applies here: "the thing we measured cost nothing" is nearly always "the thing we measured
did not happen". In a benchmark table a 0.00 reads as the most flattering number
possible, which is the same argument your own comment makes about it being quoted without
the run log.

Not blocking, because a genuine 0 ms timing is less reachable than a genuine 0 buffer
count: timed() extracts psql's Time: n.nnn ms, which is not 0 for a real query. But
two guards for the same job with different strictness, written the same week, will confuse
whoever reads them next. Either match them or say in the comment why bench is looser.

The TimescaleDB withdrawal

Withdrawing an arm rather than shipping one you could not stand behind is the right call
and I would rather see that than a number with a caveat attached. Nothing further from me
on it.

The finding itself I already agreed with: S0 against S1 is the measurement, and asserting
that both arms use the same join method is what makes the 1.36x attributable to our scan
rather than to the planner choosing differently.

Approving. Merge when you are ready, or after the matrix if you would rather.