docs/benchmarks.md cross-engine table was measured before column projection landed and understates pgColumnar by ~50x on q6

[jdatcmd]

Summary

docs/benchmarks.md publishes a cross-engine query latency table measured on a build where column projection was silently discarded. It understates pgColumnar substantially, in user-facing documentation.

Evidence

docs/benchmarks.md was last committed 2026-08-02 12:53 (9d55442). #339, which made the reader honour the projection, merged at 17:33 the same day -- 4.7 hours later. Every number in that table therefore comes from a build in which every query read and decoded every column of the table regardless of what it referenced (#338).

The published table (docs/benchmarks.md:378-391) currently states:

query	pgColumnar	TimescaleDB	heap	Citus
q6	81,966	1,771	14,699	8,810
q4	18,856	5,098	17,290	6,694
q5	22,671	10,437	21,247	15,407

Measured today on the bench, current main, same 100M fixture, fixtures asserted:

arm	time	buffers
pgColumnar q6, projection+fold, parallel	1,559 ms	70,416
pgColumnar q6, projection+fold, serial	4,357 ms	69,924
TimescaleDB q6, serial	5,597 ms	603,211
heap q6, parallel	9,067 ms	2,435,728

So the published q6 figure is off by a factor of roughly 50, in the direction of making the project look worse than it is. The table also shows pgColumnar losing to heap on q4 and q5; whether that still holds is unknown, because those are grouped shapes and nobody has measured what projection alone does to them.

What needs doing

1. Re-run the whole cross-engine table on current main. Not a one-row patch. Only q6 has been re-measured, and patching a single row of a table whose other rows come from a different build would be worse than leaving it consistent-but-stale.
2. Resolve the TimescaleDB parallel fault first, or the TimescaleDB column cannot be reproduced. On the bench box its parallel path fails:

   ERROR:  could not read blocks 0..0 in file "base/16384/6345917": read only 0 of 8192 bytes
   CONTEXT:  parallel worker
   

   TimescaleDB 2.29.0, 3 chunks, all compressed. This looks like a fault in this installation rather than a property of TimescaleDB, and it means any cross-engine parallel comparison from this box is currently serial-vs-parallel and therefore unfair in pgColumnar's favour.
3. State the GUC configuration in the table. Three of the four levers default off (enable_ungrouped_vector_agg, enable_parallel_vector_agg, enable_group_vectorization). A number measured with them on is not what a default install gives, and the table should say which it is reporting.
4. Check the storage figures too. The body of Vectorized decompression + aggregation for full-scan aggregates (~4× behind TimescaleDB) #289 says 2.67 GB for pgColumnar against TimescaleDB's 7.79 GB; docs/benchmarks.md:369-373 says 6.4 GB against 7.8 GB at the same row count. The TimescaleDB side agrees between the two, the pgColumnar side does not. One of them is wrong and I could not determine which.

Why this is worth a separate issue

A stale number in an issue misleads the people working on it. A stale number in docs/ misleads everyone who reads the project. This one is also stale in the unusual direction -- it makes pgColumnar look far worse than it measures today -- so it is unlikely to be caught by anyone defending a claim.

[jdatcmd]

Measured. Three corrections to this issue, and one finding that has to ship with the table.

Correction 1: I had the storage claim backwards

This issue says the docs/benchmarks.md storage figures and #289's body disagree, and implies the docs may be wrong. Measured on the bench:

engine	measured	docs/benchmarks.md says
pgColumnar	6,587 MB	6.4 GB
TimescaleDB	7,975 MB	7.8 GB
heap	22 GB	22 GB

The published storage table is correct. It was #289's body, claiming 2.67 GB for pgColumnar, that was wrong. That part of this issue is withdrawn; no storage change is needed.

Correction 2: the apparent correctness failures were my instrument

An initial cross-check reported q4, q5 and q6 producing different results between pgColumnar and heap. That was an md5 over the text of avg() output, which changes in the last digits under cross-worker float reassociation. Re-checked properly on q4, over all 48,000 groups:

• max relative difference between engines: 2.9e-15
• groups present in one engine but not the other: 0
• q6 count: identical (10,224,062)

No correctness problem. Recording it because "3 of 8 queries return wrong answers" is the sort of claim that should not be left standing on a bad test.

Correction 3: the old and new numbers are not the same experiment

Two differences make a row-by-row replacement misleading:

• Indexes. Both cpu_pgc and cpu_heap now carry a (hostname, time DESC) btree. The published table has heap at 11,045 ms on q1 where it now measures 6 ms, a factor of ~1,800. That is not drift; it is almost certainly a run without that index. What the published table measured is not recoverable, because its queries were never written down and bench-tsdb/BENCHMARK_RESULTS.md, cited by Vectorized decompression + aggregation for full-scan aggregates (~4× behind TimescaleDB) #289 as "full data + method", has never existed in this repository.
• TimescaleDB parallel is unavailable on this box. Its parallel path faults:

  ERROR:  could not read blocks 0..0 in file "base/16384/6345917": read only 0 of 8192 bytes
  CONTEXT:  parallel worker
  

  Confirmed not ours: it persists with every pgcolumnar planner hook disabled, and the chunks use the heap AM. Neither workaround is usable: enable_vectorized_aggregation=off still errors, and enable_columnarscan=off silently returns zero rows for a query whose answer is 10,224,062.

So the replacement table will state its methodology and publish its SQL, which the current one does not. I am re-running with all engines serial to match the published methodology rather than mixing arms.

The finding that has to ship with the table: #353

q1, q2 and q3 measure 10.3 s, 123.5 s and 162.0 s against heap's 6, 8 and 8 ms, on an identical Index Scan plan over an equivalent index. The cost is a flat ~28.7 ms per row, unchanged across a 12x difference in row count.

Cause, now confirmed by controlled experiment and filed as #353: the default stripe_row_limit of 150,000 puts a 21-column table over the 32 MB fetch cache cap, so every index fetch re-decodes the whole row group. Same data, same query, same plan, only the group size differing:

stripe_row_limit	per row
150,000 (default)	32.98 ms
100,000	0.177 ms
50,000	0.125 ms

186x, from the shipped default alone.

Publishing q2 at 123 seconds without that explanation would be worse than leaving the stale table: a reader would conclude the storage format is unsuited to point queries, when the actual cause is one default and a cache that cannot hold a default-sized group.

Parallel measurements, for reference

Not the replacement table (methodology differs), but the shape of current main with pgColumnar and heap parallel, TimescaleDB serial:

q	pgc default	pgc all levers	heap par	tsdb serial
q1	10,335	10,253	6	0
q2	123,546	119,534	8	1
q3	161,972	156,701	8	2
q4	7,322	7,863	13,912	4,515
q5	16,443	28,494	17,378	9,576
q6	2,292	1,592	9,083	6,956
q7	368	365	31	196
q8	5,124	5,124	8,715	10,171

pgColumnar wins q4, q6 and q8 against heap and has the smallest storage. The published claim that it loses to heap on q4 and q5 does not reproduce.

One anomaly I cannot yet explain and am not going to hand-wave: q5 is 1.73x slower with the levers on (16,443 to 28,494), which contradicts an earlier measurement of the same shape where enable_group_vectorization helped. Tracking separately rather than publishing a number I do not understand.