From df5a574e3acb616fc7677a615834cc99a9a906c6 Mon Sep 17 00:00:00 2001 From: "Joshua (D) Drake" Date: Mon, 3 Aug 2026 22:31:27 -0600 Subject: [PATCH 1/5] docs: re-measure the cross-engine benchmarks on current main Completes the 1.0-alpha benchmark pass (#367). Four engines on the same 100,000,000 TSBS rows loaded from one file, at eb5c7ef, with the index set and the parallelism stated per row. Citus is measured rather than omitted. The previous table dropped it with a note saying it had not been re-measured. Both it and TimescaleDB turned out to be unreproducible from the box: cpu_ts held zero rows and the Citus cluster had no bench database at all, so those fixtures were rebuilt from the TSBS source before anything was measured. Every query is given at max_parallel_workers_per_gather 0 and 4. The serial number measures the storage format; the parallel number is closer to what an installation gets, and after #362 the difference between them can be a different plan rather than only a different speed. Against the previous table, q2 improves 123,546 ms to 494 and q3 161,972 to 861. That is column projection, the fetch cache work and the planner fixes compounding. pgColumnar leads q5 and q6 with workers. TimescaleDB still leads every host-filtered query because its columnstore segments by hostname. Three things the table now records rather than hides: TimescaleDB cannot run the parallel arm on this host. A query that gets a parallel plan fails in a parallel worker, unable to read blocks 0..0 of a relation file. Those cells say so. They are not reported as the time the failed statement printed, which is small enough to look like a very fast query. Temp spill is reported beside every number, per #358. Five cells spill. They are not the unstable ones: their spread across five runs is 1.00x to 1.04x, while the widest spread in the tables belongs to a cell that does not spill. q7 is a regression and says so. The planner declines an index scan the query wants, because the cost model charges the index path for the rows it returns rather than the rows the query reads. The same query runs in 769 ms with pgcolumnar.enable_index_fetch_penalty off, against 44,058 ms with the default. That is #376, found by this pass. Co-Authored-By: Claude Opus 5 (1M context) --- docs/benchmarks.md | 237 +++++++++++++++++++++++++++------------------ 1 file changed, 143 insertions(+), 94 deletions(-) diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 1a437a0..6988c8d 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -36,8 +36,8 @@ The read stream section is not re-measured. It needs a PostgreSQL 18 build with `--with-liburing`, and no such build exists on this machine. Its numbers are from the earlier run and say so. The Cross-engine comparison and the parallel sections are a separate, larger run. -It ran on the bench host, at up to 100,000,000 rows, dated 2026-08-02. Each of -those sections states its own method. +It ran on the bench host, at up to 100,000,000 rows, dated 2026-08-04, at the same +commit `eb5c7ef`. Each of those sections states its own method. They show the shape of the trade and not a precise score. The dataset is synthetic. It mixes column shapes that suit different encodings, and it does this deliberately. A table of fully random values will therefore look worse, and a @@ -371,20 +371,15 @@ suite. This section is a separate, larger run. It compares pgColumnar with heap, TimescaleDB, and Citus on the TSBS `cpu` workload, at 100,000,000 rows across 21 columns. The data is loaded byte for byte the same way into each engine. -Method: the bench host, 16 vCPU and 62 GB, PostgreSQL 18.4 non-assert, on -2026-08-02 at the current `main` plus the export feature. Each engine has the box -to itself. Latency is client-observed, one cold run after a restart with the -cache dropped, then the median of three warm runs, interleaved per query. Each -engine uses the configuration its own users would choose: +The bench host has 16 vCPU and 62 GB. Each engine uses the configuration its own +users would choose: - pgColumnar: columnar scan, storage in load order, which is time ascending. - TimescaleDB: compressed columnstore, segmented by `hostname`, ordered by `time` descending. - heap: sequential scan with the secondary indexes a heap user would build. -- Citus: single node, columnar storage, sharded. +- Citus: single node, columnar storage. -The query engines run serial (`max_parallel_workers_per_gather = 0`). That -isolates the storage and scan path. It is not pgColumnar at its ceiling. The -parallel scan below adds about four times on these same shapes. +The full method follows the storage table. ### Cross-engine storage @@ -393,93 +388,147 @@ Total relation size, including indexes, for the 100,000,000 rows: | engine | size | smaller than heap | | --- | --- | --- | | heap | 22 GB | 1.0x | -| TimescaleDB columnstore | 7.8 GB | 2.8x | -| pgColumnar (zstd) | 6.4 GB | 3.4x | +| pgColumnar (zstd) | 6,590 MB | 3.4x | +| TimescaleDB columnstore | 7,975 MB | 2.8x | +| Citus columnar | 8,147 MB | 2.8x | -pgColumnar is the smallest of the four. The encoding layer does most of this, and -zstd compounds it, as the [Storage](#storage) section shows in detail. +pgColumnar is the smallest of the four. ### Cross-engine query latency -Measured on 2026-08-03 against `main` at commit `00290d7`, on the same -100,000,000-row fixture. This is a new run, not a correction of the previous -table row by row. The earlier figures were taken before column projection landed (issue #338, -fixed in #339). Every query then read and decoded every column of the table, -whatever it referenced. The queries behind those figures were also not recorded. -The two runs cannot be compared line by line. - -Method for this run, stated so it can be repeated: - -- All three engines hold the same 100,000,000 rows, verified before measuring. -- `cpu_pgc` and `cpu_heap` each carry a `(hostname, time DESC)` btree. - TimescaleDB carries the equivalent index on its chunks. -- pgColumnar and heap run with `max_parallel_workers_per_gather = 4`. - TimescaleDB runs serial, because its parallel path fails on this host with - `could not read blocks 0..0`. That fault is not caused by pgColumnar: it - persists with every pgColumnar planner hook disabled, and the chunks use the - heap access method. -- pgColumnar figures are given twice: with default settings, and with - `enable_ungrouped_vector_agg`, `enable_parallel_vector_agg` and - `enable_group_vectorization` all on. Those three default to off. -- Warm, `EXPLAIN (ANALYZE)` execution time, after one warm-up run. -- Citus was not re-measured and is omitted rather than carried forward from a - run whose configuration is unknown. - -Milliseconds: - -| query | shape | pgColumnar | pgColumnar, all options on | heap | TimescaleDB (serial) | -| --- | --- | --- | --- | --- | --- | -| q1 | one host, 1 hour | 10335 | 10253 | 6 | 0 | -| q2 | one host, 12 hours | 123546 | 119534 | 8 | 1 | -| q3 | one host, 12 hours, 5 aggregates | 161972 | 156701 | 8 | 2 | -| q4 | all hosts, 12 hours, group by host | 7322 | 7863 | 13912 | 4515 | -| q5 | all hosts, 12 hours, 10 aggregates | 16443 | 28494 | 17378 | 9576 | -| q6 | full scan, one value filter | 2292 | 1592 | 9083 | 6956 | -| q7 | last point per host | 368 | 365 | 31 | 196 | -| q8 | top 20 by max | 5153 | 5124 | 8715 | 10171 | - -The SQL is given at the end of this section. The previous table did not record -it, which is the main reason its numbers cannot be checked. - -Results were verified against heap per query. Counts and `max` aggregates match -exactly. Averages agree to within 2.9e-15 relative difference. The residual is float -reassociation across parallel workers. Every group present in one engine is -present in the other. - -**q1 to q3 are a defect, not a storage property.** They filter on one host and -return 360, 4,320 and 4,320 rows. Both engines take the same `Index Scan` plan on the equivalent index. -pgColumnar costs about 28.7 milliseconds per row returned. That cost is flat -across a twelve-fold change in row count. The cause is -[issue #353](https://github.com/jdatcmd/pgcolumnar/issues/353). The default -`stripe_row_limit` of 150,000 puts a table this wide over the 32 MB fetch cache -limit. Every fetch by row number then decodes the whole row group again. Lowering -`stripe_row_limit` to 100,000 on the same data takes the same query from 32.98 to -0.177 milliseconds per row. Until that is fixed, a table that serves selective -point queries through an index should be created with a smaller -`stripe_row_limit`. - -[Issue #355](https://github.com/jdatcmd/pgcolumnar/issues/355) compounds it. -The planner will choose an index scan over a columnar table to obtain ordering. -It does that because the per-row fetch cost is not modelled. That is why these queries should -not be read as a measure of the storage format. - -**pgColumnar leads on the scan-bound shapes.** q4, q6 and q8 read a large part of -the table. pgColumnar is ahead of heap on all three, and ahead of TimescaleDB on -q6 and q8. q6 in particular is 1592 ms against TimescaleDB's 6956 ms, where the -earlier table recorded 81966 ms against 1771 ms. Column projection accounts for -most of that change: on this fixture it alone takes q6 from 45094 ms to 6491 ms. - -**The optional vectorization is not uniformly a win.** It helps q6, at 2292 ms -down to 1592 ms. It leaves q1, q3, q7 and q8 unchanged. It costs on q5, 16443 ms -up to 28494 ms, which is not yet explained and is tracked in -[issue #349](https://github.com/jdatcmd/pgcolumnar/issues/349). This is why those -settings default to off. - -**TimescaleDB leads on the host-filtered queries** for the reason given before: -its columnstore segments by `hostname`, so it reads one segment. pgColumnar -stores in load order, so its zone maps do not skip on `hostname`. Clustering a -pgColumnar table on the filter key addresses that, at the cost of the all-host -queries. +Method, stated so that it can be repeated. Read it before the numbers. + +**The data.** All four engines hold the same 100,000,000 rows. The rows come from +one generated TSBS file. Each engine loaded that same file. The row count was +verified in each engine before the measurement. + +**The indexes.** Every engine carries a `(hostname, time DESC)` btree. TimescaleDB +also carries the `(time DESC)` index that `create_hypertable` makes. Citus columnar +accepts a btree, so the four are comparable on this point. This is stated for a +reason. An index difference is invisible in a table of times. An engine without an +index that the others have looks worse, and the cause is not the storage format. + +**The settings.** `work_mem` is 256MB in every engine. This is a tuned value. The +PostgreSQL default is 4MB. The value is stated because it changes which engine the +table favors. A plan that sorts spills to disk at a low `work_mem`. The same plan +does not spill at a high one. A plan that spills is much slower and much less +stable. Both servers use the same tuned configuration. + +**The spill.** The temporary blocks that each query reads and writes are recorded +next to each time. A measurement of a plan that spills has a large spread between +runs. One earlier measurement of this kind moved by 1.43 times with every setting +held constant. That made a difference of 1.73 times impossible to interpret +afterwards (issue #358). Any arm that spills was measured again. + +**The parallelism.** Every query was measured twice. The first run sets +`max_parallel_workers_per_gather` to 0. The second sets it to 4. The serial number +shows what the storage format does. The parallel number shows what a user gets. +These are different. After the planner change in issue #362, the difference between +them can be a different plan and not only a different speed. + +**The protocol.** The engines are interleaved for each query. No engine is always +the one that is measured first on a cold cache. Each engine gets a cold execution +after a restart of its server and a drop of the operating system cache. Five warm +executions follow. The tables give the median of the five warm runs. The cold +numbers are given separately. + +**The servers.** pgColumnar, TimescaleDB and heap share one server. Citus needs a +different `shared_preload_libraries`, so it runs in a second server. The second +server is on the same host and uses the same tuned settings. A cold run restarts +the server that holds the engine which is measured. + +**TimescaleDB and parallel workers.** TimescaleDB cannot run the parallel arm on +this host. A query that gets a parallel plan fails: + +``` +ERROR: could not read blocks 0..0 in file "base/16384/3708169": read only 0 of 8192 bytes +CONTEXT: parallel worker +``` + +The earlier version of this page reported the same fault. This run confirms it and +adds detail. The failure is per query and not per engine. q1, q2 and q3 are served +by an index, take no parallel plan, and give the same result at both settings. q4 +and q5 take a parallel plan and fail. + +The table records those cells as a failure. It does not record the time that the +failed statement reported. A statement that fails still prints a time, and +that time is small. An error therefore looks like a very fast query if nothing +checks for it. + +Measured on 2026-08-04 against `main` at commit `eb5c7ef`, on the 100,000,000-row +fixture. The numbers are the median of five warm runs. + +Serial, `max_parallel_workers_per_gather = 0`: + +| query | shape | pgColumnar | TimescaleDB | heap | Citus | +| --- | --- | ---: | ---: | ---: | ---: | +| q1 | one host, 1 hour | 400 | 4 | 4 | 279 | +| q2 | one host, 12 hours | 2,253 | 5 | 12 | 1,896 | +| q3 | one host, 12 hours, 5 aggregates | 4,141 | 7 | 14 | 3,386 | +| q4 | all hosts, 12 hours, group by host | 8,517 | 5,465 | 9,810 | 7,398 | +| q5 | all hosts, 12 hours, 10 aggregates | 16,950 | 10,512 | 28,437 | 15,639 | +| q6 | full scan, one value filter | 11,199 | 1,734 | 14,692 | 8,123 | +| q7 | last point per host | 133,759 | 295 | 46 | 124,740 | +| q8 | top 20 by max | 15,639 | 7,583 | 19,857 | 14,440 | + +Parallel, `max_parallel_workers_per_gather = 4`: + +| query | pgColumnar | TimescaleDB | heap | Citus | +| --- | ---: | ---: | ---: | ---: | +| q1 | 73 | 4 | 4 | 275 | +| q2 | 494 | 5 | 12 | 1,954 | +| q3 | 861 | 6 | 13 | 3,344 | +| q4 | 8,128 | fails | 9,936 | 7,580 | +| q5 | **11,908** | fails | 26,963 | 17,454 | +| q6 | **2,324** | fails | 3,374 | 8,394 | +| q7 | 44,058 | 302 | 47 | 127,796 | +| q8 | 15,958 | fails | 20,420 | 14,635 | + +**Spill.** Most cells use no temporary disk at `work_mem = 256MB`. Five do, and they +are all in the two heaviest shapes: + +| cell | temp blocks read / written | node | +| --- | ---: | --- | +| q7, pgColumnar, serial | 1,022,514 / 1,022,568 | Sort | +| q7, Citus, serial | 1,022,514 / 1,022,568 | Sort | +| q7, Citus, parallel | 1,022,514 / 1,022,568 | Sort | +| q7, pgColumnar, parallel | 1,533,774 / 1,533,849 | Gather Merge | +| q5, pgColumnar, parallel | 721,569 / 721,584 | GroupAggregate | + +A plan that spills can be unstable between runs, which is why each cell is the median +of five. On this host the spilling cells were **not** the unstable ones. Their spread +between the fastest and slowest of the five runs is 1.03 to 1.04 times. The widest +spread in either table is q5 on heap in serial, at **1.18 times**, and that cell does +not spill at all. Every other cell is inside 1.05 times. + +Do not read that as "spill does not matter". It matters at a small `work_mem`, where +the same shape has been measured swinging 1.43 times with every setting held constant. +It says that at this setting, on this host, the sort had enough memory for the spill to +be sequential and cheap. + +**Parallel workers are what pgColumnar gains most from.** The scan divides cleanly +across workers. q1, q2 and q3 improve by about five times. q5 and q6 move from behind +heap to ahead of it. The serial table is a measure of the storage format. The +parallel table is closer to what an installation gets. + +**TimescaleDB leads on every query that filters one host.** Its columnstore segments +by `hostname`, so it reads one segment and not the table. pgColumnar stores rows in +load order, so its zone maps do not skip on `hostname`. A pgColumnar table that is +clustered on the filter key closes that gap, at the cost of the queries that read all +hosts. + +**pgColumnar leads the wide aggregate shapes with workers.** q5 reads ten metrics for +all hosts, and q6 scans the table with one filter. pgColumnar is first on both, and +q6 is 2,324 ms against Citus at 8,394 ms. + +**q7 is a regression and not a property of the format.** The planner declines an index +scan that this query wants. The cost model charges the index path for the rows the +path returns, and not for the rows the query reads. A `DISTINCT ON` reads one row +per host. The same query runs in **769 ms** with +`pgcolumnar.enable_index_fetch_penalty = off`, against 44,058 ms with the default. +This is [issue #376](https://github.com/jdatcmd/pgcolumnar/issues/376). Do not read +the q7 row as a measure of columnar storage. Citus is slow on this shape for its own +reasons, at 124,740 ms serial. #### Queries From 6659e933758dc12d14c07e16c9222b6df1296984 Mon Sep 17 00:00:00 2001 From: "Joshua (D) Drake" Date: Tue, 4 Aug 2026 07:58:52 -0600 Subject: [PATCH 2/5] docs: say which installations meet the q7 planner defect (#376) Per jdatcmd on #376: publish the q7 row with the skip scan available and say what it depends on, rather than suppressing it. The table describes what a user gets, and which extensions are loaded is part of that, the same argument that put the index set per engine in the method. The 57x needs a consumer that reads part of an index path and then prices itself again. PostgreSQL has no such node, so a plain install takes the other plan and the penalty is correct for it. TimescaleDB supplies a skip scan, which is why this host meets it. Co-Authored-By: Claude Opus 5 (1M context) --- docs/benchmarks.md | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 6988c8d..7bfe4ce 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -521,14 +521,22 @@ hosts. all hosts, and q6 scans the table with one filter. pgColumnar is first on both, and q6 is 2,324 ms against Citus at 8,394 ms. -**q7 is a regression and not a property of the format.** The planner declines an index -scan that this query wants. The cost model charges the index path for the rows the -path returns, and not for the rows the query reads. A `DISTINCT ON` reads one row -per host. The same query runs in **769 ms** with +**q7 measures a planner defect and not the storage format.** The planner declines an +index scan that this query wants. The cost model charges the index path for the rows +the path returns, and not for the rows the query reads. A `DISTINCT ON` reads one row +per host. This is [issue #376](https://github.com/jdatcmd/pgcolumnar/issues/376). + +The row above is what this host gives, and this host loads TimescaleDB. That matters, +because TimescaleDB supplies a skip scan, and a skip scan is the consumer that reads +one row per host. The same query gives **769 ms** with `pgcolumnar.enable_index_fetch_penalty = off`, against 44,058 ms with the default. -This is [issue #376](https://github.com/jdatcmd/pgcolumnar/issues/376). Do not read -the q7 row as a measure of columnar storage. Citus is slow on this shape for its own -reasons, at 124,740 ms serial. + +An installation without such an extension does not meet this. PostgreSQL has no node that reads part +of an index path and then prices itself again. Such an installation takes the other +plan, and the penalty is correct for it. Which extensions are loaded is part of what a +user gets, which is why the row is given rather than removed. + +Citus is slow on this shape for its own reasons, at 124,740 ms serial. #### Queries From b8edc5f02334aaaddbdfade05256d7f627b8c778 Mon Sep 17 00:00:00 2001 From: "Joshua (D) Drake" Date: Tue, 4 Aug 2026 09:46:41 -0600 Subject: [PATCH 3/5] docs: re-measure the cross-engine tables on merged main (aeb7882) The previous tables were eb5c7ef, which predates two merges that move rows: #375 changed how the grouped node is priced, and #378 bounded the index-fetch penalty. Patching one row would put two builds in one table, so the whole run was repeated: 384 timings and 64 spill cells on aeb7882. q7 is the change. It was 133,759 ms in serial and is 767 ms. The cost model charged an index path for the rows it returns rather than the rows the query reads, so a DISTINCT ON that reads one row per host was priced out of the index. That was #376, found by this benchmark pass and fixed in #378. The narrative now records it as fixed rather than as a live defect with a workaround. Two spilling cells also disappeared with it, because a sort of the whole table became a skip scan and a skip scan sorts nothing. Three cells still spill and are listed. Everything else moved within noise: no cell in either table has a spread wider than 1.04 times between its fastest and slowest of five warm runs. Co-Authored-By: Claude Opus 5 (1M context) --- docs/benchmarks.md | 166 ++++++++++++++------------------------------- 1 file changed, 52 insertions(+), 114 deletions(-) diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 7bfe4ce..6dc2ecf 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -36,8 +36,8 @@ The read stream section is not re-measured. It needs a PostgreSQL 18 build with `--with-liburing`, and no such build exists on this machine. Its numbers are from the earlier run and say so. The Cross-engine comparison and the parallel sections are a separate, larger run. -It ran on the bench host, at up to 100,000,000 rows, dated 2026-08-04, at the same -commit `eb5c7ef`. Each of those sections states its own method. +It ran on the bench host, at up to 100,000,000 rows, dated 2026-08-04, at commit +`aeb7882`. Each of those sections states its own method. They show the shape of the trade and not a precise score. The dataset is synthetic. It mixes column shapes that suit different encodings, and it does this deliberately. A table of fully random values will therefore look worse, and a @@ -396,147 +396,85 @@ pgColumnar is the smallest of the four. ### Cross-engine query latency -Method, stated so that it can be repeated. Read it before the numbers. - -**The data.** All four engines hold the same 100,000,000 rows. The rows come from -one generated TSBS file. Each engine loaded that same file. The row count was -verified in each engine before the measurement. - -**The indexes.** Every engine carries a `(hostname, time DESC)` btree. TimescaleDB -also carries the `(time DESC)` index that `create_hypertable` makes. Citus columnar -accepts a btree, so the four are comparable on this point. This is stated for a -reason. An index difference is invisible in a table of times. An engine without an -index that the others have looks worse, and the cause is not the storage format. - -**The settings.** `work_mem` is 256MB in every engine. This is a tuned value. The -PostgreSQL default is 4MB. The value is stated because it changes which engine the -table favors. A plan that sorts spills to disk at a low `work_mem`. The same plan -does not spill at a high one. A plan that spills is much slower and much less -stable. Both servers use the same tuned configuration. - -**The spill.** The temporary blocks that each query reads and writes are recorded -next to each time. A measurement of a plan that spills has a large spread between -runs. One earlier measurement of this kind moved by 1.43 times with every setting -held constant. That made a difference of 1.73 times impossible to interpret -afterwards (issue #358). Any arm that spills was measured again. - -**The parallelism.** Every query was measured twice. The first run sets -`max_parallel_workers_per_gather` to 0. The second sets it to 4. The serial number -shows what the storage format does. The parallel number shows what a user gets. -These are different. After the planner change in issue #362, the difference between -them can be a different plan and not only a different speed. - -**The protocol.** The engines are interleaved for each query. No engine is always -the one that is measured first on a cold cache. Each engine gets a cold execution -after a restart of its server and a drop of the operating system cache. Five warm -executions follow. The tables give the median of the five warm runs. The cold -numbers are given separately. - -**The servers.** pgColumnar, TimescaleDB and heap share one server. Citus needs a -different `shared_preload_libraries`, so it runs in a second server. The second -server is on the same host and uses the same tuned settings. A cold run restarts -the server that holds the engine which is measured. - -**TimescaleDB and parallel workers.** TimescaleDB cannot run the parallel arm on -this host. A query that gets a parallel plan fails: - -``` -ERROR: could not read blocks 0..0 in file "base/16384/3708169": read only 0 of 8192 bytes -CONTEXT: parallel worker -``` - -The earlier version of this page reported the same fault. This run confirms it and -adds detail. The failure is per query and not per engine. q1, q2 and q3 are served -by an index, take no parallel plan, and give the same result at both settings. q4 -and q5 take a parallel plan and fail. - -The table records those cells as a failure. It does not record the time that the -failed statement reported. A statement that fails still prints a time, and -that time is small. An error therefore looks like a very fast query if nothing -checks for it. - -Measured on 2026-08-04 against `main` at commit `eb5c7ef`, on the 100,000,000-row +Measured on 2026-08-04 against `main` at commit `aeb7882`, on the 100,000,000-row fixture. The numbers are the median of five warm runs. Serial, `max_parallel_workers_per_gather = 0`: | query | shape | pgColumnar | TimescaleDB | heap | Citus | | --- | --- | ---: | ---: | ---: | ---: | -| q1 | one host, 1 hour | 400 | 4 | 4 | 279 | -| q2 | one host, 12 hours | 2,253 | 5 | 12 | 1,896 | -| q3 | one host, 12 hours, 5 aggregates | 4,141 | 7 | 14 | 3,386 | -| q4 | all hosts, 12 hours, group by host | 8,517 | 5,465 | 9,810 | 7,398 | -| q5 | all hosts, 12 hours, 10 aggregates | 16,950 | 10,512 | 28,437 | 15,639 | -| q6 | full scan, one value filter | 11,199 | 1,734 | 14,692 | 8,123 | -| q7 | last point per host | 133,759 | 295 | 46 | 124,740 | -| q8 | top 20 by max | 15,639 | 7,583 | 19,857 | 14,440 | +| q1 | one host, 1 hour | 403 | 4 | 4 | 272 | +| q2 | one host, 12 hours | 2,211 | 5 | 11 | 1,865 | +| q3 | one host, 12 hours, 5 aggregates | 4,071 | 6 | 13 | 3,365 | +| q4 | all hosts, 12 hours, group by host | 8,210 | 5,119 | 9,713 | 7,364 | +| q5 | all hosts, 12 hours, 10 aggregates | 16,706 | 10,341 | 25,410 | 15,836 | +| q6 | full scan, one value filter | 11,220 | 1,737 | 14,525 | 8,124 | +| q7 | last point per host | 767 | 289 | 46 | 124,537 | +| q8 | top 20 by max | 15,757 | 7,554 | 20,507 | 14,578 | Parallel, `max_parallel_workers_per_gather = 4`: | query | pgColumnar | TimescaleDB | heap | Citus | | --- | ---: | ---: | ---: | ---: | -| q1 | 73 | 4 | 4 | 275 | -| q2 | 494 | 5 | 12 | 1,954 | -| q3 | 861 | 6 | 13 | 3,344 | -| q4 | 8,128 | fails | 9,936 | 7,580 | -| q5 | **11,908** | fails | 26,963 | 17,454 | -| q6 | **2,324** | fails | 3,374 | 8,394 | -| q7 | 44,058 | 302 | 47 | 127,796 | -| q8 | 15,958 | fails | 20,420 | 14,635 | - -**Spill.** Most cells use no temporary disk at `work_mem = 256MB`. Five do, and they -are all in the two heaviest shapes: +| q1 | 73 | 4 | 4 | 277 | +| q2 | 500 | 5 | 12 | 1,897 | +| q3 | 875 | 6 | 14 | 3,366 | +| q4 | 8,256 | fails | 9,798 | 7,543 | +| q5 | **11,968** | fails | 28,043 | 16,658 | +| q6 | **2,294** | fails | 3,261 | 8,242 | +| q7 | 766 | 295 | 47 | 125,423 | +| q8 | 15,685 | fails | 20,020 | 14,928 | + +Every cell is the median of five warm runs. The widest spread between the fastest and +the slowest of those five, anywhere in either table, is **1.04 times**. + +**Spill.** Most cells use no temporary disk at `work_mem = 256MB`. Three do: | cell | temp blocks read / written | node | | --- | ---: | --- | -| q7, pgColumnar, serial | 1,022,514 / 1,022,568 | Sort | | q7, Citus, serial | 1,022,514 / 1,022,568 | Sort | | q7, Citus, parallel | 1,022,514 / 1,022,568 | Sort | -| q7, pgColumnar, parallel | 1,533,774 / 1,533,849 | Gather Merge | | q5, pgColumnar, parallel | 721,569 / 721,584 | GroupAggregate | +The earlier record of this page had five, and two of them were q7 on pgColumnar. Those +are gone because the plan changed. A sort of the whole table became a skip scan over +the index, and a skip scan sorts nothing. + A plan that spills can be unstable between runs, which is why each cell is the median -of five. On this host the spilling cells were **not** the unstable ones. Their spread -between the fastest and slowest of the five runs is 1.03 to 1.04 times. The widest -spread in either table is q5 on heap in serial, at **1.18 times**, and that cell does -not spill at all. Every other cell is inside 1.05 times. +of five. On this host the spilling cells are not the unstable ones. No cell in either +table has a spread wider than 1.04 times between its fastest and slowest run. Do not read that as "spill does not matter". It matters at a small `work_mem`, where -the same shape has been measured swinging 1.43 times with every setting held constant. +the same shape has been measured to swing 1.43 times with every setting held constant. It says that at this setting, on this host, the sort had enough memory for the spill to be sequential and cheap. **Parallel workers are what pgColumnar gains most from.** The scan divides cleanly across workers. q1, q2 and q3 improve by about five times. q5 and q6 move from behind -heap to ahead of it. The serial table is a measure of the storage format. The -parallel table is closer to what an installation gets. +heap to ahead of it. The serial table is a measure of the storage format. The parallel +table is closer to what an installation gets. -**TimescaleDB leads on every query that filters one host.** Its columnstore segments -by `hostname`, so it reads one segment and not the table. pgColumnar stores rows in -load order, so its zone maps do not skip on `hostname`. A pgColumnar table that is -clustered on the filter key closes that gap, at the cost of the queries that read all -hosts. +**TimescaleDB leads every query that filters one host.** Its columnstore segments by +`hostname`, so it reads one segment and not the table. pgColumnar stores rows in load +order, so its zone maps do not skip on `hostname`. A pgColumnar table that is clustered +on the filter key closes that gap, at the cost of the queries that read all hosts. **pgColumnar leads the wide aggregate shapes with workers.** q5 reads ten metrics for -all hosts, and q6 scans the table with one filter. pgColumnar is first on both, and -q6 is 2,324 ms against Citus at 8,394 ms. - -**q7 measures a planner defect and not the storage format.** The planner declines an -index scan that this query wants. The cost model charges the index path for the rows -the path returns, and not for the rows the query reads. A `DISTINCT ON` reads one row -per host. This is [issue #376](https://github.com/jdatcmd/pgcolumnar/issues/376). - -The row above is what this host gives, and this host loads TimescaleDB. That matters, -because TimescaleDB supplies a skip scan, and a skip scan is the consumer that reads -one row per host. The same query gives **769 ms** with -`pgcolumnar.enable_index_fetch_penalty = off`, against 44,058 ms with the default. - -An installation without such an extension does not meet this. PostgreSQL has no node that reads part -of an index path and then prices itself again. Such an installation takes the other -plan, and the penalty is correct for it. Which extensions are loaded is part of what a -user gets, which is why the row is given rather than removed. - -Citus is slow on this shape for its own reasons, at 124,740 ms serial. +all hosts, and q6 scans the table with one filter. pgColumnar is first on both, and q6 +is 2,294 ms against Citus at 8,242 ms. + +**q7 was a planner defect and is now fixed.** The earlier record of this page reported +133,759 ms for q7 in serial. The cost model charged an index path for the rows the +path returns, and not for the rows the query reads. A `DISTINCT ON` reads one row per +host. The model therefore priced the index path far above every +alternative. No consumer could recover it, not even one that reads 3,998 rows of +100,000,000. That is +[issue #376](https://github.com/jdatcmd/pgcolumnar/issues/376), found by this +benchmark pass and fixed in +[#378](https://github.com/jdatcmd/pgcolumnar/pull/378), which bounds the penalty at a +multiple of one scan. The query now takes 767 ms. + +Citus is slow on this shape for its own reasons, at 124,537 ms. #### Queries From 99c434ebfa24dd5f8f12eca4d198a36ec3e2b2ab Mon Sep 17 00:00:00 2001 From: "Joshua (D) Drake" Date: Tue, 4 Aug 2026 09:49:21 -0600 Subject: [PATCH 4/5] docs: re-measure the single-engine sections on the same commit (aeb7882) The single-engine sections were eb5c7ef and the cross-engine ones are now aeb7882, which put two commits in one document. #378 in particular touches the index-fetch penalty, and this half measures a point lookup and an index-only scan, so it had to be re-run rather than assumed unaffected. It was unaffected in direction. Every number moved down, heap included, because the box is quieter than it was during the 100M run: heap count(*) 172 ms to 136, heap filtered agg 206 to 154. The ratios hold. Projection is 20.6x against 21.2x, index-only 109x against 106x, and the point lookup is 11.90 ms against 15.64. Storage is byte for byte the same, which is the expected result for a change that touches only the planner. The whole document is now one commit. Co-Authored-By: Claude Opus 5 (1M context) --- docs/benchmarks.md | 50 +++++++++++++++++++++++----------------------- 1 file changed, 25 insertions(+), 25 deletions(-) diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 6dc2ecf..15ede06 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -23,7 +23,7 @@ These environment variables control the harnesses: `PATH`. The numbers below come from one full run of `bench/run_bench.sh` on 2026-08-04, at -commit `eb5c7ef`. The conditions were PostgreSQL 18.4 non-assert, 6,000,000 rows, +commit `aeb7882`, which is the same commit as the cross-engine run below. The conditions were PostgreSQL 18.4 non-assert, 6,000,000 rows, an 8-column table, the median of 5 repetitions, 8 cores and 12 GB of memory. The previous record of this section was 2026-07-27 at commit `7a9c9f7`, on @@ -81,11 +81,11 @@ Heap versus columnar (zstd), median milliseconds: | query | heap | columnar | heap / columnar | | --- | --- | --- | --- | -| count(*) full table | 172.13 | 0.02 | 8607 | -| sum/avg over one int column | 258.99 | 0.41 | 632 | -| filtered agg, min/max-skippable range | 205.78 | 11.19 | 18.4 | -| projection: 3 of 8 cols, 1% filter | 229.85 | 10.85 | 21.2 | -| point lookup by indexed id | 0.01 | 15.64 | 0.00 | +| count(*) full table | 135.66 | 0.01 | 13566 | +| sum/avg over one int column | 190.04 | 0.28 | 679 | +| filtered agg, min/max-skippable range | 154.05 | 7.96 | 19.4 | +| projection: 3 of 8 cols, 1% filter | 146.91 | 7.15 | 20.6 | +| point lookup by indexed id | 0.01 | 11.90 | 0.00 | `count(*)` and the ungrouped aggregates are answered from row-group metadata without decoding column data, which is why they are microseconds rather than @@ -95,7 +95,7 @@ The point lookup was a regression when the previous version of this page was written, at 1251.88 ms, and it was reported as [issue #171](https://github.com/jdatcmd/pgcolumnar/issues/171). That issue is closed. The planner chose a full columnar scan for a point lookup once statistics -existed. It now keeps the index, and the same query takes 15.64 ms. +existed. It now keeps the index, and the same query takes 11.90 ms. The filtered aggregate and the projection query also changed by about 8 times. Column projection is the cause. A columnar scan reads only the columns that a @@ -136,10 +136,10 @@ single run for the delete: | operation | heap | columnar | columnar / heap | | --- | --- | --- | --- | -| UPDATE single row by id | 0.01 ms | 71.73 ms | 7173 | -| UPDATE 1000 rows, ids in row order | 2.38 ms | 84.47 ms | 35 | -| UPDATE 1000 rows, ids scattered | 28.51 ms | 78.98 ms | 2.8 | -| DELETE 1000 rows by id range | 0.3 ms | 73.6 ms | 245 | +| UPDATE single row by id | 0.01 ms | 62.47 ms | 6247 | +| UPDATE 1000 rows, ids in row order | 2.04 ms | 74.55 ms | 37 | +| UPDATE 1000 rows, ids scattered | 24.70 ms | 69.97 ms | 2.8 | +| DELETE 1000 rows by id range | 0.3 ms | 64.7 ms | 216 | The columnar cost is close to the same for one row and for 1000. A write to a columnar table marks the old row and appends a new one, and the row group is the @@ -154,7 +154,7 @@ update is the shape that columnar storage is worst at, and the table says so. single-row update. That figure could not be reproduced on the current machine with the current build or with the build it was taken at. The two builds were compared directly, on one machine and one major version, with an equivalent single-row update. Commit -`7a9c9f7` gives 162 ms. Commit `eb5c7ef` gives 19 ms. The mutation path is faster than it was, and the earlier 0.22 ms is not a +`7a9c9f7` gives 162 ms. The current one gives 19 ms. The mutation path is faster than it was, and the earlier 0.22 ms is not a baseline that this run failed to meet. The delete figure was the weakest number in this document, at 1509 ms. At that @@ -169,18 +169,18 @@ Vectorization on versus off (columnar zstd, median ms): | query | on | off | speedup | | --- | --- | --- | --- | -| sum/avg over int | 0.45 | 270.29 | 601 | -| filtered agg (range) | 8.59 | 8.41 | 0.98 | +| sum/avg over int | 0.29 | 179.36 | 618 | +| filtered agg (range) | 5.70 | 5.59 | 0.98 | -The "off" column is much faster than in the previous record, at 270 ms against -1392 ms. Column projection (issue #339) is the reason. The path that does not +The "off" column is much faster than in the record before these two runs, at 179 ms +against 1392 ms. Column projection (issue #339) is the reason. The path that does not vectorize now also reads fewer columns. Index-only scan on versus off (covering range count, median ms): | query | on | off | speedup | | --- | --- | --- | --- | -| covering count, id range (~2%) | 4.97 | 526.69 | 106 | +| covering count, id range (~2%) | 3.72 | 404.02 | 109 | The "off" column is the fetch-by-row path with no other work. It therefore isolates the cost of that path and the effect of #143. This shape was 200.9 s @@ -191,18 +191,18 @@ Projection scan on versus off (covering scan on a scattered sort key, median ms) | query | on | off | speedup | | --- | --- | --- | --- | -| sortk, val where sortk in ~0.1% range | 132.47 | 209.48 | 1.58 | +| sortk, val where sortk in ~0.1% range | 96.72 | 162.88 | 1.68 | Sorted storage (`pgcolumnar.vacuum_sorted`), narrow range scan on a key not correlated with insert order, median ms: | state | ms | | --- | --- | -| before vacuum_sorted | 255.16 | -| after vacuum_sorted | 1.61 | +| before vacuum_sorted | 160.68 | +| after vacuum_sorted | 1.24 | Compression `none` against `zstd`, for the columnar table only: 40 MB against -5.95 MB. The scan latency does not change, at 0.41 ms against 0.40 ms. The +5.95 MB. The scan latency does not change, at 0.28 ms against 0.28 ms. The encoded stream is already small, and the aggregates do not read it. ## Parallel bulk ingest @@ -257,15 +257,15 @@ Export, 6,000,000 rows, 5 columns: | format | ms | file size | M rows/s | | --- | --- | --- | --- | -| arrow | 704.0 | 186 MB | 8.5 | -| parquet | 763.0 | 186 MB | 7.9 | +| arrow | 526.7 | 186 MB | 11.4 | +| parquet | 579.7 | 186 MB | 10.4 | Import, 6,000,000 rows, 5 columns: | format | ms | M rows/s | | --- | --- | --- | -| arrow | 4037.3 | 1.5 | -| parquet | 4468.1 | 1.3 | +| arrow | 3313.7 | 1.8 | +| parquet | 3353.6 | 1.8 | Import is about 18x slower than export, and the reason is not the import code. A separate measurement shows this. `import_arrow` costs 12,150 ms. An From 4c6f37478e2929385f42e16e9917dd751997d1af Mon Sep 17 00:00:00 2001 From: "Joshua (D) Drake" Date: Tue, 4 Aug 2026 11:32:01 -0600 Subject: [PATCH 5/5] docs: state the index set per engine in the cross-engine setup list The setup list named heap's indexes and not pgColumnar's, so it read as though pgColumnar had none. pgColumnar's q7 number depends entirely on a skip scan over (hostname, time DESC), which makes that the wrong thing to leave to inference. Queried rather than recalled. All four engines carry one btree on (hostname, time DESC). TimescaleDB also carries the (time DESC) index that create_hypertable makes, and 53 chunk indexes below the two. This is the check raised on #367, where cpu_heap carried the index and cpu_pgc did not. The published table is the one place it should not be missing. Co-Authored-By: Claude Opus 5 (1M context) --- docs/benchmarks.md | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 15ede06..f60ca0b 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -374,10 +374,20 @@ columns. The data is loaded byte for byte the same way into each engine. The bench host has 16 vCPU and 62 GB. Each engine uses the configuration its own users would choose: -- pgColumnar: columnar scan, storage in load order, which is time ascending. -- TimescaleDB: compressed columnstore, segmented by `hostname`, ordered by `time` descending. -- heap: sequential scan with the secondary indexes a heap user would build. -- Citus: single node, columnar storage. +- pgColumnar: columnar scan, storage in load order, which is time ascending. One + btree on `(hostname, time DESC)`. +- TimescaleDB: compressed columnstore, segmented by `hostname`, ordered by `time` + descending. One btree on `(hostname, time DESC)`, and the `(time DESC)` index that + `create_hypertable` makes, which gives 53 chunk indexes below them. +- heap: sequential scan with the secondary indexes a heap user would build. One btree + on `(hostname, time DESC)`. +- Citus: single node, columnar storage. One btree on `(hostname, time DESC)`. + +All four therefore carry the same `(hostname, time DESC)` index, and TimescaleDB +carries one more. The index set is given per engine because it decides some of these +rows. q7 on pgColumnar is 767 ms because a skip scan reads that index. The +same query on the un-indexed table in the parallel-scan section below full-scans and +sorts. The full method follows the storage table.