Skip to content

Re-measure the cross-engine benchmarks on current main (#367) #377

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jdatcmd merged 5 commits into from
Aug 4, 2026
Merged

Re-measure the cross-engine benchmarks on current main (#367) #377

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
df5a574
docs: re-measure the cross-engine benchmarks on current main
Aug 4, 2026
6659e93
docs: say which installations meet the q7 planner defect (#376)
Aug 4, 2026
b8edc5f
docs: re-measure the cross-engine tables on merged main (aeb7882)
Aug 4, 2026
99c434e
docs: re-measure the single-engine sections on the same commit (aeb7882)
Aug 4, 2026
4c6f374
docs: state the index set per engine in the cross-engine setup list
Aug 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
249 changes: 127 additions & 122 deletions docs/benchmarks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ These environment variables control the harnesses:
`PATH`.

The numbers below come from one full run of `bench/run_bench.sh` on 2026-08-04, at
commit `eb5c7ef`. The conditions were PostgreSQL 18.4 non-assert, 6,000,000 rows,
commit `aeb7882`, which is the same commit as the cross-engine run below. The conditions were PostgreSQL 18.4 non-assert, 6,000,000 rows,
an 8-column table, the median of 5 repetitions, 8 cores and 12 GB of memory.

The previous record of this section was 2026-07-27 at commit `7a9c9f7`, on
Expand All @@ -36,8 +36,8 @@ The read stream section is not re-measured. It needs a PostgreSQL 18 build with
`--with-liburing`, and no such build exists on this machine. Its numbers are from
the earlier run and say so.
The Cross-engine comparison and the parallel sections are a separate, larger run.
It ran on the bench host, at up to 100,000,000 rows, dated 2026-08-02. Each of
those sections states its own method.
It ran on the bench host, at up to 100,000,000 rows, dated 2026-08-04, at commit
`aeb7882`. Each of those sections states its own method.
They show the shape of the trade and not a precise score. The dataset is
synthetic. It mixes column shapes that suit different encodings, and it does this
deliberately. A table of fully random values will therefore look worse, and a
Expand Down Expand Up @@ -81,11 +81,11 @@ Heap versus columnar (zstd), median milliseconds:

| query | heap | columnar | heap / columnar |
| --- | --- | --- | --- |
| count(*) full table | 172.13 | 0.02 | 8607 |
| sum/avg over one int column | 258.99 | 0.41 | 632 |
| filtered agg, min/max-skippable range | 205.78 | 11.19 | 18.4 |
| projection: 3 of 8 cols, 1% filter | 229.85 | 10.85 | 21.2 |
| point lookup by indexed id | 0.01 | 15.64 | 0.00 |
| count(*) full table | 135.66 | 0.01 | 13566 |
| sum/avg over one int column | 190.04 | 0.28 | 679 |
| filtered agg, min/max-skippable range | 154.05 | 7.96 | 19.4 |
| projection: 3 of 8 cols, 1% filter | 146.91 | 7.15 | 20.6 |
| point lookup by indexed id | 0.01 | 11.90 | 0.00 |

`count(*)` and the ungrouped aggregates are answered from row-group metadata
without decoding column data, which is why they are microseconds rather than
Expand All @@ -95,7 +95,7 @@ The point lookup was a regression when the previous version of this page was
written, at 1251.88 ms, and it was reported as
[issue #171](https://github.com/jdatcmd/pgcolumnar/issues/171). That issue is
closed. The planner chose a full columnar scan for a point lookup once statistics
existed. It now keeps the index, and the same query takes 15.64 ms.
existed. It now keeps the index, and the same query takes 11.90 ms.

The filtered aggregate and the projection query also changed by about 8 times.
Column projection is the cause. A columnar scan reads only the columns that a
Expand Down Expand Up @@ -136,10 +136,10 @@ single run for the delete:

| operation | heap | columnar | columnar / heap |
| --- | --- | --- | --- |
| UPDATE single row by id | 0.01 ms | 71.73 ms | 7173 |
| UPDATE 1000 rows, ids in row order | 2.38 ms | 84.47 ms | 35 |
| UPDATE 1000 rows, ids scattered | 28.51 ms | 78.98 ms | 2.8 |
| DELETE 1000 rows by id range | 0.3 ms | 73.6 ms | 245 |
| UPDATE single row by id | 0.01 ms | 62.47 ms | 6247 |
| UPDATE 1000 rows, ids in row order | 2.04 ms | 74.55 ms | 37 |
| UPDATE 1000 rows, ids scattered | 24.70 ms | 69.97 ms | 2.8 |
| DELETE 1000 rows by id range | 0.3 ms | 64.7 ms | 216 |

The columnar cost is close to the same for one row and for 1000. A write to a
columnar table marks the old row and appends a new one, and the row group is the
Expand All @@ -154,7 +154,7 @@ update is the shape that columnar storage is worst at, and the table says so.
single-row update. That figure could not be reproduced on the current machine
with the current build or with the build it was taken at. The two builds were compared directly, on
one machine and one major version, with an equivalent single-row update. Commit
`7a9c9f7` gives 162 ms. Commit `eb5c7ef` gives 19 ms. The mutation path is faster than it was, and the earlier 0.22 ms is not a
`7a9c9f7` gives 162 ms. The current one gives 19 ms. The mutation path is faster than it was, and the earlier 0.22 ms is not a
baseline that this run failed to meet.

The delete figure was the weakest number in this document, at 1509 ms. At that
Expand All @@ -169,18 +169,18 @@ Vectorization on versus off (columnar zstd, median ms):

| query | on | off | speedup |
| --- | --- | --- | --- |
| sum/avg over int | 0.45 | 270.29 | 601 |
| filtered agg (range) | 8.59 | 8.41 | 0.98 |
| sum/avg over int | 0.29 | 179.36 | 618 |
| filtered agg (range) | 5.70 | 5.59 | 0.98 |

The "off" column is much faster than in the previous record, at 270 ms against
1392 ms. Column projection (issue #339) is the reason. The path that does not
The "off" column is much faster than in the record before these two runs, at 179 ms
against 1392 ms. Column projection (issue #339) is the reason. The path that does not
vectorize now also reads fewer columns.

Index-only scan on versus off (covering range count, median ms):

| query | on | off | speedup |
| --- | --- | --- | --- |
| covering count, id range (~2%) | 4.97 | 526.69 | 106 |
| covering count, id range (~2%) | 3.72 | 404.02 | 109 |

The "off" column is the fetch-by-row path with no other work. It therefore
isolates the cost of that path and the effect of #143. This shape was 200.9 s
Expand All @@ -191,18 +191,18 @@ Projection scan on versus off (covering scan on a scattered sort key, median ms)

| query | on | off | speedup |
| --- | --- | --- | --- |
| sortk, val where sortk in ~0.1% range | 132.47 | 209.48 | 1.58 |
| sortk, val where sortk in ~0.1% range | 96.72 | 162.88 | 1.68 |

Sorted storage (`pgcolumnar.vacuum_sorted`), narrow range scan on a key not
correlated with insert order, median ms:

| state | ms |
| --- | --- |
| before vacuum_sorted | 255.16 |
| after vacuum_sorted | 1.61 |
| before vacuum_sorted | 160.68 |
| after vacuum_sorted | 1.24 |

Compression `none` against `zstd`, for the columnar table only: 40 MB against
5.95 MB. The scan latency does not change, at 0.41 ms against 0.40 ms. The
5.95 MB. The scan latency does not change, at 0.28 ms against 0.28 ms. The
encoded stream is already small, and the aggregates do not read it.

## Parallel bulk ingest
Expand Down Expand Up @@ -257,15 +257,15 @@ Export, 6,000,000 rows, 5 columns:

| format | ms | file size | M rows/s |
| --- | --- | --- | --- |
| arrow | 704.0 | 186 MB | 8.5 |
| parquet | 763.0 | 186 MB | 7.9 |
| arrow | 526.7 | 186 MB | 11.4 |
| parquet | 579.7 | 186 MB | 10.4 |

Import, 6,000,000 rows, 5 columns:

| format | ms | M rows/s |
| --- | --- | --- |
| arrow | 4037.3 | 1.5 |
| parquet | 4468.1 | 1.3 |
| arrow | 3313.7 | 1.8 |
| parquet | 3353.6 | 1.8 |

Import is about 18x slower than export, and the reason is not the import code.
A separate measurement shows this. `import_arrow` costs 12,150 ms. An
Expand Down Expand Up @@ -371,20 +371,25 @@ suite. This section is a separate, larger run. It compares pgColumnar with heap,
TimescaleDB, and Citus on the TSBS `cpu` workload, at 100,000,000 rows across 21
columns. The data is loaded byte for byte the same way into each engine.

Method: the bench host, 16 vCPU and 62 GB, PostgreSQL 18.4 non-assert, on
2026-08-02 at the current `main` plus the export feature. Each engine has the box
to itself. Latency is client-observed, one cold run after a restart with the
cache dropped, then the median of three warm runs, interleaved per query. Each
engine uses the configuration its own users would choose:
The bench host has 16 vCPU and 62 GB. Each engine uses the configuration its own
users would choose:

- pgColumnar: columnar scan, storage in load order, which is time ascending.
- TimescaleDB: compressed columnstore, segmented by `hostname`, ordered by `time` descending.
- heap: sequential scan with the secondary indexes a heap user would build.
- Citus: single node, columnar storage, sharded.
- pgColumnar: columnar scan, storage in load order, which is time ascending. One
btree on `(hostname, time DESC)`.
- TimescaleDB: compressed columnstore, segmented by `hostname`, ordered by `time`
descending. One btree on `(hostname, time DESC)`, and the `(time DESC)` index that
`create_hypertable` makes, which gives 53 chunk indexes below them.
- heap: sequential scan with the secondary indexes a heap user would build. One btree
on `(hostname, time DESC)`.
- Citus: single node, columnar storage. One btree on `(hostname, time DESC)`.

The query engines run serial (`max_parallel_workers_per_gather = 0`). That
isolates the storage and scan path. It is not pgColumnar at its ceiling. The
parallel scan below adds about four times on these same shapes.
All four therefore carry the same `(hostname, time DESC)` index, and TimescaleDB
carries one more. The index set is given per engine because it decides some of these
rows. q7 on pgColumnar is 767 ms because a skip scan reads that index. The
same query on the un-indexed table in the parallel-scan section below full-scans and
sorts.

The full method follows the storage table.

### Cross-engine storage

Expand All @@ -393,93 +398,93 @@ Total relation size, including indexes, for the 100,000,000 rows:
| engine | size | smaller than heap |
| --- | --- | --- |
| heap | 22 GB | 1.0x |
| TimescaleDB columnstore | 7.8 GB | 2.8x |
| pgColumnar (zstd) | 6.4 GB | 3.4x |
| pgColumnar (zstd) | 6,590 MB | 3.4x |
| TimescaleDB columnstore | 7,975 MB | 2.8x |
| Citus columnar | 8,147 MB | 2.8x |

pgColumnar is the smallest of the four. The encoding layer does most of this, and
zstd compounds it, as the [Storage](#storage) section shows in detail.
pgColumnar is the smallest of the four.

### Cross-engine query latency

Measured on 2026-08-03 against `main` at commit `00290d7`, on the same
100,000,000-row fixture. This is a new run, not a correction of the previous
table row by row. The earlier figures were taken before column projection landed (issue #338,
fixed in #339). Every query then read and decoded every column of the table,
whatever it referenced. The queries behind those figures were also not recorded.
The two runs cannot be compared line by line.

Method for this run, stated so it can be repeated:

- All three engines hold the same 100,000,000 rows, verified before measuring.
- `cpu_pgc` and `cpu_heap` each carry a `(hostname, time DESC)` btree.
TimescaleDB carries the equivalent index on its chunks.
- pgColumnar and heap run with `max_parallel_workers_per_gather = 4`.
TimescaleDB runs serial, because its parallel path fails on this host with
`could not read blocks 0..0`. That fault is not caused by pgColumnar: it
persists with every pgColumnar planner hook disabled, and the chunks use the
heap access method.
- pgColumnar figures are given twice: with default settings, and with
`enable_ungrouped_vector_agg`, `enable_parallel_vector_agg` and
`enable_group_vectorization` all on. Those three default to off.
- Warm, `EXPLAIN (ANALYZE)` execution time, after one warm-up run.
- Citus was not re-measured and is omitted rather than carried forward from a
run whose configuration is unknown.

Milliseconds:

| query | shape | pgColumnar | pgColumnar, all options on | heap | TimescaleDB (serial) |
| --- | --- | --- | --- | --- | --- |
| q1 | one host, 1 hour | 10335 | 10253 | 6 | 0 |
| q2 | one host, 12 hours | 123546 | 119534 | 8 | 1 |
| q3 | one host, 12 hours, 5 aggregates | 161972 | 156701 | 8 | 2 |
| q4 | all hosts, 12 hours, group by host | 7322 | 7863 | 13912 | 4515 |
| q5 | all hosts, 12 hours, 10 aggregates | 16443 | 28494 | 17378 | 9576 |
| q6 | full scan, one value filter | 2292 | 1592 | 9083 | 6956 |
| q7 | last point per host | 368 | 365 | 31 | 196 |
| q8 | top 20 by max | 5153 | 5124 | 8715 | 10171 |

The SQL is given at the end of this section. The previous table did not record
it, which is the main reason its numbers cannot be checked.

Results were verified against heap per query. Counts and `max` aggregates match
exactly. Averages agree to within 2.9e-15 relative difference. The residual is float
reassociation across parallel workers. Every group present in one engine is
present in the other.

**q1 to q3 are a defect, not a storage property.** They filter on one host and
return 360, 4,320 and 4,320 rows. Both engines take the same `Index Scan` plan on the equivalent index.
pgColumnar costs about 28.7 milliseconds per row returned. That cost is flat
across a twelve-fold change in row count. The cause is
[issue #353](https://github.com/jdatcmd/pgcolumnar/issues/353). The default
`stripe_row_limit` of 150,000 puts a table this wide over the 32 MB fetch cache
limit. Every fetch by row number then decodes the whole row group again. Lowering
`stripe_row_limit` to 100,000 on the same data takes the same query from 32.98 to
0.177 milliseconds per row. Until that is fixed, a table that serves selective
point queries through an index should be created with a smaller
`stripe_row_limit`.

[Issue #355](https://github.com/jdatcmd/pgcolumnar/issues/355) compounds it.
The planner will choose an index scan over a columnar table to obtain ordering.
It does that because the per-row fetch cost is not modelled. That is why these queries should
not be read as a measure of the storage format.

**pgColumnar leads on the scan-bound shapes.** q4, q6 and q8 read a large part of
the table. pgColumnar is ahead of heap on all three, and ahead of TimescaleDB on
q6 and q8. q6 in particular is 1592 ms against TimescaleDB's 6956 ms, where the
earlier table recorded 81966 ms against 1771 ms. Column projection accounts for
most of that change: on this fixture it alone takes q6 from 45094 ms to 6491 ms.

**The optional vectorization is not uniformly a win.** It helps q6, at 2292 ms
down to 1592 ms. It leaves q1, q3, q7 and q8 unchanged. It costs on q5, 16443 ms
up to 28494 ms, which is not yet explained and is tracked in
[issue #349](https://github.com/jdatcmd/pgcolumnar/issues/349). This is why those
settings default to off.

**TimescaleDB leads on the host-filtered queries** for the reason given before:
its columnstore segments by `hostname`, so it reads one segment. pgColumnar
stores in load order, so its zone maps do not skip on `hostname`. Clustering a
pgColumnar table on the filter key addresses that, at the cost of the all-host
queries.
Measured on 2026-08-04 against `main` at commit `aeb7882`, on the 100,000,000-row
fixture. The numbers are the median of five warm runs.

Serial, `max_parallel_workers_per_gather = 0`:

| query | shape | pgColumnar | TimescaleDB | heap | Citus |
| --- | --- | ---: | ---: | ---: | ---: |
| q1 | one host, 1 hour | 403 | 4 | 4 | 272 |
| q2 | one host, 12 hours | 2,211 | 5 | 11 | 1,865 |
| q3 | one host, 12 hours, 5 aggregates | 4,071 | 6 | 13 | 3,365 |
| q4 | all hosts, 12 hours, group by host | 8,210 | 5,119 | 9,713 | 7,364 |
| q5 | all hosts, 12 hours, 10 aggregates | 16,706 | 10,341 | 25,410 | 15,836 |
| q6 | full scan, one value filter | 11,220 | 1,737 | 14,525 | 8,124 |
| q7 | last point per host | 767 | 289 | 46 | 124,537 |
| q8 | top 20 by max | 15,757 | 7,554 | 20,507 | 14,578 |

Parallel, `max_parallel_workers_per_gather = 4`:

| query | pgColumnar | TimescaleDB | heap | Citus |
| --- | ---: | ---: | ---: | ---: |
| q1 | 73 | 4 | 4 | 277 |
| q2 | 500 | 5 | 12 | 1,897 |
| q3 | 875 | 6 | 14 | 3,366 |
| q4 | 8,256 | fails | 9,798 | 7,543 |
| q5 | **11,968** | fails | 28,043 | 16,658 |
| q6 | **2,294** | fails | 3,261 | 8,242 |
| q7 | 766 | 295 | 47 | 125,423 |
| q8 | 15,685 | fails | 20,020 | 14,928 |

Every cell is the median of five warm runs. The widest spread between the fastest and
the slowest of those five, anywhere in either table, is **1.04 times**.

**Spill.** Most cells use no temporary disk at `work_mem = 256MB`. Three do:

| cell | temp blocks read / written | node |
| --- | ---: | --- |
| q7, Citus, serial | 1,022,514 / 1,022,568 | Sort |
| q7, Citus, parallel | 1,022,514 / 1,022,568 | Sort |
| q5, pgColumnar, parallel | 721,569 / 721,584 | GroupAggregate |

The earlier record of this page had five, and two of them were q7 on pgColumnar. Those
are gone because the plan changed. A sort of the whole table became a skip scan over
the index, and a skip scan sorts nothing.

A plan that spills can be unstable between runs, which is why each cell is the median
of five. On this host the spilling cells are not the unstable ones. No cell in either
table has a spread wider than 1.04 times between its fastest and slowest run.

Do not read that as "spill does not matter". It matters at a small `work_mem`, where
the same shape has been measured to swing 1.43 times with every setting held constant.
It says that at this setting, on this host, the sort had enough memory for the spill to
be sequential and cheap.

**Parallel workers are what pgColumnar gains most from.** The scan divides cleanly
across workers. q1, q2 and q3 improve by about five times. q5 and q6 move from behind
heap to ahead of it. The serial table is a measure of the storage format. The parallel
table is closer to what an installation gets.

**TimescaleDB leads every query that filters one host.** Its columnstore segments by
`hostname`, so it reads one segment and not the table. pgColumnar stores rows in load
order, so its zone maps do not skip on `hostname`. A pgColumnar table that is clustered
on the filter key closes that gap, at the cost of the queries that read all hosts.

**pgColumnar leads the wide aggregate shapes with workers.** q5 reads ten metrics for
all hosts, and q6 scans the table with one filter. pgColumnar is first on both, and q6
is 2,294 ms against Citus at 8,242 ms.

**q7 was a planner defect and is now fixed.** The earlier record of this page reported
133,759 ms for q7 in serial. The cost model charged an index path for the rows the
path returns, and not for the rows the query reads. A `DISTINCT ON` reads one row per
host. The model therefore priced the index path far above every
alternative. No consumer could recover it, not even one that reads 3,998 rows of
100,000,000. That is
[issue #376](https://github.com/jdatcmd/pgcolumnar/issues/376), found by this
benchmark pass and fixed in
[#378](https://github.com/jdatcmd/pgcolumnar/pull/378), which bounds the penalty at a
multiple of one scan. The query now takes 767 ms.

Citus is slow on this shape for its own reasons, at 124,537 ms.

#### Queries

Expand Down
Loading