From 029b4d51c517e98a868a48724bddf5ce5aa519fa Mon Sep 17 00:00:00 2001 From: "Joshua (D) Drake" <136637981+ChronicallyJD@users.noreply.github.com> Date: Wed, 5 Aug 2026 17:54:36 -0600 Subject: [PATCH 1/2] docs: record what the external validation established (#437) Four changes, each backed by a measurement rather than by the report alone. aarch64 is no longer untested. limitations.md said every recorded result comes from x86_64 and the suites have never run on aarch64 (#242). An independent gate on a Graviton3 host passed every suite with the same per-suite result as x86_64. The big-endian half of that sentence stays true and stays. The owner asked that the arm against x86 timing ratios not be published, so they are not here. The correctness update is a different claim and it is. stripe_row_limit is documented as the setting that governs fetch-by-index cost, with the sweep and with its cost. chunk_group_row_limit is named explicitly as NOT the setting for this, because the external report recommended it and it does nothing here: isolated one knob at a time, chunk_group alone moved a point lookup from 245.4 ms to 252.3 ms, and stripe_row_limit alone moved it to 19.3 ms. The floor is stated with the benefit. Below 10000 rows a selective range query regresses, because a smaller row group makes more metadata to read. A tuning knob documented without its cost is a trap. Cross-architecture portability is stated as narrowly as the format supports. The native format stores integers in HOST byte order, which the format specification says at line 11 and columnar.h:57 repeats. The validation moved a data directory from x86_64 to aarch64 and read it correctly, but both are little-endian, so that test cannot distinguish portable from host-endian. The docs now say same byte order, note that PostgreSQL's own heap format has the same rule, and say a big-endian move is unsupported and untested. index.md gains a "when not to use pgColumnar" section from the reporter's ledger numbers, attributed to them. It keeps the two rows where columnar still wins, because a section that lists only losses is not the honest version. Logical replication limitations are deliberately absent: #439 is writing those. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01UqprqkCXuH8SegiZejE1Tw --- docs/administration.md | 29 +++++++++++++++++++++++++++++ docs/index.md | 24 ++++++++++++++++++++++++ docs/limitations.md | 17 +++++++++-------- 3 files changed, 62 insertions(+), 8 deletions(-) diff --git a/docs/administration.md b/docs/administration.md index 49cd649..18ef102 100644 --- a/docs/administration.md +++ b/docs/administration.md @@ -49,6 +49,27 @@ range filters but hold less data per vector. `pgcolumnar.stripe_row_limit` (defa table with `pgcolumnar.set_options` when a specific access pattern calls for it, and measure the result. +`pgcolumnar.stripe_row_limit` is the setting that governs the cost of a fetch by +index. A fetch decodes the row group that holds the row. A large row group makes +each fetch expensive. Lower this setting for a table that takes many point +lookups. `pgcolumnar.chunk_group_row_limit` does not change this cost. Use +`stripe_row_limit` for this, not `chunk_group_row_limit`. + +Measured on 500,000 rows of 1 KiB incompressible data, which is the shape where +the effect is largest: + +| `stripe_row_limit` | total size | point lookup | full aggregate scan | +| --- | ---: | ---: | ---: | +| 150000 (default) | 527.0 MB | 244.1 ms | 38.7 ms | +| 50000 | 527.1 MB | 80.0 ms | 37.9 ms | +| 10000 | 527.3 MB | 18.5 ms | 35.4 ms | +| 2000 | 529.4 MB | 5.6 ms | 37.9 ms | + +The cost of a smaller row group is small on this data. Size grows by 0.4 percent +at 2000 rows. Scan throughput does not change. There is a floor, so do not go +lower than needed. A selective range query was slower at 2000 rows than at 10000. +A smaller row group makes more metadata to read. + ## Compaction and vacuum There are two distinct operations, and the difference matters: @@ -260,6 +281,14 @@ A columnar table is an ordinary WAL-logged relation. Install and preload the extension on any server that restores or replicates a columnar table, because reading the table requires the access method. +A physical copy moves between hosts of the same byte order. The native format +stores integers in host byte order, which the format specification states. This +is the same rule that PostgreSQL's own heap format follows, so a columnar table is +no more restricted than the rest of the cluster. An independent validation on +2026-08-05 moved a data directory from x86_64 to aarch64 and read it correctly. +Both of those are little-endian. A move to a big-endian host is not supported and +is not tested. + ## Security ### Server-side file access diff --git a/docs/index.md b/docs/index.md index e0a5625..2b063da 100644 --- a/docs/index.md +++ b/docs/index.md @@ -43,6 +43,30 @@ row updates and deletes, and for point lookups that return whole rows. pgColumnar supports updates, deletes, and indexes, but its storage layout is built for append-mostly data. See [Limitations and compatibility](limitations.md). +### When not to use pgColumnar + +The clearest case against columnar storage is a table whose bytes are mostly one +large value per row. An independent validation measured this on 2026-08-05, on a +2,000,000 row ledger table. Each row held 1 KiB of incompressible data. The +figures below are theirs. + +| operation | heap | pgColumnar | | +| --- | ---: | ---: | --- | +| point lookup by indexed uuid | 1.0 ms | 203 ms | 200x slower | +| ordered 100,000 row full-row segment | 42 ms | 998 ms | 24x slower | +| full-table full-row export scan | 630 ms | 1747 ms | 2.8x slower | +| bulk `INSERT ... SELECT` of 2M rows | 19.9 s | 42.6 s | 2.1x slower | +| table size | 2604 MB | 2051 MB | 1.27x smaller | +| narrow `GROUP BY` aggregate | 266 ms | 115 ms | **2.3x faster** | + +Read the last two rows with the others. Columnar storage still wins the narrow +aggregate, which is what it is for. It wins little on size, because only about a +quarter of the bytes are the kind that compress. Every row-wise operation loses. + +Use heap when most of your bytes are one large value per row. Use heap when point +lookups are the main access pattern. Use pgColumnar when queries read a few +columns out of many. + ## How it fits together A columnar table is an ordinary PostgreSQL relation. It works with transactions, diff --git a/docs/limitations.md b/docs/limitations.md index 5f81275..26ace42 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -24,9 +24,10 @@ on read. It does not only stamp the version on write cross-major `pg_upgrade`. Before, only a suite that nobody ran covered it ([issue #257](https://github.com/commandprompt/pgcolumnar/issues/257)). -What that testing does not cover is worth stating alongside it. Every result -recorded in this repository comes from x86_64. The suites have not been run on -aarch64 or on a big-endian platform +What that testing does not cover is worth stating alongside it. The suites run +and pass on aarch64. An independent validation on 2026-08-05 ran the full gate on +a Graviton3 host. Every suite passed, and the per-suite result was the same as the +x86_64 run. The suites have still not been run on a big-endian platform ([issue #242](https://github.com/commandprompt/pgcolumnar/issues/242)). Unaligned reads are one class that a reader could expect to differ by @@ -35,11 +36,11 @@ builds with the clang address checks and undefined-behaviour checks. These repor a misaligned load on any host. That gate exists because the project found and fixed such a read one time. -What stays untested on another architecture is what a sanitizer on x86_64 cannot -see. Memory ordering is the main item. x86_64 puts stores in a more strict order -than aarch64. Thus a missing barrier in concurrent code can be invisible on one -architecture and a defect on the other. Another architecture is untested. It is -not known to be broken. Give it that status when you build there. +What stays hard to cover is what a sanitizer on x86_64 cannot see. Memory +ordering is the main item. x86_64 puts stores in a more strict order than aarch64. +Thus a missing barrier in concurrent code can be invisible on one architecture and +a defect on the other. The aarch64 gate runs the concurrency and isolation suites +on real hardware. That is evidence. It is not a proof. PostgreSQL 19 coverage is against 19beta2, not a final release. From 3e4827d461338e3cec187749b3e7ae9741f006be Mon Sep 17 00:00:00 2001 From: "Joshua (D) Drake" <136637981+ChronicallyJD@users.noreply.github.com> Date: Wed, 5 Aug 2026 18:10:56 -0600 Subject: [PATCH 2/2] docs: limitations.md said the extension runs anywhere, which the format does not (#437) The Host architecture section said "The rest of the extension runs on any architecture PostgreSQL supports." The CODE does. A DATA DIRECTORY does not. The native format stores multi-byte values in host byte order. The format specification says so at line 11, and columnar.h:57 repeats it. So the sentence was true about the binary and misleading about the data, in the one section a reader consults before moving a cluster. This is the same claim added to administration.md in the previous commit, in the section that contradicted it. Both now say the same thing: same byte order, the same rule PostgreSQL's own heap format follows, and a big-endian move unsupported and untested. It also picks up the aarch64 update, since this section is where a reader looks for it. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01UqprqkCXuH8SegiZejE1Tw --- docs/limitations.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/docs/limitations.md b/docs/limitations.md index 26ace42..1595762 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -144,8 +144,18 @@ Three behaviors depend on the major: ## Host architecture -The Arrow and Parquet import and export functions run on little-endian hosts -only. The rest of the extension runs on any architecture PostgreSQL supports. +The suites run and pass on x86_64 and on aarch64. The Arrow and Parquet import +and export functions run on little-endian hosts only. + +The code runs on any architecture PostgreSQL supports. A data directory does not +move between architectures of different byte order. The native format stores each +multi-byte value in host byte order. The format specification states this. PostgreSQL's own heap format follows the same rule, so a columnar table +is no more restricted than the rest of the cluster. + +A move to a big-endian host is not supported and is not tested. The format does +not record the byte order of the host that wrote it. A read therefore has no +check that could refuse such a move. See +[Backup and restore](administration.md#backup-and-restore). ## Workload and access patterns