diff --git a/docs/administration.md b/docs/administration.md index 49cd649..18ef102 100644 --- a/docs/administration.md +++ b/docs/administration.md @@ -49,6 +49,27 @@ range filters but hold less data per vector. `pgcolumnar.stripe_row_limit` (defa table with `pgcolumnar.set_options` when a specific access pattern calls for it, and measure the result. +`pgcolumnar.stripe_row_limit` is the setting that governs the cost of a fetch by +index. A fetch decodes the row group that holds the row. A large row group makes +each fetch expensive. Lower this setting for a table that takes many point +lookups. `pgcolumnar.chunk_group_row_limit` does not change this cost. Use +`stripe_row_limit` for this, not `chunk_group_row_limit`. + +Measured on 500,000 rows of 1 KiB incompressible data, which is the shape where +the effect is largest: + +| `stripe_row_limit` | total size | point lookup | full aggregate scan | +| --- | ---: | ---: | ---: | +| 150000 (default) | 527.0 MB | 244.1 ms | 38.7 ms | +| 50000 | 527.1 MB | 80.0 ms | 37.9 ms | +| 10000 | 527.3 MB | 18.5 ms | 35.4 ms | +| 2000 | 529.4 MB | 5.6 ms | 37.9 ms | + +The cost of a smaller row group is small on this data. Size grows by 0.4 percent +at 2000 rows. Scan throughput does not change. There is a floor, so do not go +lower than needed. A selective range query was slower at 2000 rows than at 10000. +A smaller row group makes more metadata to read. + ## Compaction and vacuum There are two distinct operations, and the difference matters: @@ -260,6 +281,14 @@ A columnar table is an ordinary WAL-logged relation. Install and preload the extension on any server that restores or replicates a columnar table, because reading the table requires the access method. +A physical copy moves between hosts of the same byte order. The native format +stores integers in host byte order, which the format specification states. This +is the same rule that PostgreSQL's own heap format follows, so a columnar table is +no more restricted than the rest of the cluster. An independent validation on +2026-08-05 moved a data directory from x86_64 to aarch64 and read it correctly. +Both of those are little-endian. A move to a big-endian host is not supported and +is not tested. + ## Security ### Server-side file access diff --git a/docs/index.md b/docs/index.md index e0a5625..2b063da 100644 --- a/docs/index.md +++ b/docs/index.md @@ -43,6 +43,30 @@ row updates and deletes, and for point lookups that return whole rows. pgColumnar supports updates, deletes, and indexes, but its storage layout is built for append-mostly data. See [Limitations and compatibility](limitations.md). +### When not to use pgColumnar + +The clearest case against columnar storage is a table whose bytes are mostly one +large value per row. An independent validation measured this on 2026-08-05, on a +2,000,000 row ledger table. Each row held 1 KiB of incompressible data. The +figures below are theirs. + +| operation | heap | pgColumnar | | +| --- | ---: | ---: | --- | +| point lookup by indexed uuid | 1.0 ms | 203 ms | 200x slower | +| ordered 100,000 row full-row segment | 42 ms | 998 ms | 24x slower | +| full-table full-row export scan | 630 ms | 1747 ms | 2.8x slower | +| bulk `INSERT ... SELECT` of 2M rows | 19.9 s | 42.6 s | 2.1x slower | +| table size | 2604 MB | 2051 MB | 1.27x smaller | +| narrow `GROUP BY` aggregate | 266 ms | 115 ms | **2.3x faster** | + +Read the last two rows with the others. Columnar storage still wins the narrow +aggregate, which is what it is for. It wins little on size, because only about a +quarter of the bytes are the kind that compress. Every row-wise operation loses. + +Use heap when most of your bytes are one large value per row. Use heap when point +lookups are the main access pattern. Use pgColumnar when queries read a few +columns out of many. + ## How it fits together A columnar table is an ordinary PostgreSQL relation. It works with transactions, diff --git a/docs/limitations.md b/docs/limitations.md index 5f81275..1595762 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -24,9 +24,10 @@ on read. It does not only stamp the version on write cross-major `pg_upgrade`. Before, only a suite that nobody ran covered it ([issue #257](https://github.com/commandprompt/pgcolumnar/issues/257)). -What that testing does not cover is worth stating alongside it. Every result -recorded in this repository comes from x86_64. The suites have not been run on -aarch64 or on a big-endian platform +What that testing does not cover is worth stating alongside it. The suites run +and pass on aarch64. An independent validation on 2026-08-05 ran the full gate on +a Graviton3 host. Every suite passed, and the per-suite result was the same as the +x86_64 run. The suites have still not been run on a big-endian platform ([issue #242](https://github.com/commandprompt/pgcolumnar/issues/242)). Unaligned reads are one class that a reader could expect to differ by @@ -35,11 +36,11 @@ builds with the clang address checks and undefined-behaviour checks. These repor a misaligned load on any host. That gate exists because the project found and fixed such a read one time. -What stays untested on another architecture is what a sanitizer on x86_64 cannot -see. Memory ordering is the main item. x86_64 puts stores in a more strict order -than aarch64. Thus a missing barrier in concurrent code can be invisible on one -architecture and a defect on the other. Another architecture is untested. It is -not known to be broken. Give it that status when you build there. +What stays hard to cover is what a sanitizer on x86_64 cannot see. Memory +ordering is the main item. x86_64 puts stores in a more strict order than aarch64. +Thus a missing barrier in concurrent code can be invisible on one architecture and +a defect on the other. The aarch64 gate runs the concurrency and isolation suites +on real hardware. That is evidence. It is not a proof. PostgreSQL 19 coverage is against 19beta2, not a final release. @@ -143,8 +144,18 @@ Three behaviors depend on the major: ## Host architecture -The Arrow and Parquet import and export functions run on little-endian hosts -only. The rest of the extension runs on any architecture PostgreSQL supports. +The suites run and pass on x86_64 and on aarch64. The Arrow and Parquet import +and export functions run on little-endian hosts only. + +The code runs on any architecture PostgreSQL supports. A data directory does not +move between architectures of different byte order. The native format stores each +multi-byte value in host byte order. The format specification states this. PostgreSQL's own heap format follows the same rule, so a columnar table +is no more restricted than the rest of the cluster. + +A move to a big-endian host is not supported and is not tested. The format does +not record the byte order of the host that wrote it. A read therefore has no +check that could refuse such a move. See +[Backup and restore](administration.md#backup-and-restore). ## Workload and access patterns