From fe193649c7af50701eb72fde788d354e6a11cce3 Mon Sep 17 00:00:00 2001 From: ChronicallyJD Date: Fri, 7 Aug 2026 13:28:54 -0600 Subject: [PATCH] docs: a vector is chunk_group_row_limit rows, not a fixed 1024 (#491 follow-up) CONTEXT.md defines a vector as "a fixed run of 1024 values inside a column chunk". That is the native (PGCN v1) geometry from spec section 4. It is not what the classic path does, and CONTEXT.md is the document people will trust. A vector holds up to pgcolumnar.chunk_group_row_limit rows: default 10000, PGC_USERSET, range 100 to INT_MAX, per-table overridable. Rebuilding the same 200,000 rows at three settings and running the same predicate: chunk_group_row_limit Columnar Vectors Skipped 10000 4 5000 8 1024 39 Each matches the arithmetic for the row group that gets read. The 1024 run also grows a "Rows Removed by Filter: 64" line, because 190000 is not a multiple of 1024, so the straddling vector is decoded and filtered rather than skipped -- independent confirmation that the boundary moved. Where 1024 is real: COLUMNAR_NATIVE_VECTOR_LENGTH in columnar.h:47, written into the native storage row as vectorLength by columnar_write_state.c:529. Spec section 4 also gives a 122880-row group limit, which is likewise not what ships (stripe_row_limit defaults to 150000). There is no pgcolumnar.vector_length GUC. This also removes a contradiction inside the file: the misleading-words section says chunk_group_row_limit "does not control the group counters", which is true, but a vector fixed at 1024 would leave that setting sizing nothing. It sizes the vector, so it moves Vectors Skipped instead. That table is now in the file. Two further entries in the same section, from the same review: - projection names two unrelated things -- the secondary physical ordering, and enable_column_projection / "Columnar Projected Columns" for reading fewer columns. Same collision shape as chunk group. - pruning and filtering are different outcomes and a plan prints both. Chunk Groups Removed by Filter is work never done; Rows Removed by Filter is work done and discarded. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01QRQYekvivA4RLDnndhanHK --- CONTEXT.md | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) diff --git a/CONTEXT.md b/CONTEXT.md index 429dbf1..cf4d231 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -45,8 +45,11 @@ Section 2 of the spec is authoritative. The short form: across all columns. A relation is a sequence of row groups. It is the write unit and the parallel-scan unit. - **Column chunk**: one column's data within one row group. -- **Vector**: a fixed run of 1024 values inside a column chunk. The unit of - decode, of data skipping, and of vectorized execution. +- **Vector**: a run of up to `pgcolumnar.chunk_group_row_limit` rows (default + 10000) inside a column chunk. The unit of encoding, of data skipping within a + row group, and of vectorized execution. The native format fixes this at 1024 + values; the classic path does not, and `Columnar Vectors Skipped` moves with + the setting. - **Page**: the on-disk container of one column chunk's encoded vectors. A contiguous byte range in the relation's main fork, which is why the buffer manager, WAL, and page checksums apply. @@ -75,6 +78,31 @@ because it is user-facing and it is in dumps. **`chunk_group_row_limit` is a different setting and does not control the group counters.** It is "maximum number of rows per chunk group", default 10000, from the 1.0-dev lineage. Setting it does not change how many groups a scan reports. +What it does size is the VECTOR, so it moves `Columnar Vectors Skipped` instead. +Rebuilding the same 200,000 rows and running the same predicate: + +| `chunk_group_row_limit` | `Columnar Vectors Skipped` | +| ---: | ---: | +| 10000 | 4 | +| 5000 | 8 | +| 1024 | 39 | + +Each matches the arithmetic for the row group that is read: 50,000 rows at +10,000 is 5 vectors with 4 below the predicate, at 5,000 it is 10 with 8 below, +at 1,024 it is about 49 with 39 below. The 1,024 run also grows a +`Rows Removed by Filter: 64` line, because 190,000 is not a multiple of 1024, so +the straddling vector is decoded and filtered rather than skipped. + +**`projection` names two unrelated things.** A **projection** is the secondary +physical ordering defined above. `pgcolumnar.enable_column_projection` and +EXPLAIN's `Columnar Projected Columns` use the same word for reading only the +columns a query references, which has nothing to do with it. Say **column +projection** for the second and never the bare word. + +**Pruning and filtering are different outcomes, and a plan prints both.** +`Chunk Groups Removed by Filter` and `Vectors Skipped` are work never done. +`Rows Removed by Filter` is work done and thrown away. They appear on the same +node, and which one moved tells you whether a predicate actually helped. **EXPLAIN's "Chunk Groups" counters count ROW GROUPS.** This is the one most likely to produce a wrong conclusion, so it is worth stating plainly.