From deec1d3b69dbce5fdf4a06b0490ab5c29945d176 Mon Sep 17 00:00:00 2001 From: "Joshua (D) Drake" Date: Tue, 4 Aug 2026 18:04:46 -0600 Subject: [PATCH] Correct the zone-map explanation, and the reasoning that produced it The page said q2 reads all 667 row groups because neither key is sorted. Half of that is wrong, and the wrong half is mine: I wrote it in #381 and jdatcmd approved it. Re-measured on the same table, one predicate at a time: hostname only 667 of 667 read, 0 removed 10,791 ms time only 118 of 667 read, 549 removed 2,415 ms both, as q2 118 of 667 read, 549 removed 2,238 ms Time pruning already removes 82 percent of the groups. Hostname removes none. The zone maps work. The gap is a hostname clustering failure. From the zone_map catalog, decoded and checked against min(time) first: all 667 groups record the same minimum and maximum for hostname, so no group can be ruled out. A group's time span averages about 6 minutes, which is 0.30 percent of the table's range. The old 667 of 667 figure is exactly what the hostname-only predicate gives, and the page attributed it to full q2. Nobody could reproduce that pairing, so per jdatcmd's call on #391 it is replaced with a fresh measurement rather than explained. A new section covers the reasoning error, because it will recur otherwise. pg_stats.correlation measures value order against physical row order across the whole relation. Skipping depends on each group's minimum and maximum being narrow, which is local. This table separates the two: groups tight to about 6 minutes, but rotated rather than ascending, so correlation is near zero while skipping still removes 549 of 667 groups. The instrument that answers the question is Columnar Chunk Groups Removed by Filter, one predicate at a time. Closes #391. --- docs/benchmarks.md | 57 ++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 48 insertions(+), 9 deletions(-) diff --git a/docs/benchmarks.md b/docs/benchmarks.md index a4e9203..d15f409 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -375,8 +375,9 @@ The bench host has 16 vCPU and 62 GB. Each engine uses the configuration its own users would choose: - pgColumnar: columnar scan, storage in load order. One btree on - `(hostname, time DESC)`. The load order is not sorted on either key: the measured - correlation with physical order is 0.013 for `time` and -0.004 for `hostname`. + `(hostname, time DESC)`. The load order is not globally sorted on either key. It is + locally ordered on `time`, which is what matters for skipping. See + [what prunes and what does not](#what-prunes-and-what-does-not). - TimescaleDB: compressed columnstore, segmented by `hostname`, ordered by `time` descending. One btree on `(hostname, time DESC)`, and the `(time DESC)` index that `create_hypertable` makes, which gives 53 chunk indexes below them. @@ -481,16 +482,54 @@ q2, with `EXPLAIN (ANALYZE)`: ``` Columnar Chunk Groups Total: 667 -Columnar Chunk Groups Read: 667 -Columnar Chunk Groups Removed by Filter: 0 -Rows Removed by Filter: 99995680 +Columnar Chunk Groups Read: 118 +Columnar Chunk Groups Removed by Filter: 549 +Columnar Vectors Skipped: 40 +Rows Removed by Filter: 17295680 ``` -It reads all 667 row groups and filters 99,995,680 rows to return 4,320. The zone maps cannot help, -because neither key is sorted in the stored order. Every stripe holds all 4,000 hosts. -The minimum and maximum `hostname` of each group therefore covers the whole set. +The two predicates behave differently, and only one of them is the problem. Running each +alone on the same table: + +| predicate | groups read | groups removed | time | +| --- | ---: | ---: | ---: | +| `hostname` only | 667 of 667 | 0 | 10,791 ms | +| `time` only | 118 of 667 | 549 | 2,415 ms | +| both, as q2 | 118 of 667 | 549 | 2,238 ms | + +**Time pruning already removes 82 percent of the row groups. Hostname removes none.** +The zone maps are working. The gap is a `hostname` clustering failure, not a zone-map +failure. + +The catalog says why. Every one of the 667 groups records the same minimum and the same +maximum for `hostname`. Every group holds all 4,000 hosts, so no group can be ruled out. + +For `time` the picture is the opposite. A group spans about 6 minutes on average, which +is **0.30 percent** of the table's 2 day 21 hour range. + TimescaleDB answers the same query in 5 milliseconds. It excludes all but one -chunk on time, then reads one `hostname` segment through an index. +chunk on time, then reads one `hostname` segment through an index. It is the second half +that we lack, not the first. + +### What prunes and what does not + +An earlier version of this page argued from `pg_stats.correlation`, which reads 0.0133 +for `time` and -0.0036 for `hostname`, and concluded that neither key was ordered. The +conclusion was wrong for `time`, and the instrument was the reason. + +`correlation` measures how a column's values track physical row order **across the whole +relation**. Zone-map skipping does not depend on that. It depends on whether each +**group's** minimum and maximum are narrow against the predicate, which is a local +property. + +This table is the case that separates them. Its groups are individually tight on `time`, +about 6 minutes each. But the sequence of groups is rotated rather than ascending, with +group 1 beginning at 14:00:10 and group 663 at 13:31:00. A whole-relation correlation is +therefore near zero. Skipping still removes 549 groups of 667. + +**So do not infer pruning from `correlation`.** The instrument that answers the question +is `Columnar Chunk Groups Removed by Filter` in `EXPLAIN (ANALYZE)`. Run one predicate at +a time, because a conjunction hides which half is doing the work. **So this table measures pgColumnar in the layout that suits it least.** A user with this shape would cluster the table on `hostname`. That is what