Correct the zone-map explanation, and the reasoning that produced it (#391) - #406
Open
ChronicallyJD wants to merge 1 commit into
Open
Correct the zone-map explanation, and the reasoning that produced it (#391)#406ChronicallyJD wants to merge 1 commit into
ChronicallyJD wants to merge 1 commit into
Conversation
The page said q2 reads all 667 row groups because neither key is sorted. Half of that is wrong, and the wrong half is mine: I wrote it in commandprompt#381 and jdatcmd approved it. Re-measured on the same table, one predicate at a time: hostname only 667 of 667 read, 0 removed 10,791 ms time only 118 of 667 read, 549 removed 2,415 ms both, as q2 118 of 667 read, 549 removed 2,238 ms Time pruning already removes 82 percent of the groups. Hostname removes none. The zone maps work. The gap is a hostname clustering failure. From the zone_map catalog, decoded and checked against min(time) first: all 667 groups record the same minimum and maximum for hostname, so no group can be ruled out. A group's time span averages about 6 minutes, which is 0.30 percent of the table's range. The old 667 of 667 figure is exactly what the hostname-only predicate gives, and the page attributed it to full q2. Nobody could reproduce that pairing, so per jdatcmd's call on commandprompt#391 it is replaced with a fresh measurement rather than explained. A new section covers the reasoning error, because it will recur otherwise. pg_stats.correlation measures value order against physical row order across the whole relation. Skipping depends on each group's minimum and maximum being narrow, which is local. This table separates the two: groups tight to about 6 minutes, but rotated rather than ascending, so correlation is near zero while skipping still removes 549 of 667 groups. The instrument that answers the question is Columnar Chunk Groups Removed by Filter, one predicate at a time. Closes commandprompt#391.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #391. Follows your plan on that issue, including the three verification steps.
The correction
The page said q2 reads all 667 groups because neither key is sorted. Re-measured on the
same table, one predicate at a time:
hostnameonlytimeonlyTime pruning already removes 82 percent of the groups. Hostname removes none. The
zone maps work. The gap is a
hostnameclustering failure, which is the second ofTimescaleDB's two mechanisms, not the first.
Verification, per your three steps
1. Decomposition with
EXPLAIN (ANALYZE, BUFFERS). Above. Buffers came out atshared hit=70995for the hostname arm against20491for the time arm, which is thesame story in a second currency.
2. Physical order from the zone maps, not
pg_stats. Decoded thebyteaminima andmaxima out of
pgcolumnar.zone_map. The decode is checked before anything rests on it:the minimum across all groups' minima is
2024-01-01 00:00:00+00, which equalsmin(time)on the table.And for
hostname, the half that does not prune:All 667 groups share one minimum and one maximum. Every group holds all 4,000 hosts.
That is the clearest statement of the problem I could find, and it comes from the catalog
rather than from a plan.
3. Environment.
work_mem256 MB,shared_buffers16 GB, serial, bloom on. No spill:every arm reports
shared hitonly, no read and no temp.The 667 that could not be reproduced
Your call, taken.
667 / 667is exactly the hostname-only figure, and the page pairedit with full q2's row counts. That pairing does not occur in any arm I can produce now, so
the number is replaced with a fresh measurement rather than explained.
The methodological section, which you said matters more
Added, because the next person will otherwise repeat it. This table is a clean separator:
groups individually tight on
timeat about 6 minutes, but rotated rather thanascending, group 1 starting 14:00:10 and group 663 starting 13:31:00. A whole-relation
correlation is near zero. Skipping still removes 549 of 667.
So
correlationanswers "does value order track physical row order across the relation",and skipping asks "is each group's range narrow", which is local. The page now says to use
Columnar Chunk Groups Removed by Filter, one predicate at a time, since a conjunctionhides which half is working.
The setup list also no longer cites correlation as evidence that the load order is
unsorted, since that is the claim this corrects.
Docs only.
ste_checkclean.