Skip to content

Correct the zone-map explanation, and the reasoning that produced it (#391) - #406

Open
ChronicallyJD wants to merge 1 commit into
commandprompt:mainfrom
ChronicallyJD:fix/391-zonemap
Open

Correct the zone-map explanation, and the reasoning that produced it (#391)#406
ChronicallyJD wants to merge 1 commit into
commandprompt:mainfrom
ChronicallyJD:fix/391-zonemap

Conversation

@ChronicallyJD

Copy link
Copy Markdown
Collaborator

Closes #391. Follows your plan on that issue, including the three verification steps.

The correction

The page said q2 reads all 667 groups because neither key is sorted. Re-measured on the
same table, one predicate at a time:

predicate groups read groups removed time
hostname only 667 of 667 0 10,791 ms
time only 118 of 667 549 2,415 ms
both, as q2 118 of 667 549 2,238 ms

Time pruning already removes 82 percent of the groups. Hostname removes none. The
zone maps work. The gap is a hostname clustering failure, which is the second of
TimescaleDB's two mechanisms, not the first.

Verification, per your three steps

1. Decomposition with EXPLAIN (ANALYZE, BUFFERS). Above. Buffers came out at
shared hit=70995 for the hostname arm against 20491 for the time arm, which is the
same story in a second currency.

2. Physical order from the zone maps, not pg_stats. Decoded the bytea minima and
maxima out of pgcolumnar.zone_map. The decode is checked before anything rests on it:
the minimum across all groups' minima is 2024-01-01 00:00:00+00, which equals
min(time) on the table.

 group_number |           lo           |           hi           |   span
--------------+------------------------+------------------------+----------
            1 | 2024-01-01 14:00:10+00 | 2024-01-01 14:06:30+00 | 00:06:20
            2 | 2024-01-01 14:06:30+00 | 2024-01-01 14:12:40+00 | 00:06:10
          666 | 2024-01-01 13:49:50+00 | 2024-01-01 13:56:00+00 | 00:06:10
          667 | 2024-01-01 13:56:00+00 | 2024-01-01 14:00:10+00 | 00:04:10

 groups |   table_span    | avg_group_span  | pct_of_table
--------+-----------------+-----------------+--------------
    667 | 2 days 21:26:30 | 00:12:29.190405 |         0.30

And for hostname, the half that does not prune:

 groups | distinct_minima | distinct_maxima
--------+-----------------+-----------------
    667 |               1 |               1

All 667 groups share one minimum and one maximum. Every group holds all 4,000 hosts.
That is the clearest statement of the problem I could find, and it comes from the catalog
rather than from a plan.

3. Environment. work_mem 256 MB, shared_buffers 16 GB, serial, bloom on. No spill:
every arm reports shared hit only, no read and no temp.

The 667 that could not be reproduced

Your call, taken. 667 / 667 is exactly the hostname-only figure, and the page paired
it with full q2's row counts. That pairing does not occur in any arm I can produce now, so
the number is replaced with a fresh measurement rather than explained.

The methodological section, which you said matters more

Added, because the next person will otherwise repeat it. This table is a clean separator:
groups individually tight on time at about 6 minutes, but rotated rather than
ascending
, group 1 starting 14:00:10 and group 663 starting 13:31:00. A whole-relation
correlation is near zero. Skipping still removes 549 of 667.

So correlation answers "does value order track physical row order across the relation",
and skipping asks "is each group's range narrow", which is local. The page now says to use
Columnar Chunk Groups Removed by Filter, one predicate at a time, since a conjunction
hides which half is working.

The setup list also no longer cites correlation as evidence that the load order is
unsorted, since that is the claim this corrects.

Docs only. ste_check clean.

The page said q2 reads all 667 row groups because neither key is sorted. Half
of that is wrong, and the wrong half is mine: I wrote it in commandprompt#381 and jdatcmd
approved it.

Re-measured on the same table, one predicate at a time:

  hostname only   667 of 667 read, 0 removed    10,791 ms
  time only       118 of 667 read, 549 removed   2,415 ms
  both, as q2     118 of 667 read, 549 removed   2,238 ms

Time pruning already removes 82 percent of the groups. Hostname removes none.
The zone maps work. The gap is a hostname clustering failure.

From the zone_map catalog, decoded and checked against min(time) first: all
667 groups record the same minimum and maximum for hostname, so no group can
be ruled out. A group's time span averages about 6 minutes, which is 0.30
percent of the table's range.

The old 667 of 667 figure is exactly what the hostname-only predicate gives,
and the page attributed it to full q2. Nobody could reproduce that pairing, so
per jdatcmd's call on commandprompt#391 it is replaced with a fresh measurement rather than
explained.

A new section covers the reasoning error, because it will recur otherwise.
pg_stats.correlation measures value order against physical row order across
the whole relation. Skipping depends on each group's minimum and maximum being
narrow, which is local. This table separates the two: groups tight to about 6
minutes, but rotated rather than ascending, so correlation is near zero while
skipping still removes 549 of 667 groups. The instrument that answers the
question is Columnar Chunk Groups Removed by Filter, one predicate at a time.

Closes commandprompt#391.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

benchmarks.md: the zone-map explanation is half wrong, and the correlation argument behind it does not hold

1 participant