Skip to content

pgcolumnar.analyze(): exact null_frac, n_distinct and histogram_bounds (#414 slices 1-3) - #475

Merged
ChronicallyJD merged 5 commits into
commandprompt:mainfrom
ChronicallyJD:feat/414-analyze
Aug 7, 2026
Merged

pgcolumnar.analyze(): exact null_frac, n_distinct and histogram_bounds (#414 slices 1-3)#475
ChronicallyJD merged 5 commits into
commandprompt:mainfrom
ChronicallyJD:feat/414-analyze

Conversation

@ChronicallyJD

Copy link
Copy Markdown
Collaborator

Slices 1-3 of #414. Slices 1 and 2 were already written and are here rebased onto current main; slice 3 is new.

slice statistic how
1 null_frac exact, from zone maps, no decode
2 n_distinct exact, from a single-column read
3 histogram_bounds exact ends, from that same read

PG18+ only: it writes through pg_restore_attribute_stats, which 15-17 do not have. The suite skips below 18 rather than writing pg_statistic directly.

Slice 3

percentile_disc over an array of fractions returns actual column values, one per fraction, in a single ordered pass. Fraction 1.0 is therefore the true maximum and 0.0 the true minimum. percentile_cont would interpolate and invent values the column does not hold, which is wrong for a histogram of stored data and impossible for a non-numeric type.

That is the case that matters: a range predicate above the sampled maximum is where the planner's estimate collapses, and a value held by one row in 500,000 is one a 30,000-row sample misses.

Guarded to btree-orderable types, and skipped when the column has fewer distinct values than buckets, which is where core emits no histogram either.

What this slice does not claim. Core excludes most-common-values from the histogram. This function writes none, so there is nothing to double-count and the two are consistent as written. Writing both without that exclusion would over-count those values in selectivity, which is why most_common_vals is a separate slice rather than a line added here.

Two fixture findings that invalidated the plan I posted

The planned column cannot test this. g % 100 gives 100 distinct values, and core stores every one as a most-common-value and emits no histogram at all, so the check compared against an empty array and failed on its own premise rather than on the behaviour. The ordinary values now span 100,000 distinct.

"Core misses the outlier" cannot be a gate. It is probabilistic by definition, and it also depends on how our own access method hands rows to the sampler. Measured 99,999 on one run and 1,000,000 on the next, on identical data. Gating on it would be exactly the flaky-by-construction shape the fixture was built to avoid.

So it is reported, and exactness is asserted instead. The expected value comes from an independent SELECT max(), not from the code path under test, so the check still fails whenever our bounds are wrong. It does not need core to be unlucky.

The error check exists because I needed it

While writing the slice the function raised

ERROR:  record "att" has no field "atttypid"

and the redirect swallowed it. The call did nothing, pg_stats still held core's numbers, and the failure presented as "our maximum is wrong" rather than "our function did not run". The assertion caught it; the diagnosis needed the error. So the call's success is now a check of its own.

Gate

PG18  analyze_function   rc=0  checks run: 16  PASSED
PG18  harness_selftest   rc=0  checks run: 40  PASSED
PG19  analyze_function   rc=0  checks run: 16  PASSED
PG19  harness_selftest   rc=0  checks run: 40  PASSED

harness_selftest alongside because this registers analyze_function in the matrix for the first time; #473's sortedness and registration checks are what keep that honest.

Still open on #414, and named rather than implied

  • slice 3b: most_common_vals / most_common_freqs, with the MCV-exclusion constraint above and a differential harness alongside it.
  • the catch that should decide the feature: autoanalyze will not call this. Stale statistics silently produce bad plans, which is a worse failure than the stale-vacuum case. That is Future: a background worker to schedule columnar vacuum and analyze, which autovacuum cannot reach #415's territory and it is not resolved by this PR.
  • The issue's headline "exact bounds from zone maps without reading data" is partly undercut, as recorded in the design comment on the issue: slice 2 already reads the whole column, so bounds fall out of that read for free. The zone-map advantage is real for null_frac and largely illusory for bounds.

ChronicallyJD and others added 5 commits August 6, 2026 22:38
… from zone maps

Core ANALYZE decodes essentially the whole table: a fixed 30,000-row sample
falls in every row group, so every group is decoded for every column. Measured
on 3M rows x 20 columns, 1237 MB, serial: ANALYZE 6,302 ms against 268 ms to
decode one column, and 7,680 ms to decode all nineteen text columns outright.

Not fixable in the AM callbacks: acquire_sample_rows copies whole tuples, so
asking for one column saves 6% (6,073 vs 6,302 ms), and shrinking
chunk_group_row_limit tenfold changes nothing (6,466 vs 6,771 ms) because a
fixed-size sample touches proportionally more groups.

This slice collects null_frac only, exactly, from the zone maps.

Work in progress: slices 2-5 (single-column n_distinct, histogram with exact
endpoints, and the differential-against-core verification) are not here yet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqprqkCXuH8SegiZejE1Tw
…n read

null_frac is metadata only; n_distinct is the slice that has to actually read a
column, and it is where the case for this function stands or falls. A projected
single-column read of the 3M x 20 fixture costs 268 ms against core ANALYZE's
6,302 ms, because core's fixed 30,000-row sample lands in every row group and so
decodes every column.

Mirrors core's sign convention from analyze.c: absolute count normally, negated
fraction once the distinct count passes 10% of rows. Getting that backwards
produces plausible wrong estimates rather than an error, so the suite pins its
fixture to the absolute-count side and asserts it.

The suite's non-destruction check moved from n_distinct to correlation. This
slice writes n_distinct, so watching it there would have asserted nothing while
still printing PASS.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UqprqkCXuH8SegiZejE1Tw
…prompt#414 slice 3)

percentile_disc over an array of fractions returns ACTUAL column values, one per
fraction, in a single ordered pass. Fraction 1.0 is therefore the true maximum and
0.0 the true minimum. percentile_cont would interpolate and invent values the
column does not hold, which is wrong for a histogram of stored data and
impossible for a non-numeric type.

That is the case that matters. A range predicate above the sampled maximum is
where the planner's estimate collapses, and a value held by one row in 500,000 is
one a 30,000-row sample misses.

Guarded to types with a btree ordering, and skipped when the column holds fewer
distinct values than buckets, which is where core emits no histogram either
because the most-common-value list already describes the column.

Bounds what this slice claims: core EXCLUDES most-common-values from the
histogram. This function writes none, so there is nothing to double count and the
two are consistent as written. Writing both without that exclusion would
over-count those values in selectivity, which is why most_common_vals is a
separate slice rather than a line added here.

Two fixture findings, both of which invalidated the plan as written:

The planned column (g % 100) cannot test this at all. With 100 distinct values
core stores every one as a most-common-value and emits NO histogram, so the check
compared against an empty array and failed on its own premise. The ordinary values
now span 100,000 distinct.

And "core misses the outlier" cannot be a gate. It is probabilistic by
definition, and it also depends on how our own access method hands rows to the
sampler: measured 99,999 on one run and 1,000,000 on the next, on identical data.
Gating on it would be the flaky-by-construction shape the fixture was built to
avoid. It is reported, and exactness is asserted instead -- against an
independent SELECT max(), so the check still fails whenever our bounds are wrong.

The error check exists because I needed it. The function raised

    ERROR:  record "att" has no field "atttypid"

and the redirect swallowed it, so the call did nothing, pg_stats still held core's
numbers, and the failure presented as "our maximum is wrong" rather than "our
function did not run". The assertion caught it; the diagnosis needed the error.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L2DvnWDM7g27ubDCQdXhky
…cy (commandprompt#414)

CI went red on PG17 the moment this suite was registered:

    FAIL  pgcolumnar.analyze() needs pg_restore_attribute_stats (PG18+); this server is 17
    FAIL   PG17  (124 ran, 7 skipped)

The gate used pgc_skip, which is the wrong instrument. pgc_skip is for a missing
DEPENDENCY -- pyarrow, nm -- which is an environment defect, so it fails by
default and has to be waived deliberately, because somebody should install the
thing. A major that does not ship pg_restore_attribute_stats is not a defect
anyone can fix: 15 to 17 genuinely lack it, the same way 15 lacks WITHOUT
OVERLAPS. Failing there is a red nobody can act on, which is the kind that
teaches readers to discount red.

So it reports SKIP and runs no checks, which pgc_summary turns into
PGC_EXIT_SKIPPED and the matrix records as SKIP (commandprompt#447, commandprompt#455). Same shape as
pg19_vacuum_options on anything below 19; verified both exit 66 on PG17.

The major is asserted before the comparison. An unreadable version must not be
mistaken for an old one, or a broken environment reports SKIP and reads as
"this major does not support it".

Gated on a FRESH tree per major. Two false reds while checking this were mine,
not the code: the first was leftover postmasters holding the port band, and the
second was building PG17 and PG18 in one tree, so PG18 linked stale objects and
the postmaster would not load the library. Both present as "no cluster of our own
after 8 attempts", which looks nothing like its cause.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L2DvnWDM7g27ubDCQdXhky
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants