sort_status reports ~90% decay immediately after a full recluster when the table has a projection

[jdatcmd]

Summary

pgcolumnar.sort_status reports ~90% decay on a table that was just fully reclustered, whenever the table has a projection. The rows are correctly ordered; only the reporting is wrong.

Reproduction, with a control

PG17, current main (00a0955). Identical fixture in both arms: 200,000 rows, stripe_row_limit => 20000, then pgcolumnar.recluster('t','id','k') with no concurrent writer at all.

arm	sorted_rows	appended_rows
no projection	200000	0
with a projection	20000	180000

The only difference between the arms is pgcolumnar.add_projection(t,'p',ARRAY['k','id'],ARRAY['k']). The control is what makes this a finding rather than an observation: without the projection the same code path reports the run correctly, so the projection is the cause.

20000 is exactly one group at this stripe_row_limit, i.e. the run is being truncated after the rewrite's very first group.

Cause

record_online_sorted_extent (src/columnar_vacuum.c) walks the rewrite's own reserved stripe ids and stops at the first gap, on the reasoning that a gap means some other session took that id:

runEnd = ours[0];
for (i = 1; i < nOurs; i++)
{
    if (ours[i] != runEnd + 1)
        break;              /* a foreign reservation took this id */
    runEnd = ours[i];
}

That reasoning is false within the rewrite's own transaction when the table has a projection. The projection fan-out writes through a separate write state (ColumnarWriteRow(w->innerWs, ...), src/columnar_write_state.c) but reserves from the same base-relation stripe counter, and the projection's group is recorded under its own storage id (ColumnarNextStorageId()), so it never appears in the base relation's group list.

ColumnarWriteStateStripeIds returns only the base write state's reservations. Base and projection draws therefore alternate, ours comes back as 17, 19, 21, ... and the run walk breaks at the very first step. Nothing foreign is involved -- the rewrite is competing with itself.

Direction of the error

Safe: it reports more decay than exists, which at worst prompts an unnecessary re-sort. The opposite direction (claiming ordered rows that are not) is #342. But a table that reports 90% decay immediately after a successful full recluster makes sort_status useless for its stated purpose on any table with a projection, which is deciding when a re-sort is worth the cost.

Relationship to #342 / #344

Not the same defect and not fixed by #344. #342 was the mark claiming groups it should not; this is the mark refusing groups it should claim. #344 changes the mark to a range [ours[0], runEnd], which fixes the over-claim but leaves runEnd computed by the same consecutive-run walk, so this under-claim survives unchanged. Verified against a tree containing #344's fix.

Suggested direction

The run walk needs to distinguish "an id another session took" from "an id my own projection fan-out took". The projection's reservations are known to the same transaction, so the simplest fix is to have ColumnarWriteStateStripeIds return the base write state's ids plus those of its projection inner write states, and treat that union as ours. Then the gap test means what it says again.

Worth checking whether anything else draws from the base counter within one transaction the same way.

Note on provenance

Found during an audit prompted by #342. Two other candidate findings from that same audit (silent row duplication in concurrent recluster, and index-only scans double-counting after a group retire) did not reproduce under controlled retest and are deliberately not filed here.