feat: fold-path payload deferral with an adaptive gate (#405 step 2)

[jdatcmd]

Implements the owner-scheduled rebuild of #405 Step 2, grounded in @ChronicallyJD's corrected measurement on the issue (the review that refuted my retraction). Design amendments recorded in design/ISSUE_405_LATE_MATERIALIZATION.md (in this PR).

What lands

• Two-phase gather on deferred groups: keys first; failing/deleted/vector-skipped rows consume payload slots without reading; survivors materialize payload post-key. The counter lives at that stable post-key site — Columnar Fold Payload Loads — and on a deferred group equals survivors × payload values, the identity whose absence let the original Step 2's counter read 0 and mean nothing.
• Step 4 as a runtime-adaptive gate (amendment, rationale in the doc): defer unless observed survival across previous groups exceeds ½, first group optimistic. No planner plumbing, no stats dependence, one comparison per group (Vectorized decompression + aggregation for full-scan aggregates (~4× behind TimescaleDB) #289 guard holds), observable as Columnar Fold Deferred Groups: X of Y. Push back to the planner design if you disagree — the suite binds behavior, not mechanism.
• Step 3 by construction: the new label is fold-specific; no line means two quantities on two plans.

The numbers

Measured in-suite on the 120k-row fixture: deferred loads 4,800 (1,200 survivors × 4) vs 480,000 forced-eager — the pre-change cost, 100×. At 99% survival the gate defers exactly the optimistic first group.

Proofs

• RED-first (counter lines absent on main; recorded).
• Forcing eager reds the identity at exactly 480,000 and 0-of-6 deferred, correctness green.
• Removing the adaptive disable reds the 99% arm at 6-of-6. This proof initially ran against unmutated code — a sed delimiter collision — and was redone; noted in the commit because a removal proof that did not run is a green lie.
• Three-way correctness (deferred fold == row path == heap mirror) at both selectivities; fold pinned via the post-EXPLAIN ANALYZE reports "Columnar Batch Fold: yes" for executions that fell back to the row path #602 ANALYZE line; plain-aggregate tlists deliberately (the EXPLAIN ANALYZE reports "Columnar Batch Fold: yes" for executions that fell back to the row path #602 tlist trap would silently row-path an expression over aggregates).

Also from this work

Issue #616 (filed against stats() during suite development) is retracted and closed: the suite variable was named GROUPS — a bash builtin that silently ignores assignments — so it read root's gid forever while every probe read correctly. Two hours, one false accusation, one durable gotcha note.

Verification

13/13 on PG17 (lane), PG18, PG19. Nine fold-family suites unregressed. ASAN+UBSAN gate green over the four fold suites (decode-path change → the #584 rule). -Wshadow -Werror clean.

Scope bounds, per the census: only sum/avg over int2/int4/float4/float8 fold and defer; int8/numeric/min-max never fold; q24 untouched (byval gate); the whole path sits behind enable_ungrouped_vector_agg (default off).

🤖 Generated with Claude Code

[ChronicallyJD]

Approve. The reorder keeps the present-index invariant in every row disposition, and the counter identity is real.

This builds the gap I measured, and the thing that had to be right is the packed-stream present index: cpresent[col] must advance once per valid row whether or not the value is read, or every later row's fetch_att misaligns and the aggregate is silently wrong. I traced all four dispositions on a deferred group:

• Phase 1 gathers only key columns — deferred payload columns are not even cursor-advanced here.
• vec-skipped, deleted, and key-fail each add a loop that advances cpresent for valid payload columns without fetching — the slot is consumed, not read.
• survivor → Phase 2 fetches and advances.

So each row's payload slots are consumed exactly once, and the net cpresent movement matches the eager path (which advanced all needed columns in one loop). A column that is both a scan key and an aggregate input is fetched in Phase 1 as a key and used directly, never double-counted (only !ciskey payload increments the counter). The three-way correctness arm (deferred == row path == heap mirror) is the right pin for "the reorder did not corrupt an aggregate", and it is there at both selectivities.

• Counter identity: foldPayloadLoads at the post-key site equals survivors × payload values on a deferred group — the exact identity whose absence let the original Step 2's counter read 0 and mean nothing. Forcing eager reds it at 480,000 (candidates × payload) and 0 of 6 deferred; the difference is the saving. Good.
• Adaptive gate: candRows/survRows accumulate across groups and deferOn is computed at each group's top from the prior groups' survival, first group optimistic. At 99% survival it defers exactly group 1 (1 of 6) and goes eager after — no planner plumbing, one comparison per group, the #289 guard intact.
• Scope: byval-only, enable_ungrouped_vector_agg default off, q24 untouched — matches the census from the #601 correction.

I appreciated the honesty on the way: the removal proof that first ran against unmutated code (sed delimiter collision) redone rather than trusted, and #616 retracted for the GROUPS bash-builtin gotcha. Both are the standard.

Scope: verified by close read of every disposition path (a wrong cpresent advance is a silent wrong-aggregate, so that is where I spent the read); I did not re-run native_fold_deferral.sh (bench + sanitizer box are mid-gate on my recluster PR). The identity + three-way correctness + your 13/13 and fold-family sanitizer cover the execution. Approving.