fix: a rewrite's own projection ids are not foreign reservations (#345) - #347
Merged
Conversation
pgcolumnar.sort_status reported ~90% decay on a table that had just been fully reclustered, whenever the table had a projection. The rows were correctly ordered; only the reporting was wrong. record_online_sorted_extent walks the rewrite's own reserved stripe ids and stops at the first gap, on the reasoning that a gap means another session took that id. That reasoning is false within the rewrite's own transaction. A projection writes through its own inner write state but calls ColumnarWriteRow with the BASE relation, so it reserves from this relation's stripe counter; its groups are then recorded under the projection's own storage id and never appear in the base relation's group list. ColumnarWriteStateStripeIds returned only the base write state's reservations, so base and projection draws alternated, ours came back as 17, 19, 21, ... and the run walk broke at the first step. The rewrite was competing with itself. The ids are now unioned with the projection fan-out's, taking only those at or above the rewrite's own first id, so anything drawn before it began is still excluded. A projection id inside the run is harmless to sort_status: it counts base-relation groups, and a projection's group number is not one. Reproduced with a control, which is what establishes the projection as the cause rather than assuming it. Both arms are identical except for add_projection: noproj sorted 200000, appended 0 (correct) withproj sorted 20000, appended 180000 (wrong) Removal proof: with the union reverted, the withproj arm fails at 20000/180000 against 200000/0 while the noproj control stays green, so the test is pinned to this mechanism and not to breakage in general. This is the opposite error to #342 in the same function. #342 was the mark claiming groups it should not; this is the mark refusing groups it should. The range fix in #344 addressed the first and left runEnd computed by the same consecutive-run walk, so this survived it. Direction of error here is the safe one -- more decay reported than exists, so at worst an unnecessary re-sort -- but a table reporting 90% decay right after a successful full recluster makes sort_status useless for deciding when a re-sort is worth its cost. Full 15-19 matrix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011miCFRSatixeNRw3w5yNq8
Collaborator
|
Reviewed — correct and mergeable. This is the fix I'd scoped for #345 (include the rewrite's own projection reservations in
No objections. |
This was referenced Aug 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #345.
pgcolumnar.sort_statusreported ~90% decay on a table that had just been fully reclustered, whenever the table had a projection. The rows were correctly ordered; only the reporting was wrong.Cause
record_online_sorted_extentwalks the rewrite's own reserved stripe ids and stops at the first gap, reasoning that a gap means another session took that id. That is false within the rewrite's own transaction.A projection writes through its own inner write state but calls
ColumnarWriteRowwith the base relation, so it reserves from this relation's stripe counter. Its groups are then recorded under the projection's own storage id and never appear in the base relation's group list.ColumnarWriteStateStripeIdsreturned only the base write state's reservations, so base and projection draws alternated,ourscame back as 17, 19, 21, ... and the run walk broke at the first step.The rewrite was competing with itself.
Fix
The ids are unioned with the projection fan-out's, taking only those at or above the rewrite's own first id -- so anything drawn before this rewrite began is still excluded and #342's guarantee is untouched. A projection id landing inside the run is harmless:
sort_statuscounts base-relation groups, and a projection's group is recorded under a different storage id.Reproduction, with a control
Both arms identical except for
add_projection:Removal proof
The control staying green in both runs is the point: it pins the test to this mechanism rather than to breakage in general.
Relationship to #342 / #344
Opposite errors in the same function. #342 was the mark claiming groups it should not; this is the mark refusing groups it should. #344 changed the mark to a range
[ours[0], runEnd], which fixed the over-claim but leftrunEndcomputed by the same consecutive-run walk, so this survived it untouched.Direction of error here is the safe one -- more decay reported than exists, so at worst an unnecessary re-sort. But a table reporting 90% decay right after a successful full recluster makes
sort_statususeless for the one thing it exists to do.Gate
Full 15-19 matrix,
ALL VERSIONS PASSED, all suites.🤖 Generated with Claude Code
https://claude.ai/code/session_011miCFRSatixeNRw3w5yNq8