ISR-11280 Fix unbounded growth of Role status.conditions#13
Open
shreyansjainl wants to merge 1 commit into
Open
ISR-11280 Fix unbounded growth of Role status.conditions#13shreyansjainl wants to merge 1 commit into
shreyansjainl wants to merge 1 commit into
Conversation
appendRoleStatusCondition trimmed conditions to the last 5 entries into a local variable (roleStatusConditions) but then appended the new condition onto the original, untrimmed role.Status.Conditions field. The trim was silently discarded every time, so conditions accumulated forever instead of being capped at 5. In production this caused individual Role objects to grow to 8,000+ conditions (900KB+), exceeding etcd's request size limit and making every subsequent status update fail with "etcdserver: request is too large" - which in turn caused the controller to retry rapidly, making the object grow even faster. Fix: append onto the already-trimmed roleStatusConditions instead of the untrimmed role.Status.Conditions, matching the pattern already used correctly in GrantReconciler.appendGrantStatusCondition and GrantStatementReconciler.appendGrantStatementStatusCondition.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
appendRoleStatusConditioninrole_controller.gocomputes a trimmed slice (last 5 conditions) into a local variable, but then appends the new condition onto the original, untrimmedrole.Status.Conditionsfield — silently discarding the trim on every call.Roleobject accumulate 8,000+ conditions (900KB+), which exceeded etcd's request size limit. Every subsequent status write then failed withetcdserver: request is too large, and the resulting error-driven requeue caused the object to grow even faster — a self-reinforcing loop that never recovers on its own.GrantReconciler.appendGrantStatusCondition,GrantStatementReconciler.appendGrantStatementStatusCondition) already implement this trim correctly (they assign the trimmed slice back to the real.Status.Conditionsfield before appending) — this PR bringsRoleReconcilerin line with that existing, correct pattern.Fix
One-line change: append onto the already-trimmed
roleStatusConditionslocal variable instead of the untrimmedrole.Status.Conditionsfield.Test plan
kubectl get roles.postgresql.facets.cloud -Ain a live cluster that multipleRoleobjects had condition counts in the 8,000-8,900 range and object sizes near 1MB..status.conditionson the affected objects (via a direct PATCH to the/statussubresource) to unblock reconciliation immediately.[]metav1.Conditionvariables of identical type (no local Go toolchain was available to rungo build, but the change cannot introduce a compile error).len(role.Status.Conditions) <= 6after repeated calls toappendRoleStatusConditionwith alternating reasons, to prevent regression.🤖 Generated with Claude Code