Skip to content

ISR-11280 Fix unbounded growth of Role status.conditions#13

Open
shreyansjainl wants to merge 1 commit into
Facets-cloud:mainfrom
shreyansjainl:fix/status-conditions-unbounded-growth
Open

ISR-11280 Fix unbounded growth of Role status.conditions#13
shreyansjainl wants to merge 1 commit into
Facets-cloud:mainfrom
shreyansjainl:fix/status-conditions-unbounded-growth

Conversation

@shreyansjainl

Copy link
Copy Markdown

Summary

  • appendRoleStatusCondition in role_controller.go computes a trimmed slice (last 5 conditions) into a local variable, but then appends the new condition onto the original, untrimmed role.Status.Conditions field — silently discarding the trim on every call.
  • In production, this let a single Role object accumulate 8,000+ conditions (900KB+), which exceeded etcd's request size limit. Every subsequent status write then failed with etcdserver: request is too large, and the resulting error-driven requeue caused the object to grow even faster — a self-reinforcing loop that never recovers on its own.
  • The sibling reconcilers (GrantReconciler.appendGrantStatusCondition, GrantStatementReconciler.appendGrantStatementStatusCondition) already implement this trim correctly (they assign the trimmed slice back to the real .Status.Conditions field before appending) — this PR brings RoleReconciler in line with that existing, correct pattern.

Fix

One-line change: append onto the already-trimmed roleStatusConditions local variable instead of the untrimmed role.Status.Conditions field.

Test plan

  • Confirmed via kubectl get roles.postgresql.facets.cloud -A in a live cluster that multiple Role objects had condition counts in the 8,000-8,900 range and object sizes near 1MB.
  • Manually truncated .status.conditions on the affected objects (via a direct PATCH to the /status subresource) to unblock reconciliation immediately.
  • Verified the fix is a type-safe swap between two []metav1.Condition variables of identical type (no local Go toolchain was available to run go build, but the change cannot introduce a compile error).
  • Recommend adding a unit test asserting len(role.Status.Conditions) <= 6 after repeated calls to appendRoleStatusCondition with alternating reasons, to prevent regression.

🤖 Generated with Claude Code

appendRoleStatusCondition trimmed conditions to the last 5 entries into
a local variable (roleStatusConditions) but then appended the new
condition onto the original, untrimmed role.Status.Conditions field.
The trim was silently discarded every time, so conditions accumulated
forever instead of being capped at 5.

In production this caused individual Role objects to grow to 8,000+
conditions (900KB+), exceeding etcd's request size limit and making
every subsequent status update fail with "etcdserver: request is too
large" - which in turn caused the controller to retry rapidly, making
the object grow even faster.

Fix: append onto the already-trimmed roleStatusConditions instead of
the untrimmed role.Status.Conditions, matching the pattern already
used correctly in GrantReconciler.appendGrantStatusCondition and
GrantStatementReconciler.appendGrantStatementStatusCondition.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant