fix(gateway-operator): recover interrupted and exhausted deployments - #3136
Open
sanjulaonline wants to merge 4 commits into
Open
fix(gateway-operator): recover interrupted and exhausted deployments#3136sanjulaonline wants to merge 4 commits into
sanjulaonline wants to merge 4 commits into
Conversation
Two independent defects each left an APIGateway permanently wedged. Helm: InstallOrUpgrade chose install versus upgrade purely on whether release history existed, and treated any history error as "absent". A release interrupted mid-operation keeps history in a pending state, so the operator kept calling upgrade and Helm kept refusing it. Read the full history, decide from the latest revision's status, and recover: purge and reinstall a revision-1 pending-install (nothing successful to lose), roll back a pending-upgrade or pending-rollback to the newest successful revision first, and report a clear error when no safe revision exists rather than deleting anything. Revisions are compared numerically, only a genuine not-found means absent, and install/upgrade now run context-aware with cleanup on failure. Reconciliation: exhausting the retry budget recorded the gateway as Deployed while persisting Programmed=False with observedGeneration equal to the CR generation. Neither the already-programmed branch nor the newer-generation branch then applied, so reconciliation fell through to "nothing to do" and only a generation bump could revive it. Record an explicit Failed state, keep reporting the failure truthfully, and give the gateway a bounded retry after the configured sync period. A change to the referenced ConfigMap or to deployment-affecting annotations retries immediately and resets the retry budget, while an unchanged failure requeues without deploying or writing status so a status-driven reconcile cannot hot-loop. Also removes an equal-generation check that was unreachable inside the newer-generation branch. Fixes wso2#3105
sanjulaonline
requested review from
AnuGayan,
Arshardh,
CrowleyRajapakse,
HeshanSudarshana,
HiranyaKavishani,
Induwara04,
Krishanx92,
PasanT9,
Piumal1999,
RakhithaRR,
Tharsanan1,
Thushani-Jayasekera,
VirajSalaka,
ashera96,
chamilaadhi,
dushaniw,
hisanhunais,
lasanthaS,
malinthaprasan,
pubudu538,
renuka-fernando,
senthuran16,
tgtshanika,
tharikaGitHub,
tharindu1st and
thivindu
as code owners
August 4, 2026 19:42
Contributor
📝 WalkthroughWalkthroughThe gateway controller now persists failed deployment state, fingerprints deployment inputs, and performs bounded recovery. The Helm client now plans recovery from release history, including pending-release rollback and purge-and-install paths, with context-aware execution. ChangesGateway deployment recovery
Helm release recovery
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant APIGateway
participant GatewayController
participant GatewayTracker
participant HelmClient
APIGateway->>GatewayController: trigger reconciliation
GatewayController->>GatewayTracker: inspect generation, input hash, and retry state
GatewayTracker-->>GatewayController: return recovery decision
GatewayController->>HelmClient: deploy with input hash
HelmClient-->>GatewayController: return deployment result
GatewayController->>APIGateway: persist status and retry timing
sequenceDiagram
participant HelmClient
participant HelmHistory
participant RecoveryPlanner
participant HelmRelease
HelmClient->>HelmHistory: read release history
HelmHistory-->>RecoveryPlanner: return revisions and statuses
RecoveryPlanner-->>HelmClient: return recovery operation
HelmClient->>HelmRelease: execute install, upgrade, purge, or rollback
HelmRelease-->>HelmClient: return operation result
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (5)
kubernetes/gateway-operator/internal/helm/client.go (2)
240-249: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winReject a non-positive
revisionbefore running the rollback.
planReleasealways supplies a positiverollbackRevisionforoperationRollbackThenUpgradetoday. If that guarantee changes,action.RollbacktreatsVersionzero as "roll back to the previous revision". The release would then move to a revision the planner never selected. A guard keeps the destructive step tied to the plan.♻️ Proposed guard
func (c *Client) rollback(ctx context.Context, actionConfig *action.Configuration, opts InstallOrUpgradeOptions, revision int) error { log := log.FromContext(ctx) + if revision <= 0 { + return fmt.Errorf("refusing to roll back release %q to non-positive revision %d", opts.ReleaseName, revision) + } + client := action.NewRollback(actionConfig) client.Version = revision🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kubernetes/gateway-operator/internal/helm/client.go` around lines 240 - 249, Update Client.rollback to validate revision before constructing or running the rollback action, returning an error immediately when revision is non-positive. Preserve the existing rollback configuration for positive revisions so the operation remains tied to the planner-selected revision.
220-225: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winSet a bounded default timeout for Helm Wait steps.
action.NewUninstallandaction.NewRollbackdo not set a defaultTimeout. Helm’s wait logic uses the timeout passed in, soWait=trueandTimeout=0can create a wait with no configured budget rather than the expected CLI-backed default. Use a shared fallback timeout inpurge/rollback; the same fallback should apply to the install/upgrade path too.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kubernetes/gateway-operator/internal/helm/client.go` around lines 220 - 225, The Helm action flows in purge, rollback, and install/upgrade need a shared bounded default timeout when waiting. Define or reuse one fallback timeout and assign it to each action’s Timeout before applying the optional opts.Timeout override, preserving explicit positive timeout values across action.NewUninstall, action.NewRollback, and the install/upgrade action setup.kubernetes/gateway-operator/internal/controller/apigateway_controller.go (2)
396-413: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract the duplicated failed-recovery block.
Lines 399-413 repeat lines 338-353 exactly, including both log messages. Two copies of the same decision will drift when the recovery policy changes. Extract one helper that takes the log context and returns the result.
♻️ Proposed helper
// recoverFailed applies the bounded recovery decision for a failed tracking entry. func (r *GatewayReconciler) recoverFailed( ctx context.Context, log *slog.Logger, gatewayConfig *apiv1.APIGateway, trackingKey string, crGeneration int64, entry *GatewayTrackingEntry, configHash, inputHash string, ) (ctrl.Result, error) { deployNow, requeueAfter, reason := r.decideFailedRecovery(entry, inputHash, time.Now()) if !deployNow { log.Debug("Failed APIGateway awaiting its retry window", slog.String("name", gatewayConfig.Name), slog.Int64("generation", crGeneration), slog.Duration("retryIn", requeueAfter)) return ctrl.Result{RequeueAfter: requeueAfter}, nil } log.Info("Recovering failed APIGateway deployment", slog.String("name", gatewayConfig.Name), slog.Int64("generation", crGeneration), slog.String("reason", reason)) return r.deploy(ctx, gatewayConfig, trackingKey, crGeneration, configHash, inputHash) }Then both sites become:
if trackingEntry.Status == GatewayTrackingStatusFailed { - deployNow, requeueAfter, reason := r.decideFailedRecovery(trackingEntry, inputHash, time.Now()) - if !deployNow { - log.Debug("Failed APIGateway awaiting its retry window", - slog.String("name", gatewayConfig.Name), - slog.Int64("generation", crGeneration), - slog.Duration("retryIn", requeueAfter)) - return ctrl.Result{RequeueAfter: requeueAfter}, nil - } - log.Info("Recovering failed APIGateway deployment", - slog.String("name", gatewayConfig.Name), - slog.Int64("generation", crGeneration), - slog.String("reason", reason)) - return r.deploy(ctx, gatewayConfig, trackingKey, crGeneration, currentConfigHash, inputHash) + return r.recoverFailed(ctx, log, gatewayConfig, trackingKey, crGeneration, + trackingEntry, currentConfigHash, inputHash) }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kubernetes/gateway-operator/internal/controller/apigateway_controller.go` around lines 396 - 413, Extract the duplicated failed-recovery logic from the branches around decideFailedRecovery into a shared GatewayReconciler.recoverFailed helper, preserving the existing retry-window and recovery log messages and deploy behavior. Pass the relevant context, logger, gatewayConfig, tracking key, generation, config hash, input hash, and tracking entry, then replace both inline blocks with calls to the helper.
750-755: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winApply a minimum floor to the retry window.
retryWindowreturnsSyncPeriodwhenever it is positive. A small configuredSyncPeriod, for example 1 second, makes an exhausted gateway run a Helm operation every second forever, because each recovery attempt re-exhausts the budget and re-arms the window. Clamp the window to a minimum so recovery stays bounded independently of the resync cadence.♻️ Proposed floor
+const minRetryWindow = 1 * time.Minute + func (r *GatewayReconciler) retryWindow() time.Duration { if r.Config != nil && r.Config.Reconciliation.SyncPeriod > 0 { - return r.Config.Reconciliation.SyncPeriod + if r.Config.Reconciliation.SyncPeriod < minRetryWindow { + return minRetryWindow + } + return r.Config.Reconciliation.SyncPeriod } return 10 * time.Minute }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kubernetes/gateway-operator/internal/controller/apigateway_controller.go` around lines 750 - 755, Update GatewayReconciler.retryWindow to enforce a minimum retry-window duration when Config.Reconciliation.SyncPeriod is configured, returning the larger of the configured period and the established floor. Preserve the existing default for missing or non-positive configuration, so recovery cannot be re-armed at an excessively short cadence.kubernetes/gateway-operator/internal/controller/apigateway_recovery_test.go (1)
478-492: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider covering the retry-budget wiring inside
processGatewayDeployment.The decision tests replace
deployGateway, so theretryCountForcall inprocessGatewayDeployment(controller lines 490-499) never runs.retryCountForis covered directly at lines 556-575, but the wiring that preserves an exhausted count across a recovery attempt is not. Without that coverage, a future change that dropsInputHashfrom the new Processing entry would silently grant a fresh retry budget on every recovery, and every test here would still pass.Add one test that calls
processGatewayDeploymentwith a seeded exhausted entry and asserts the storedRetryCountandInputHashsurvive when the inputs are unchanged.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kubernetes/gateway-operator/internal/controller/apigateway_recovery_test.go` around lines 478 - 492, Add a focused test for processGatewayDeployment that seeds a processing tracking entry with an exhausted RetryCount and its existing InputHash, then invokes the deployment with unchanged inputs and verifies the resulting stored entry preserves both values. Use the existing recovery test helpers and tracking symbols, ensuring the test exercises retryCountFor wiring rather than the deployGateway-stubbed decision path.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@kubernetes/gateway-operator/internal/helm/client.go`:
- Around line 337-339: Update the comment above client.CleanupOnFail to state
that it removes resources newly created by a failed upgrade, rather than rolling
back the release. In the same Helm client configuration, set client.Atomic =
true to restore the previous revision state when the upgrade fails.
In `@kubernetes/gateway-operator/internal/helm/recovery_test.go`:
- Around line 66-71: Update the “uninstalled installs” case in the recovery
tests to expect the purge operation instead of operationInstall, matching the
planned behavior of planRelease for an uninstalled release history.
In `@kubernetes/gateway-operator/internal/helm/recovery.go`:
- Around line 110-116: Plan uninstalled releases as operationPurgeThenInstall in
the StatusUninstalled branch of recovery.go, ensuring the held release name is
purged before installation. Update the corresponding “uninstalled installs” case
in recovery_test.go to expect the new recovery plan.
---
Nitpick comments:
In `@kubernetes/gateway-operator/internal/controller/apigateway_controller.go`:
- Around line 396-413: Extract the duplicated failed-recovery logic from the
branches around decideFailedRecovery into a shared
GatewayReconciler.recoverFailed helper, preserving the existing retry-window and
recovery log messages and deploy behavior. Pass the relevant context, logger,
gatewayConfig, tracking key, generation, config hash, input hash, and tracking
entry, then replace both inline blocks with calls to the helper.
- Around line 750-755: Update GatewayReconciler.retryWindow to enforce a minimum
retry-window duration when Config.Reconciliation.SyncPeriod is configured,
returning the larger of the configured period and the established floor.
Preserve the existing default for missing or non-positive configuration, so
recovery cannot be re-armed at an excessively short cadence.
In `@kubernetes/gateway-operator/internal/controller/apigateway_recovery_test.go`:
- Around line 478-492: Add a focused test for processGatewayDeployment that
seeds a processing tracking entry with an exhausted RetryCount and its existing
InputHash, then invokes the deployment with unchanged inputs and verifies the
resulting stored entry preserves both values. Use the existing recovery test
helpers and tracking symbols, ensuring the test exercises retryCountFor wiring
rather than the deployGateway-stubbed decision path.
In `@kubernetes/gateway-operator/internal/helm/client.go`:
- Around line 240-249: Update Client.rollback to validate revision before
constructing or running the rollback action, returning an error immediately when
revision is non-positive. Preserve the existing rollback configuration for
positive revisions so the operation remains tied to the planner-selected
revision.
- Around line 220-225: The Helm action flows in purge, rollback, and
install/upgrade need a shared bounded default timeout when waiting. Define or
reuse one fallback timeout and assign it to each action’s Timeout before
applying the optional opts.Timeout override, preserving explicit positive
timeout values across action.NewUninstall, action.NewRollback, and the
install/upgrade action setup.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 9433164e-3055-419e-8f10-4b0d10f09ad7
📒 Files selected for processing (5)
kubernetes/gateway-operator/internal/controller/apigateway_controller.gokubernetes/gateway-operator/internal/controller/apigateway_recovery_test.gokubernetes/gateway-operator/internal/helm/client.gokubernetes/gateway-operator/internal/helm/recovery.gokubernetes/gateway-operator/internal/helm/recovery_test.go
An uninstalled release keeps its name in Helm's storage when history was retained, so Helm's availableName check rejects a plain install with "cannot re-use a name that is still in use" unless Install.Replace is set, which this package does not set. Plan that state as purge-then-install instead: the uninstall already removed the resources, so dropping the retained history discards nothing live, and an already-uninstalled release is purgeable because KeepHistory is false. Also corrects the upgrade comment: CleanupOnFail deletes resources the failed upgrade created, it does not roll the release back. Related to wso2#3105
Remove the History.Max handling. In helm.sh/helm/v3@v3.21.3 action.History.Run returns cfg.Releases.History(name) directly and never reads Max, so setting it was a no-op and the comments describing it as a truncation control were wrong. The real improvements stay: the returned history is now actually used, revisions are compared numerically, and a history error that is not "release not found" is propagated instead of being read as an absent release. Add a release executor seam so InstallOrUpgrade's dispatch is testable: the planner alone cannot catch a correct plan that is dispatched wrongly, which is how the uninstalled-release bug reached review. Purge and rollback action construction is split out so KeepHistory=false and the planned rollback revision are assertable without a cluster. Drop the tracking entry when a status write fails. A Processing or Deployed entry tells the next reconcile that another operation owns the generation, but the reconcile that owned it has already returned; the next one then skipped and returned nil, ending the requeue chain the returned error had started. A failed initial-condition patch and a failed Programmed=True patch both wedged the gateway this way. Rename the spec.infrastructure annotation test to say what it covers, and add the metadata-annotation case from the issue. Changing metadata.annotations is not a deployment input and is deliberately not an immediate retry trigger; recovery there is the bounded retry window. Related to wso2#3105
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
kubernetes/gateway-operator/internal/controller/apigateway_controller.go (1)
21-22: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse an approved post-quantum hash for the deployment input fingerprint.
deploymentInputHash()still usesSHA-256; replace it withSHA-3orBLAKE3at a quantum-safe size while keepingInputHashdeterministic for retry decisions.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@kubernetes/gateway-operator/internal/controller/apigateway_controller.go` around lines 21 - 22, Update deploymentInputHash() to replace SHA-256 with an approved SHA-3 or BLAKE3 hash configured at a quantum-safe output size, while preserving deterministic InputHash generation for retry decisions and updating the related imports.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@kubernetes/gateway-operator/internal/controller/apigateway_controller.go`:
- Around line 21-22: Update deploymentInputHash() to replace SHA-256 with an
approved SHA-3 or BLAKE3 hash configured at a quantum-safe output size, while
preserving deterministic InputHash generation for retry decisions and updating
the related imports.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 1990cdb7-949b-447c-b6cb-b68f0f441eee
📒 Files selected for processing (7)
kubernetes/gateway-operator/internal/controller/apigateway_controller.gokubernetes/gateway-operator/internal/controller/apigateway_recovery_test.gokubernetes/gateway-operator/internal/controller/apigateway_statuspatch_test.gokubernetes/gateway-operator/internal/helm/client.gokubernetes/gateway-operator/internal/helm/dispatch_test.gokubernetes/gateway-operator/internal/helm/recovery.gokubernetes/gateway-operator/internal/helm/recovery_test.go
🚧 Files skipped from review as they are similar to previous changes (3)
- kubernetes/gateway-operator/internal/helm/recovery_test.go
- kubernetes/gateway-operator/internal/helm/recovery.go
- kubernetes/gateway-operator/internal/helm/client.go
The exhaustion status-patch test configured a deployment spy but never exercised it: it called decideFailedRecovery directly and silenced the unused variable, so it did not prove the reconcile path recovers. Run the real decision path twice after the failed status write instead. Inside the retry window it must not deploy and must still return a bounded RequeueAfter; once the window has elapsed it must deploy exactly once. The tracker entry now carries the gateway's real input fingerprint so the unchanged-input path is the one under test. Also correct the GatewayTracker comment, which still claimed entries persist until the CR is deleted. Related to wso2#3105
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
An
APIGatewaycan end up permanently wedged. Two independent defects combine, and each is worth fixing on its own.1. An interrupted Helm operation is never recovered.
InstallOrUpgradechose between install and upgrade purely on whether release history existed:A release stuck in
pending-installdoes have history, so the operator called upgrade, which Helm refuses withanother operation (install/upgrade/rollback) is in progress. Nothing inspectedrel.Info.Status, so no pending state was ever recovered. Two further problems in the same three lines: the returned history was discarded entirely, andreleaseExists := err == niltreated any error — a transient storage failure, an unreachable cluster — as "the release is absent", which would start a fresh install over a live release.2. After retry exhaustion the CR cannot be re-driven.
handleGatewayDeploymentErrorrecorded exhaustion asGatewayTrackingStatusDeployedwhile persistingProgrammed=False/DeploymentFailedwithobservedGeneration == metadata.generation.decideAndProcessthen had two live branches: one requiringProgrammed=True, the other gated oncrGeneration > statusObservedGen. An equal-generation failure satisfies neither, so control fell through to "nothing to do". Restarting the operator, re-annotating, fixing theconfigRefConfigMap and waiting for the resync all did nothing. There was also an equal-generation branch nested inside thecrGeneration > statusObservedGenblock, which was unreachable.3. A failed status write wedged the gateway too (found during a second-pass audit of this PR). The tracker was moved to
ProcessingbeforesetGatewayInitialConditions, and toDeployedbefore theProgrammed=Truepatch. If the patch failed, the reconcile returned an error — but the tracker now told the next reconcile that another operation owned the generation, so it skipped and returnednil, ending the requeue chain that the error had started.Fixes #3105
Related, deliberately out of scope: #3104 is the most common trigger for (1) — a
RollingUpdate+ReadWriteOncedeadlock that fails the operator'sWait: trueHelm operation — but it is a separate Helm-chart problem and is not touched here.Goals
Recover a Helm release left in a pending state, and make an exhausted
APIGatewayrecoverable without editing its generation — without introducing a hot reconcile loop.Approach
Helm recovery
planReleaseis a pure function deciding the operation from the release's stored history:deployedfailed,superseded,unknownuninstalled(history retained)pending-installat revision 1, no successful revisionpending-installbeyond revision 1, successful revision existspending-upgrade,pending-rollbackuninstallinguninstalledmust purge rather than install:Install.availableNamein helm v3.21.3 rejects a name whose history is retained withcannot re-use a name that is still in useunlessReplaceis set, which this package does not set. The uninstall already removed the resources, so only history is discarded.The safety property for a purge is that no live resources are discarded — not that no successful revision exists. An uninstalled release may still hold earlier
supersededrevisions in retained history; those resources are already gone.Other Helm changes: revisions are compared numerically rather than trusting history order; only
errors.Is(err, driver.ErrReleaseNotFound)counts as absent and every other error is propagated;install/upgradeuseRunWithContextso a cancelled reconcile or operator shutdown aborts the wait; andCleanupOnFailis set on upgrade and rollback.CleanupOnFaildeletes resources the failed operation created — it does not roll the release back; the release still records afailedrevision, whichplanReleaseupgrades from next time.Atomicis deliberately not enabled: it would change failure semantics and add a second wait inside the operator's Helm timeout, and recovery does not need it.Reconciliation recovery
GatewayTrackingStatusFailedreplaces "record the exhausted deployment as Deployed". The persisted condition still truthfully reportsProgrammed=False/DeploymentFailed.sync_period, soRequeueAftermatches the operator's own resync cadence.configRefConfigMap values andspec.infrastructurelabels/annotations — everything that changes what would be rendered. Both are already-marshalled YAML with sorted keys, so it does not depend on Go map ordering.Not supported:
kubectl annotateas an immediate retry triggerThe issue lists re-annotating the CR as an attempted recovery. That remains not an immediate trigger, deliberately.
metadata.annotationsare not deployment inputs — onlyspec.infrastructurelabels/annotations reach the rendered values viacommonAnnotations— so fingerprinting them would retry on changes that cannot alter the outcome. It is also a hot-loop risk: this codebase already writes operator-managed annotations onto resources it reconciles (httproute_controller.go), so any such annotation added toAPIGatewaylater would retrigger deployments on the operator's own writes. Making re-annotation a supported reset means introducing a documented reset annotation, which is a public API decision for the maintainers rather than something to add silently here.The gateway is still not wedged by this: it recovers on the bounded retry window, on an operator restart, and on a ConfigMap correction.
TestDecideAndProcess_metadataAnnotationOnlyChangeWaitsForRetryWindowasserts that honest behaviour so the limitation cannot be mistaken for a fix.User stories
As an operator whose gateway failed to install, I can fix the cause — correct the ConfigMap, restart the operator, or simply wait for the resync — and see the gateway recover, instead of having to delete the
APIGatewayCR and the Helm release by hand.Documentation
N/A — no configuration, API or CRD surface changes. Recovery behaviour that was previously impossible now happens automatically.
Automation tests
Unit tests
Integration tests
Verification performed, from
kubernetes/gateway-operator:go build ./...go vet ./...go test ./...go test $(go list ./... | grep -v /e2e) -coverprofile …make testgo-test line)go test -race ./internal/helm ./internal/controllergo test -count=1 ./internal/helm ./internal/controller×3gofmt -l/gofmt -don the changed filesgit diff --checkGo is not installed on the machine used, so every command above ran in a
golang:1.26container matching the module's toolchain.Security checks
The only destructive operations are Helm purges, and both cases are guarded by
planRelease: a revision-1pending-installthat never deployed, and an already-uninstalledrelease whose resources are gone and whose history alone remains. Any pending state without a safe recovery target returns an error instead.uninstallingis never resolved automatically.Samples
N/A
Related PRs
RollingUpdate+ReadWriteOncedeadlock that most often triggers the interrupted install fixed here. Separate Helm-chart problem, deliberately out of scope.Test environment