Do not report host deletion as completed on an unknown API error - #2057
Open
somanchi004-code wants to merge 1 commit into
Open
Do not report host deletion as completed on an unknown API error#2057somanchi004-code wants to merge 1 commit into
somanchi004-code wants to merge 1 commit into
Conversation
deleteHost() fetches the host StatefulSet and treats any error from that Get as "StatefulSet not found - already deleted", emitting a DeleteCompleted event and returning nil. Only NotFound actually means the host is gone; a Forbidden, a timeout or any other transient API error takes the same branch. That early return skips both deleteTables() - which the surrounding comment notes is required so ZooKeeper stops tracking the host's tables - and Controller.deleteHost(), which deletes the host's PVCs. Those PVCs carry no owner reference (see model/common/creator/pvc.go, where it is commented out to stay compatible with the PV retain policy), so the operator's own call is the only thing that ever reclaims them. Classify the error instead: keep the existing behaviour for NotFound, and on any other error emit DeleteFailed and return it rather than claiming the host was deleted. apiErrors.IsNotFound is already used this way elsewhere in the package, for example in worker-pdb.go. Signed-off-by: Somanchi Poorna Sobhita <somanchi004@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
deleteHost()fetches the host's StatefulSet and treats any error from that Get as"already deleted":
Only
NotFoundmeans the host is actually gone. AForbidden, a timeout, a 429 or areset connection all take the same branch, and the early return skips the two cleanup
steps below it:
deleteTables()— which the comment just above it notes is needed so ZooKeeper stopstracking the host's tables
Controller.deleteHost()— which deletes the host's PVCsThe PVCs matter most because they have no owner reference.
creator/pvc.gocommentsit out in two places to stay compatible with the PV retain policy, so Kubernetes GC
will not reclaim them and this call is the only thing that ever does.
This PR only changes the classification:
NotFoundkeeps today's behaviour, anythingelse emits
DeleteFailedand returns the error instead of reporting success.apiErrors.IsNotFoundis already used this way 8 times in this package —worker-pdb.go:41is the same get / classify / act shape.
Verified on a cluster
Single-node k3s v1.36.2+k3s1, operator 0.27.2, PVCs on Ceph RBD. To make the failure
deterministic I removed
getonstatefulsetsfrom the operator's ClusterRole, so thecall returns
Forbidden.Before this change, deleting the CHI removed the CR and the StatefulSet but left
the PVC and PV bound, with no CR remaining to reconcile them away:
A control run with unmodified RBAC removed CHI, StatefulSet, pod, PVC and PV cleanly,
so the normal path works — the leak is specific to the misclassified error.
After this change, the same delete reports the failure instead of claiming success:
I also re-created the CHI under the patched build first and it reconciled to
Completednormally, so theNotFoundpath is unaffected.Scope
This is deliberately narrow, and on its own it does not stop the leak. In the run
above the PVC was still orphaned and the CHI was still deleted, because the error this
patch now returns is discarded by the callers at
worker-deleter.go:634and:747andthe finalizer is removed regardless. Fixing that means deciding a retry policy, which is
not mine to decide, so I have raised it separately in #2056 rather than widening this
patch. What this change buys by itself is that the operator stops reporting a deletion
it did not perform.
I also considered simply proceeding with cleanup after a failed Get, but
PVCDeleter.HostCanDeletePVCwalks volume mounts viaapi.CurStatefulSetand defaultsto
PVCReclaimPolicyDeletewhen it finds none — so that path would delete PVCs whosereclaim policy could not be confirmed. Returning the error seemed clearly safer.
Notes
No behaviour change when the StatefulSet is genuinely absent, which is the common case
this branch was written for.
AI disclosure: I used Claude Code while investigating this. The reproduction above was
run by me on my own cluster, and I have described separately in #2056 which parts I
observed and which I only traced in the source.