Implement rollout halting and degraded node surfacing#96
Merged
Conversation
alicefr
reviewed
Jul 10, 2026
Collaborator
|
Overall looks good, I only had a question and small nits. Thanks for the work! Love the test cases! |
jlebon
force-pushed
the
pr/more-3b
branch
2 times, most recently
from
July 15, 2026 03:16
2114f0e to
676ee01
Compare
Collaborator
Author
|
Thanks for the review! Updated for comments. |
alicefr
reviewed
Jul 15, 2026
| } | ||
|
|
||
| // syncRolloutHaltedCondition sets Degraded/RolloutHalted on the pool. It | ||
| // implicitly takes priority over NodeDegraded (which checks for existing |
Collaborator
There was a problem hiding this comment.
What about if there is a NodeConflict, this will be silently wiped out? Also worth mentioning InvalidSpec and TagResolutionError even if in the code will return earlier in the reconcile
Collaborator
Author
There was a problem hiding this comment.
OK took a much larger swing at this in a follow-up: #112
Collaborator
|
Some minors nits and a question about the NodeConflict condition, but I think it is done on purpose, and it should only be integrated in the function comment. |
alicefr
previously approved these changes
Jul 15, 2026
When a daemon reports Degraded=True on a BootcNode, the controller now sets the pool's Degraded condition with reason NodeDegraded and a message listing the affected node names. Assisted-by: Pi (Claude Opus 4.6) Signed-off-by: Jonathan Lebon <jonathan@jlebon.com>
Add a new Degraded condition reason for when the controller stops assigning reboot slots because 2 or more nodes in reboot slots are unhealthy (Degraded or not Ready after rebooting into the target image). No behavioral changes yet; the constant is wired up in a follow-up commit. Assisted-by: Pi (Claude Opus 4.6) Signed-off-by: Jonathan Lebon <jonathan@jlebon.com>
When 2 or more nodes occupying reboot slots are unhealthy (BootcNode Degraded after booting the target digest, or K8s Node not Ready after update), the controller stops assigning new reboot slots and sets the pool Degraded with reason RolloutHalted. A single unhealthy node might be a hardware issue, but two suggest a bad image. The rollout resumes automatically once the unhealthy count drops below the threshold (e.g. nodes recover or are manually fixed). Assisted-by: Pi (Claude Opus 4.6) Signed-off-by: Jonathan Lebon <jonathan@jlebon.com>
The controller concurrently patches spec.unschedulable on the same Node, which bumps the resourceVersion. A Status().Update fails in that case because it checks the top-level resourceVersion. Switch to Status().Patch which avoids the conflict entirely. Assisted-by: Pi (Claude Opus 4.6) Signed-off-by: Jonathan Lebon <jonathan@jlebon.com>
To match the other simulation code in that file and also because it seems weird to wipe out the `NodeDegraded` type condition type entirely. Signed-off-by: Jonathan Lebon <jonathan@jlebon.com>
This was done in 21c3169 ("daemon: add StatusWatcher for fsnotify + polling"). Signed-off-by: Jonathan Lebon <jonathan@jlebon.com>
alicefr
approved these changes
Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Various patches that add more error-handling as described in the 3b implementation plan.