feat: Add reconciliation retry observability metrics#288
Conversation
Signed-off-by: Rawad Hossain <rawad.hossain00@gmail.com>
kubernetes-prow
Bot
added
the
kind/feature
✅ Deploy Preview for node-readiness-controller canceled.
|
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: rawadhossain The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
kubernetes-prow
Bot
added
the
needs-ok-to-test
|
Hi @rawadhossain. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Tip We noticed you've done this a few times! Consider joining the org to skip this step and gain Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
kubernetes-prow
Bot
added
size/XL
|
/assign |
AvineshTripathi
left a comment
AvineshTripathi
left a comment
There was a problem hiding this comment.
I did some initial review and I had some questions:
- Are we planning to visualize these metrics? If so, then how?
- What type of answer will an operator get from these? I feel these are metrics for the controller itself, which will tell how controller is behaving in concurrent env
| // Update rule status | ||
| if err := r.Controller.updateRuleStatus(ctx, rule); err != nil { | ||
| log.Error(err, "Failed to update rule status", "rule", rule.Name) | ||
| metrics.ReconcileRequeue.WithLabelValues(rule.Name, "status_update_error").Inc() |
There was a problem hiding this comment.
there are other places too in Reconcile where we requeue for e.g. https://github.com/kubernetes-sigs/node-readiness-controller/blob/main/internal/controller/nodereadinessrule_controller.go#L151
maybe a defer function would make more sense here. Thoughts?
2 participants
Description
This PR adds two new metrics
What each metric does
node_readiness_api_conflicts_totalTracks how many API conflicts occur during controller operations. Previously, only conflicts that exhausted all retries were visible through
node_readiness_failures_total. If a retry succeeded, there was no signal that a conflict had happened. This metric makes those retries visible and helps identify API server contention earlier.node_readiness_reconcile_requeue_totalTracks why a rule reconciliation is requeued. Right now, we know a rule is being requeued, but we don't know which stage caused it. This metric records the reason for the requeue, making it easier to identify where reconciliation is failing.
The metric is labeled by
reason, currently covering:status_update_errorcleanup_errortaint_cleanup_errorRelated to Issue #182
Type of Change
/kind feature
Testing
Checklist
make testpassesmake lintpasses