fix: stop controller error-ratio alerts flapping

[aflor024]

Summary

ControllerReconcileErrorRatioCritical has been firing and resolving on a loop rather than latching — roughly 100 fire/resolve pairs into #alerts-production-critical in 14h for a single unchanged fault, and because severity: critical also matches the grafana-oncall route, it pages on-call on every cycle. The underlying fault deserves one page, not a hundred.

Measuring prod turned up two independent flap mechanisms, not one:

• Sparse series. instance-projector reconciles a median of 4 objects per 10 minutes. 28 of 85 windows contained no increments at all, so the denominator went to zero, 0/0 dropped the series, and Alertmanager resolved — then the next reconcile restarted for: 10m and it fired again. It never once crossed back under the threshold (0 of 57 data-bearing windows), so this is not the ratio dipping. A 30-minute range keeps the series continuous at that volume.
• Dilution. A constant error rate against a variable success rate crosses the threshold in both directions as unrelated healthy traffic rises and falls, with the controller no healthier either way — the dnsrecordset-powerdns and staging gateway case already described on ControllerReconcileErrorRatio flaps with unrelated traffic volume and omits the cluster #326. keep_firing_for: 30m holds the alert through those dips.

Both changes are load-bearing and independently guarded: reverting the range fails the sparse fixture, and removing keep_firing_for fails the dilution fixture.

The sparse-series half is not only a noise fix. datum-cloud/infra#3742 diagnosed the same mechanism from the opposite end: GrafanaAlertRuleGroup, GrafanaContactPoint, GrafanaFolder and GrafanaNotificationPolicy on prod reconcile about once every 15 minutes and have been failing 100% with zero successes for over 30 days, yet ControllerReconcileErrorRatioCritical sits permanently in pending for them and has never fired — a reconcile period longer than the evaluation window means for: resets on every reconcile and the rule cannot latch. Every alert-rule, contact-point and notification-routing change committed to Git in that window silently never reached Grafana. The 30-minute range is what makes that condition alertable at all, so the same change that quiets a flapping controller also surfaces a month-old silent failure.

Warning

This raises the unalertable threshold from a 10-minute to a 30-minute reconcile period; it does not remove it. A controller reconciling less often than every 30 minutes is still invisible to this rule. The general class infra#3742 describes stays open.

Grouping now includes cluster and namespace, so a page says where the failing controller runs instead of only naming the reconciler — #326's second ask.

Note

The rule still has no job or namespace selector, so it evaluates every controller-runtime binary scraped into the cluster. That is how an nso-slo-owned alert ends up paging about instance-projector, which runs in compute-system. Scoping it to NSO would strand the controllers above with no coverage at all, so that ownership question is left open on #326 rather than settled here.

The fault behind the current pages is dead data, not an outage — 3 orphaned Instances in one project namespace, tracked as datum-cloud/compute#194. This PR stops the paging pattern; it does not clean those up.

task test-prometheus-rules was not referenced by any workflow, so the fixtures this issue asks for would not have guarded anything. Added a job that runs them.

Test plan

• task test-prometheus-rules — all three suites pass
• Sparse-series fixture fails when the range is reverted to 10m
• Dilution fixture fails when keep_firing_for is removed
• promtool confirms a held alert retains its last annotations, and that the hold is bounded (resolves once keep_firing_for lapses)
• New CI job runs green on this PR

Related to #326

[cla-assistant]

All committers have signed the CLA.

[cla-assistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}