Skip to content

Controllers in inactive groups still update group ID and replace premature check with controller SDK backoff. - #809

Open
philbrookes wants to merge 3 commits into
mainfrom
gh-780
Open

Controllers in inactive groups still update group ID and replace premature check with controller SDK backoff.#809
philbrookes wants to merge 3 commits into
mainfrom
gh-780

Conversation

@philbrookes

@philbrookes philbrookes commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

This PR combines three related changes for the DNS groups feature:

1. Refactor: replace manual reconcile backoff with SDK rate limiter (#778)

  • Replaces manual nextReconcileAt/lastReconcileAt tracking with the SDK's built-in ItemExponentialFailureRateLimiter
  • Removes RequeueTime and WriteCounter from DNSRecordStatus, simplifying the CRD
  • Adds specOrDeletionChangedPredicate to filter watch events to spec changes (generation changed) and deleting objects, preventing status-only updates from causing tight reconciliation loops
  • Uses defaultValidationRequeue consistently for pre-deletion status transitions

2. Feat: always write groupID to provider registry for inactive controllers (#779)

  • Adds a registryOnly mode to applyChanges that strips Creates, Deletes, and target-changing Updates from the plan, allowing only TXT registry label updates (e.g. groupID) to be applied
  • Inactive group controllers now call applyChanges(registryOnly=true) to write accurate groupID labels to TXT registry entries without modifying DNS records
  • Fixes stale group values in TXT records that prevented the active controller's unpublishInactiveGroups from correctly identifying and cleaning up inactive group records
  • Propagates errors from inactive registry updates through status conditions for faster requeue on failure

3. Test: integration test for groupID registry self-correction (#780)

  • Adds a test that tampers TXT registry group labels on an inactive controller's authoritative record and verifies the inactive controller corrects them via registryOnly applyChanges
  • Adds tamperRegistryGroup helper to simulate stale/incorrect group labels by modifying authoritative record TXT endpoints
  • Updates filterDNSEndpoints to clear Labels for stable endpoint comparisons

Fixes #778, #779, #780

Test plan

  • Unit tests pass (make test-unit)
  • Integration tests pass (make test-integration) — all 46 specs
  • New integration test verifies inactive controllers write groupID to registry without modifying zone DNS records
  • New integration test verifies inactive controllers correct tampered registry group entries

Summary by CodeRabbit

  • Breaking Changes

    • Removed the queuedAt and validFor fields from DNS record status information and documentation.
    • Removed the --valid-for configuration option and corresponding environment variable.
  • Improvements

    • Reconciliation now uses rate-limited exponential backoff between configured minimum and maximum requeue intervals.
    • Added validation for requeue interval limits.
    • Inactive grouped records keep registry entries synchronised without altering authoritative DNS endpoints.
    • Improved handling of reconciliation conflicts, registry update failures and unnecessary status-only updates.

@coderabbitai

coderabbitai Bot commented Aug 4, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 95b0d916-076f-43c5-a11f-aa8c35c62f5c

📥 Commits

Reviewing files that changed from the base of the PR and between cda782a and e28bfa8.

📒 Files selected for processing (20)
  • CLAUDE.md
  • api/v1alpha1/dnsrecord_types.go
  • api/v1alpha1/zz_generated.deepcopy.go
  • bundle/manifests/dns-operator.clusterserviceversion.yaml
  • bundle/manifests/kuadrant.io_dnsrecords.yaml
  • charts/dns-operator/templates/manifests.yaml
  • cmd/main.go
  • config/crd/bases/kuadrant.io_dnsrecords.yaml
  • docs/RFC/distributed-dns/dnsrecord-lifecycle.md
  • docs/reference/dnsrecord.md
  • internal/controller/base_dnsrecord_reconciler.go
  • internal/controller/dnshealthcheckprobe_reconciler.go
  • internal/controller/dnsrecord_controller.go
  • internal/controller/dnsrecord_controller_runtime_reconfiguration_test.go
  • internal/controller/dnsrecord_controller_test.go
  • internal/controller/dnsrecord_groups_test.go
  • internal/controller/helper_test.go
  • internal/controller/remote_dnsrecord_controller.go
  • internal/controller/suite_test.go
  • internal/external-dns/registry/group.go
💤 Files with no reviewable changes (6)
  • api/v1alpha1/zz_generated.deepcopy.go
  • bundle/manifests/kuadrant.io_dnsrecords.yaml
  • config/crd/bases/kuadrant.io_dnsrecords.yaml
  • internal/controller/dnsrecord_controller_runtime_reconfiguration_test.go
  • charts/dns-operator/templates/manifests.yaml
  • docs/reference/dnsrecord.md
🚧 Files skipped from review as they are similar to previous changes (12)
  • internal/controller/remote_dnsrecord_controller.go
  • docs/RFC/distributed-dns/dnsrecord-lifecycle.md
  • CLAUDE.md
  • internal/controller/helper_test.go
  • internal/controller/dnsrecord_groups_test.go
  • internal/controller/dnsrecord_controller_test.go
  • cmd/main.go
  • api/v1alpha1/dnsrecord_types.go
  • bundle/manifests/dns-operator.clusterserviceversion.yaml
  • internal/controller/dnshealthcheckprobe_reconciler.go
  • internal/controller/dnsrecord_controller.go
  • internal/controller/suite_test.go

📝 Walkthrough

Walkthrough

The change removes manual DNS record validity tracking and the --valid-for configuration. Controllers now use bounded exponential rate limiting. Inactive grouped records update registry metadata without changing authoritative DNS endpoints.

Changes

DNS reconciliation and registry updates

Layer / File(s) Summary
Status and configuration contracts
api/v1alpha1/..., config/crd/..., bundle/manifests/..., charts/dns-operator/..., cmd/main.go, docs/..., CLAUDE.md
Removes QueuedAt and ValidFor from status contracts, schemas, and documentation. Removes the valid-for flag and environment variable. Renames ValidityDuration to RequeueDuration and validates requeue bounds.
Rate-limited reconciliation flow
internal/controller/dnsrecord_controller.go, internal/controller/base_dnsrecord_reconciler.go, internal/controller/remote_dnsrecord_controller.go, internal/controller/dnshealthcheckprobe_reconciler.go
Uses explicit requeue results, typed exponential rate limiting, filtered watches, and configured validation intervals.
Inactive-group registry reconciliation
internal/external-dns/registry/group.go, internal/controller/dnsrecord_controller.go, internal/controller/remote_dnsrecord_controller.go, internal/controller/base_dnsrecord_reconciler.go
Preserves registry entries for inactive groups and suppresses changes to authoritative DNS endpoints.
Validation and test updates
internal/controller/dnsrecord_groups_test.go, internal/controller/helper_test.go, internal/controller/dnsrecord_controller_test.go, internal/controller/suite_test.go
Adds registry inspection and label-correction helpers. Tests validate inactive-group registry handling and updated controller setup calls.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant DNSRecordReconciler
  participant RateLimiter
  participant GroupRegistry
  participant DNSProvider
  participant KubernetesAPI
  DNSRecordReconciler->>RateLimiter: select bounded requeue delay
  RateLimiter->>DNSRecordReconciler: return reconcile result
  DNSRecordReconciler->>GroupRegistry: apply inactive-group registry updates
  GroupRegistry->>DNSProvider: suppress endpoint creates and deletes
  DNSRecordReconciler->>KubernetesAPI: persist status and WriteCounter
Loading

Possibly related PRs

Poem

A rabbit trims the validity queue,
Rate limits set the timing true.
Registry labels stay in place,
DNS endpoints keep their space.
Backoff guides each retry.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies inactive group label updates and replacement of the premature reconcile check with SDK backoff.
Linked Issues check ✅ Passed The changes rework premature reconciliation and allow specification, deletion, and group-related updates to trigger timely processing as required by [#778].
Out of Scope Changes check ✅ Passed The registry, controller, API, documentation, and test changes support the reconciliation and inactive-group propagation objectives.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch gh-780

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/controller/dnsrecord_controller.go`:
- Around line 439-445: Update DNSRecordReconciler.SetupWithManager to enforce
the supported requeue bounds before assigning defaultRequeueTime,
defaultValidationRequeue, or configuring RateLimiter: reject or clamp minRequeue
values below 5 seconds and maxRequeue values above 15 minutes, while preserving
the existing min/max ordering validation.
- Around line 503-524: Update specOrDeletionChangedPredicate.Update to accept
deletion only on the transition from a nil old deletion timestamp to a non-nil
new timestamp. Keep generation-change handling unchanged, and return false for
subsequent updates while deletion is already in progress so explicit
RequeueAfter results drive the deletion state machine.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 558d3689-7f33-48c7-a3ef-a1c5b5062334

📥 Commits

Reviewing files that changed from the base of the PR and between cda782a and 35c120a.

📒 Files selected for processing (19)
  • CLAUDE.md
  • api/v1alpha1/dnsrecord_types.go
  • api/v1alpha1/zz_generated.deepcopy.go
  • bundle/manifests/dns-operator.clusterserviceversion.yaml
  • bundle/manifests/kuadrant.io_dnsrecords.yaml
  • charts/dns-operator/templates/manifests.yaml
  • cmd/main.go
  • config/crd/bases/kuadrant.io_dnsrecords.yaml
  • docs/RFC/distributed-dns/dnsrecord-lifecycle.md
  • docs/reference/dnsrecord.md
  • internal/controller/base_dnsrecord_reconciler.go
  • internal/controller/dnshealthcheckprobe_reconciler.go
  • internal/controller/dnsrecord_controller.go
  • internal/controller/dnsrecord_controller_runtime_reconfiguration_test.go
  • internal/controller/dnsrecord_controller_test.go
  • internal/controller/dnsrecord_groups_test.go
  • internal/controller/helper_test.go
  • internal/controller/remote_dnsrecord_controller.go
  • internal/controller/suite_test.go
💤 Files with no reviewable changes (7)
  • internal/controller/dnsrecord_controller_runtime_reconfiguration_test.go
  • docs/reference/dnsrecord.md
  • api/v1alpha1/zz_generated.deepcopy.go
  • bundle/manifests/kuadrant.io_dnsrecords.yaml
  • charts/dns-operator/templates/manifests.yaml
  • config/crd/bases/kuadrant.io_dnsrecords.yaml
  • api/v1alpha1/dnsrecord_types.go

Comment thread internal/controller/dnsrecord_controller.go
Comment thread internal/controller/dnsrecord_controller.go
@philbrookes
philbrookes force-pushed the gh-780 branch 2 times, most recently from 6bf5d40 to 6fb3e0d Compare August 4, 2026 13:47
@philbrookes philbrookes moved this to Ready For Review in Kuadrant Aug 4, 2026
@philbrookes philbrookes changed the title feat: groupID registry writes for inactive controllers with integration tests Controllers in inactive groups still update group ID and replace premature check with controller SDK backoff. Aug 5, 2026
@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

Comment thread internal/controller/base_dnsrecord_reconciler.go Outdated
Comment thread internal/controller/base_dnsrecord_reconciler.go Outdated
Comment thread internal/controller/base_dnsrecord_reconciler.go Outdated
Comment thread internal/controller/dnsrecord_controller.go Outdated
@mikenairn

Copy link
Copy Markdown
Member

This has the same concern I raised in #733, with recordReceivedPrematurely removed, AwaitingValidation is set and then immediately cleared on the next reconcile (5 seconds later). There's no meaningful validation window anymore. What purpose does AwaitingValidation serve with these changes?

@philbrookes

Copy link
Copy Markdown
Contributor Author

This has the same concern I raised in #733, with recordReceivedPrematurely removed, AwaitingValidation is set and then immediately cleared on the next reconcile (5 seconds later). There's no meaningful validation window anymore. What purpose does AwaitingValidation serve with these changes?

This is true, I'd completely forgotten that issue. I wonder if we could have the awaiting validation stay true until the last write was over a X time ago and there are no changes made in this reconcile?

@philbrookes

Copy link
Copy Markdown
Contributor Author

This has the same concern I raised in #733, with recordReceivedPrematurely removed, AwaitingValidation is set and then immediately cleared on the next reconcile (5 seconds later). There's no meaningful validation window anymore. What purpose does AwaitingValidation serve with these changes?

This is true, I'd completely forgotten that issue. I wonder if we could have the awaiting validation stay true until the last write was over a X time ago and there are no changes made in this reconcile?

I see that this is exactly what we discussed before. I'm not sure if the validation loop still has enough value to be worth the effort of recoding it that way though.

Maybe the better answer here is to remove the concept of a DNSRecord verification step?

@philbrookes

Copy link
Copy Markdown
Contributor Author

This has the same concern I raised in #733, with recordReceivedPrematurely removed, AwaitingValidation is set and then immediately cleared on the next reconcile (5 seconds later). There's no meaningful validation window anymore. What purpose does AwaitingValidation serve with these changes?

This is true, I'd completely forgotten that issue. I wonder if we could have the awaiting validation stay true until the last write was over a X time ago and there are no changes made in this reconcile?

I see that this is exactly what we discussed before. I'm not sure if the validation loop still has enough value to be worth the effort of recoding it that way though.

Maybe the better answer here is to remove the concept of a DNSRecord verification step?

Mike and I discussed this offline, and came to the conclusion that we could implement a backoff using the lastTransitionTime of the ready condition, and avoid using the Requeue: true.

I will implement and push up some changes to reflect this.

Remove recordReceivedPrematurely, exponentialRequeueTime, and the
--valid-for flag. Use time.Since(LastTransitionTime) to derive a
natural ramp-up from defaultValidationRequeue to defaultRequeueTime,
removing the need for explicit backoff state. Add a
specOrDeletionChangedPredicate to prevent status-only watch events
from bypassing the calculated RequeueAfter.

Remove QueuedAt and ValidFor status fields (WriteCounter kept for
the dns_provider_write_counter Prometheus metric).

Signed-off-by: Phil Brookes <pbrookes@redhat.com>

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED
…ers (#779)

Inactive group controllers now call publishRecord to update TXT
registry labels (e.g. groupID) without creating, deleting, or
modifying DNS record targets. The GroupRegistry filters changes
based on its IsActive field — when inactive, only label-change
Updates are applied. This ensures unpublishInactiveGroups can
correctly identify inactive group records via accurate TXT entries.

Signed-off-by: Phil Brookes <pbrookes@redhat.com>

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED
Adds a test that tampers TXT registry group labels on an inactive
controller's authoritative record and verifies the inactive controller
corrects them via registryOnly applyChanges without modifying zone DNS
records.

Signed-off-by: Phil Brookes <pbrookes@redhat.com>

rh-pre-commit.version: 2.3.2
rh-pre-commit.check-secrets: ENABLED
@coderabbitai

coderabbitai Bot commented Aug 7, 2026

Copy link
Copy Markdown

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Ready For Review

2 participants