Skip to content

Fix DNS availability test on SNO#31428

Open
bentito wants to merge 1 commit into
openshift:mainfrom
bentito:OCPBUGS-85426-dns-upgrade-sno-fix
Open

Fix DNS availability test on SNO#31428
bentito wants to merge 1 commit into
openshift:mainfrom
bentito:OCPBUGS-85426-dns-upgrade-sno-fix

Conversation

@bentito

@bentito bentito commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

The Kubernetes 1.36 rebase changed the DNS test image from BIND 9.9 to BIND 9.18. BIND 9.18 uses connected UDP sockets, causing dig to fast-fail on ICMP port unreachable instead of timing out slowly (15s).

During an SNO reboot, there is an expected ~75s DNS outage. The fast-fail behavior causes dig to rapidly count failures (~75 instead of ~4), causing the test to drop below the 98% threshold. Example failed job.

Fix

  1. Normalize probe with +timeout=1 +retry=0: Ensures the probe loop always takes ~1-2s whether packets are silently dropped or rejected with ICMP, reducing measurement variance.
  2. Increase SNO disruption allowance to 15%: Lowers the success threshold to 84% to accommodate the expected ~75s outage during SNO reboots.

Assisted by Gemini

Summary by CodeRabbit

  • Tests
    • Updated DNS availability checks for Single Node OpenShift upgrades to accommodate expected DNS disruption during node reboots.
    • Improved probe timing consistency by using faster timeout and retry behavior.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 66393797-a3da-4e7c-aa1e-b2f930db2ee3

📥 Commits

Reviewing files that changed from the base of the PR and between 0f8993b and 18277cb.

📒 Files selected for processing (1)
  • test/e2e/upgrade/dns/dns.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • test/e2e/upgrade/dns/dns.go

Walkthrough

The DNS upgrade test increases tolerated Single Node OpenShift disruption from 1% to 15% and configures its dig probe with a one-second timeout and no retries.

Changes

SNO DNS validation

Layer / File(s) Summary
Disruption and probe behavior
test/e2e/upgrade/dns/dns.go
The SNO disruption allowance is raised to 15%, and DNS probes use +timeout=1 +retry=0 for bounded failure timing.

Estimated code review effort: 2 (Simple) | ~5 minutes

Suggested reviewers: deads2k, p0lyn0mial

🚥 Pre-merge checks | ✅ 14 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Ipv6 And Disconnected Network Test Compatibility ⚠️ Warning The DNS probe still resolves public google.com from the cluster, so the test requires external DNS connectivity and is not disconnected-compatible. Use an internal cluster DNS target/service name instead of a public hostname, or mark the test [Skipped:Disconnected] if external resolution is required.
✅ Passed checks (14 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: fixing the DNS availability test for Single Node OpenShift upgrades.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed No dynamic Ginkgo titles were added; the file has only static Name/DisplayName strings, and the changes are limited to DNS timeout and disruption thresholds.
Test Structure And Quality ✅ Passed The changes preserve the existing Setup/Test/Teardown pattern, use PodStartTimeout for readiness, and create only a namespaced DaemonSet cleaned up with the namespace.
Microshift Test Compatibility ✅ Passed No new Ginkgo spec was added; the existing upgrade test uses core K8s APIs only and no MicroShift-unsupported OpenShift APIs or assumptions.
Single Node Openshift (Sno) Test Compatibility ✅ Passed No new Ginkgo test was added; the change only adjusts DNS probe timing and an SNO threshold, with an existing IsSingleNode check already present.
Topology-Aware Scheduling Compatibility ✅ Passed The only change is a DNS e2e test command/threshold tweak; the DaemonSet spec has no node selectors, affinity, spread constraints, or other topology-sensitive scheduling logic.
Ote Binary Stdout Contract ✅ Passed No main/init/suite-level stdout writers were added; the only new echo is inside the DaemonSet pod command, not the test binary process.
No-Weak-Crypto ✅ Passed The DNS test changes only dig timeout/disruption settings; no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret comparisons appear in the modified file.
Container-Privileges ✅ Passed The only changed file is dns.go, and the patch only adjusts dig timing and SNO threshold; no privileged, hostPID/Network/IPC, SYS_ADMIN, or allowPrivilegeEscalation fields were added.
No-Sensitive-Data-In-Logs ✅ Passed The diff only adjusts DNS probe timing/thresholds; logged output is just timestamps, fail markers, and DNS IPs, with no passwords, tokens, PII, or other sensitive data.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested a review from deads2k July 24, 2026 13:17
@openshift-ci

openshift-ci Bot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bentito
Once this PR has been reviewed and has the lgtm label, please assign petr-muller for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci
openshift-ci Bot requested a review from p0lyn0mial July 24, 2026 13:17
@openshift-ci openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Jul 24, 2026
@jubittajohn

Copy link
Copy Markdown
Contributor

Pulled in the changes from this PR into the rebase PR #31237.
Thank you @bentito for looking into this.

Keeping this PR open until we test/verify the fix, by running payload jobs against the rebase PR.

Signed-off-by: Brett Tofel <btofel@redhat.com>
@bentito
bentito force-pushed the OCPBUGS-85426-dns-upgrade-sno-fix branch from 0f8993b to 18277cb Compare July 24, 2026 19:00
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn-upgrade-rollback

@bentito

bentito commented Jul 25, 2026

Copy link
Copy Markdown
Contributor Author

/retest

1 similar comment
@bentito

bentito commented Jul 25, 2026

Copy link
Copy Markdown
Contributor Author

/retest

@openshift-ci

openshift-ci Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

@bentito: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-trt

openshift-trt Bot commented Jul 25, 2026

Copy link
Copy Markdown

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New Test Risks for sha: 18277cb

Job Name New Test Risk
pull-ci-openshift-origin-main-e2e-gcp-csi High - "External Storage [Driver: pd.csi.storage.gke.io] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand Verify if offline PVC expansion works" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-gcp-csi High - "External Storage [Driver: pd.csi.storage.gke.io] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited and the pod is re-created on the same node after controller resize is finished" is a new test that was not present in all runs against the current commit.
pull-ci-openshift-origin-main-e2e-gcp-csi High - "External Storage [Driver: pd.csi.storage.gke.io] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited while pod is using it" is a new test that was not present in all runs against the current commit.

New tests seen in this PR at sha: 18277cb

  • "External Storage [Driver: pd.csi.storage.gke.io] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand Verify if offline PVC expansion works" [Total: 3, Pass: 3, Fail: 0, Flake: 0]
  • "External Storage [Driver: pd.csi.storage.gke.io] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited and the pod is re-created on the same node after controller resize is finished" [Total: 3, Pass: 3, Fail: 0, Flake: 0]
  • "External Storage [Driver: pd.csi.storage.gke.io] [Testpattern: Dynamic PV (xfs)] [Slow] volume-expand should resize volume when PVC is edited while pod is using it" [Total: 3, Pass: 3, Fail: 0, Flake: 0]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants