OCPBUGS-84961: apply full test processing pipeline to external binary tests

[bshaw7]

Summary

• External binary tests loaded via TestBinaries.ListTests() were missing the processing pipeline that origin's built-in tests go through in InitializeOpenShiftTestsExtensionFramework()
• Without [Suite:openshift/conformance/...] tags, the conformance suite filter excludes these tests, causing the openshift/conformance suite to drop from ~4000 to ~190 tests on OCP 4.21+
• Fixes OCPBUGS-84961

Root Cause

When tests were moved from origin's built-in ginkgo suite to external extension binaries (starting in 4.20), the processing pipeline was not applied to external binary tests. Origin's built-in tests go through four processing steps in InitializeOpenShiftTestsExtensionFramework():

1. filterOutDisabledSpecs() — removes tests known to be broken
2. addEnvironmentSelectors() — adds [Skipped:] markers for platform/network/topology-specific tests
3. addLabelsToSpecs() — adds labels like [Serial] where needed
4. appendSuiteNames() — adds [Suite:openshift/conformance/...] tags

External binary tests from TestBinaries.ListTests() skipped all four steps.

Fix

Apply the same four processing steps to external binary test specs after ListTests() aggregates them. All four functions have guards to skip specs that already have the relevant annotations.

Test Plan

• Built patched openshift-tests binary from release-4.21 branch
• Deployed OCP 4.21.8 GA cluster on AWS
• Ran openshift-tests run openshift/conformance --dry-run with stock binary: 184 tests
• Ran same command with patched binary: 3,895 tests
• Unit tests pass: go test ./pkg/test/extensions/...

Generated with Claude Code

[openshift-ci-robot]

@bshaw7: This pull request references Jira Issue OCPBUGS-84961, which is invalid:

• expected the bug to target the "4.21.z" version, but no target version was set
• release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
• expected dependent Jira Issue OCPBUGS-77783 to target a version in 4.22.0, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

• External binary tests loaded via TestBinaries.ListTests() were missing the processing pipeline that origin's built-in tests go through in InitializeOpenShiftTestsExtensionFramework()
• Without [Suite:openshift/conformance/...] tags, the conformance suite filter excludes these tests, causing the openshift/conformance suite to drop from ~4000 to ~190 tests on OCP 4.21+
• Fixes OCPBUGS-84961

Root Cause

When tests were moved from origin's built-in ginkgo suite to external extension binaries (starting in 4.20), the processing pipeline was not applied to external binary tests. Origin's built-in tests go through four processing steps in InitializeOpenShiftTestsExtensionFramework():

1. filterOutDisabledSpecs() — removes tests known to be broken
2. addEnvironmentSelectors() — adds [Skipped:] markers for platform/network/topology-specific tests
3. addLabelsToSpecs() — adds labels like [Serial] where needed
4. appendSuiteNames() — adds [Suite:openshift/conformance/...] tags

External binary tests from TestBinaries.ListTests() skipped all four steps.

Fix

Apply the same four processing steps to external binary test specs after ListTests() aggregates them. All four functions have guards to skip specs that already have the relevant annotations.

Test Plan

• Built patched openshift-tests binary from release-4.21 branch
• Deployed OCP 4.21.8 GA cluster on AWS
• Ran openshift-tests run openshift/conformance --dry-run with stock binary: 184 tests
• Ran same command with patched binary: 3,895 tests
• Unit tests pass: go test ./pkg/test/extensions/...

Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 96161762-9845-4ab6-a43a-394bb20bfb45

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bshaw7
Once this PR has been reviewed and has the lgtm label, please assign xueqzhan for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[bshaw7]

/test e2e-aws-ovn

[bshaw7]

/test e2e-aws-ovn

[bshaw7]

/test e2e-aws-ovn

[openshift-ci]

@bshaw7: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-ovn	167e226	link	false	/test e2e-aws-ovn

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[bshaw7]

Writing extension test results JSON to /logs/artifacts/junit/extension_test_result_e2e__20260604-092828.json
time="2026-06-04T11:24:10Z" level=info msg="Writing duration metrics to /logs/artifacts/junit/duration-metrics-e2e_20260604-092828-autodl.json"
Suite run returned error: 49 blocking fail, 3 informing fail, 1946 pass, 0 flaky, 2052 skip (1h50m48s)
error: error running a test suite: 49 blocking fail, 3 informing fail, 1946 pass, 0 flaky, 2052 skip (1h50m48s)       

From the CI job's build-log.txt. The key line is:

49 blocking fail, 3 informing fail, 1946 pass, 0 flaky, 2052 skip. That's 4050 total tests — proving the full conformance suite ran correctly with our fix. Before the fix, it was only ~742 tests.