From 11c5164251849fa57295e47bbe6cc8622cb7e9cd Mon Sep 17 00:00:00 2001 From: Nikhil Prajapati Date: Tue, 2 Jun 2026 07:04:58 +0530 Subject: [PATCH 1/2] OCPSTRAT-2485: Document release.openshift.io/feature-gate annotation and CVO manifest inclusion Add a new section "Understanding feature-gate-based manifest inclusion" to the feature gates guide documenting the CVO capability introduced in OCP 4.22 that conditionally includes or excludes release payload manifests based on which feature gates are enabled. Covers: - How CVO uses the release.openshift.io/feature-gate annotation - Annotation syntax (positive, negated, AND logic) - How to discover gated manifests in a release using oc adm release extract - Orphaned resource behavior when a gate is disabled - CVO log messages for manifest inclusion/exclusion Co-authored-by: Cursor --- ...nodes-cluster-enabling-features-about.adoc | 5 + ...-enabling-features-manifest-inclusion.adoc | 138 ++++++++++++++++++ .../nodes-cluster-enabling-features.adoc | 2 + 3 files changed, 145 insertions(+) create mode 100644 modules/nodes-cluster-enabling-features-manifest-inclusion.adoc diff --git a/modules/nodes-cluster-enabling-features-about.adoc b/modules/nodes-cluster-enabling-features-about.adoc index 61665dac8a94..bab24a7f6059 100644 --- a/modules/nodes-cluster-enabling-features-about.adoc +++ b/modules/nodes-cluster-enabling-features-about.adoc @@ -128,6 +128,11 @@ The following Technology Preview features are enabled by this feature set: See the _Additional resources_ sections for information on some of these features. +[NOTE] +==== +Starting in {product-title} 4.22, the Cluster Version Operator (CVO) can conditionally include or exclude release payload manifests based on which feature gates are enabled. For details, see xref:../../nodes/clusters/nodes-cluster-enabling-features.adoc#nodes-cluster-enabling-features-manifest-inclusion_nodes-cluster-enabling[Understanding feature-gate-based manifest inclusion]. +==== + //// Do not document per Derek Carr: https://github.com/openshift/api/pull/370#issuecomment-510632939 |`CustomNoUpgrade` ^[2]^ diff --git a/modules/nodes-cluster-enabling-features-manifest-inclusion.adoc b/modules/nodes-cluster-enabling-features-manifest-inclusion.adoc new file mode 100644 index 000000000000..e42390bc837c --- /dev/null +++ b/modules/nodes-cluster-enabling-features-manifest-inclusion.adoc @@ -0,0 +1,138 @@ +// Module included in the following assemblies: +// +// * nodes/clusters/nodes-cluster-enabling-features.adoc + +:_mod-docs-content-type: CONCEPT +[id="nodes-cluster-enabling-features-manifest-inclusion_{context}"] += Understanding feature-gate-based manifest inclusion + +[role="_abstract"] +Starting in {product-title} 4.22, the Cluster Version Operator (CVO) uses the `release.openshift.io/feature-gate` annotation on release payload manifests to conditionally include or exclude cluster resources based on which feature gates are enabled. When you enable or disable a feature gate, CVO automatically re-evaluates all annotated manifests and applies or stops managing the affected resources. + +[id="nodes-cluster-enabling-features-manifest-inclusion-how_{context}"] +== How feature-gate-based manifest inclusion works + +Each release payload contains manifests for all cluster components, including components that are gated behind specific feature gates. CVO reads the `release.openshift.io/feature-gate` annotation on each manifest to determine whether to include it during reconciliation: + +* If the annotation specifies a feature gate that is **enabled** on the cluster, CVO includes the manifest and applies the resource. +* If the annotation specifies a feature gate that is **not enabled**, CVO excludes the manifest and does not apply the resource. +* If a manifest has no `release.openshift.io/feature-gate` annotation, CVO always includes it. + +When you toggle a feature gate at runtime using the `FeatureGate` custom resource (CR), CVO detects the change, re-reads the payload, and re-evaluates all gated manifests. Resources whose gate conditions are now met are applied. Resources whose gate conditions are no longer met are no longer managed by CVO. + +[id="nodes-cluster-enabling-features-manifest-inclusion-syntax_{context}"] +== Annotation syntax + +The `release.openshift.io/feature-gate` annotation supports the following formats: + +.Annotation syntax for feature-gate-based manifest inclusion +[cols="1a,2a,2a",options="header"] +|=== +|Format |Meaning |Example + +|`GateName` +|Include this manifest only when `GateName` is enabled. +|`release.openshift.io/feature-gate: ClusterAPIMachineManagement` + +|`-GateName` +|Include this manifest only when `GateName` is disabled. This is used for legacy resources that should be removed when a new feature replaces them. +|`release.openshift.io/feature-gate: -ClusterAPIMachineManagement` + +|`-Gate1,-Gate2` +|Include this manifest only when **all** listed gates are disabled (AND logic). The manifest is excluded if any of the listed gates is enabled. +|`release.openshift.io/feature-gate: -ClusterAPIMachineManagement,-MachineAPIMigration` +|=== + +[NOTE] +==== +The annotation name is singular: `feature-gate` (not `feature-gates`). Using the plural form will have no effect. +==== + +[id="nodes-cluster-enabling-features-manifest-inclusion-gated_{context}"] +== Identifying gated manifests in a release + +You can discover which manifests in a release payload are gated behind feature gates by extracting the payload and searching for the annotation. + +.Prerequisites + +* You have installed the {oc-first}. +* You have access to the release image for your cluster version. + +.Procedure + +. Extract the release payload manifests to a local directory: ++ +[source,terminal] +---- +$ oc adm release extract --to=/tmp/release-manifests \ + $(oc get clusterversion version -o jsonpath='{.status.desired.image}') +---- + +. Search for manifests with the `release.openshift.io/feature-gate` annotation: ++ +[source,terminal] +---- +$ grep -rl "release.openshift.io/feature-gate" /tmp/release-manifests/ +---- + +. To see which feature gates each manifest requires, display the annotation values: ++ +[source,terminal] +---- +$ grep -r "release.openshift.io/feature-gate" /tmp/release-manifests/ \ + | sed 's|/tmp/release-manifests/||' +---- ++ +.Example output +[source,terminal] +---- +0000_30_cluster-api_11_deployment.yaml: release.openshift.io/feature-gate: ClusterAPIMachineManagement +0000_30_cluster-api_04_crd.core-cluster-api.yaml: release.openshift.io/feature-gate: -ClusterAPIMachineManagement +0000_20_crd-compatibility-checker_11_deployment.yaml: release.openshift.io/feature-gate: CRDCompatibilityRequirementOperator +---- + +[id="nodes-cluster-enabling-features-manifest-inclusion-orphaned_{context}"] +== Orphaned resources when a gate is disabled + +When CVO stops managing a manifest because its feature gate condition is no longer met, the previously applied resources **remain on the cluster**. CVO does not delete them. + +For example, if you enable the `CRDCompatibilityRequirementOperator` feature gate, CVO creates the associated namespace, deployment, CRD, and RBAC resources. If you later remove `CRDCompatibilityRequirementOperator` from `customNoUpgrade.enabled`, CVO stops managing those resources but does not delete them. The namespace, deployment, and other resources continue to exist. + +[IMPORTANT] +==== +If you disable a feature gate after enabling it, you must manually remove any orphaned resources that are no longer needed. Use `oc delete` to remove the namespace and other resources that were created by the gated manifests. +==== + +[id="nodes-cluster-enabling-features-manifest-inclusion-logs_{context}"] +== Viewing CVO manifest inclusion logs + +CVO logs information about excluded and included gated manifests. You can view these messages to understand which manifests are affected by your feature gate configuration. + +.Procedure + +. View CVO logs for feature gate manifest exclusion messages: ++ +[source,terminal] +---- +$ oc logs -n openshift-cluster-version deployment/cluster-version-operator \ + | grep "feature gate" +---- ++ +.Example output for excluded manifests +[source,terminal] +---- +I0601 04:02:58.501447 1 payload.go:217] excluding Filename: "0000_30_cluster-api_11_deployment.yaml" + Group: "apps" Kind: "Deployment" Namespace: "openshift-cluster-api" + Name: "capi-controllers": feature gate ClusterAPIMachineManagement is required but not enabled +---- ++ +.Example output when a gated manifest is included because the gate is enabled +[source,terminal] +---- +I0601 04:54:27.510394 1 payload.go:217] excluding Filename: "0000_30_cluster-api_04_crd.core-cluster-api.yaml" + Group: "apiextensions.k8s.io" Kind: "CustomResourceDefinition" + Name: "ipaddressclaims.ipam.cluster.x-k8s.io": + feature gate ClusterAPIMachineManagement is enabled but manifest requires it to be disabled +---- ++ +The second message indicates a manifest that is annotated with `-ClusterAPIMachineManagement` (negated) and is being excluded because the gate is now enabled. This is expected behavior for legacy resources that are superseded by the feature. diff --git a/nodes/clusters/nodes-cluster-enabling-features.adoc b/nodes/clusters/nodes-cluster-enabling-features.adoc index 1c0d2e71a8d0..68d318bdca99 100644 --- a/nodes/clusters/nodes-cluster-enabling-features.adoc +++ b/nodes/clusters/nodes-cluster-enabling-features.adoc @@ -17,6 +17,8 @@ include::modules/nodes-cluster-enabling-features-console.adoc[leveloffset=+1] include::modules/nodes-cluster-enabling-features-cli.adoc[leveloffset=+1] +include::modules/nodes-cluster-enabling-features-manifest-inclusion.adoc[leveloffset=+1] + [role="_additional-resources"] [id="additional-resources_nodes-cluster-enabling"] == Additional resources From c5e755367355921a241b4bf526152507294a1dd8 Mon Sep 17 00:00:00 2001 From: Nikhil Prajapati Date: Tue, 2 Jun 2026 07:32:33 +0530 Subject: [PATCH 2/2] Fix Vale linter errors: remove xref from module, convert procedure blocks to prose - Remove xref NOTE from about.adoc module (NoXrefInModules, ConceptLink) - Move cross-reference to assembly Additional resources section - Convert .Prerequisites and .Procedure block titles to prose in the CONCEPT module (AsciiDocDITA.BlockTitle) Co-authored-by: Cursor --- ...nodes-cluster-enabling-features-about.adoc | 5 --- ...-enabling-features-manifest-inclusion.adoc | 42 ++++++++----------- .../nodes-cluster-enabling-features.adoc | 2 + 3 files changed, 20 insertions(+), 29 deletions(-) diff --git a/modules/nodes-cluster-enabling-features-about.adoc b/modules/nodes-cluster-enabling-features-about.adoc index bab24a7f6059..61665dac8a94 100644 --- a/modules/nodes-cluster-enabling-features-about.adoc +++ b/modules/nodes-cluster-enabling-features-about.adoc @@ -128,11 +128,6 @@ The following Technology Preview features are enabled by this feature set: See the _Additional resources_ sections for information on some of these features. -[NOTE] -==== -Starting in {product-title} 4.22, the Cluster Version Operator (CVO) can conditionally include or exclude release payload manifests based on which feature gates are enabled. For details, see xref:../../nodes/clusters/nodes-cluster-enabling-features.adoc#nodes-cluster-enabling-features-manifest-inclusion_nodes-cluster-enabling[Understanding feature-gate-based manifest inclusion]. -==== - //// Do not document per Derek Carr: https://github.com/openshift/api/pull/370#issuecomment-510632939 |`CustomNoUpgrade` ^[2]^ diff --git a/modules/nodes-cluster-enabling-features-manifest-inclusion.adoc b/modules/nodes-cluster-enabling-features-manifest-inclusion.adoc index e42390bc837c..98c95d2fa6b4 100644 --- a/modules/nodes-cluster-enabling-features-manifest-inclusion.adoc +++ b/modules/nodes-cluster-enabling-features-manifest-inclusion.adoc @@ -51,39 +51,33 @@ The annotation name is singular: `feature-gate` (not `feature-gates`). Using the [id="nodes-cluster-enabling-features-manifest-inclusion-gated_{context}"] == Identifying gated manifests in a release -You can discover which manifests in a release payload are gated behind feature gates by extracting the payload and searching for the annotation. +You can discover which manifests in a release payload are gated behind feature gates by extracting the payload and searching for the annotation. You must have the {oc-first} installed and access to the release image for your cluster version. -.Prerequisites +To extract the release payload manifests to a local directory, run the following command: -* You have installed the {oc-first}. -* You have access to the release image for your cluster version. - -.Procedure - -. Extract the release payload manifests to a local directory: -+ [source,terminal] ---- $ oc adm release extract --to=/tmp/release-manifests \ $(oc get clusterversion version -o jsonpath='{.status.desired.image}') ---- -. Search for manifests with the `release.openshift.io/feature-gate` annotation: -+ +To search for manifests with the `release.openshift.io/feature-gate` annotation, run the following command: + [source,terminal] ---- $ grep -rl "release.openshift.io/feature-gate" /tmp/release-manifests/ ---- -. To see which feature gates each manifest requires, display the annotation values: -+ +To see which feature gates each manifest requires, display the annotation values: + [source,terminal] ---- $ grep -r "release.openshift.io/feature-gate" /tmp/release-manifests/ \ | sed 's|/tmp/release-manifests/||' ---- -+ -.Example output + +The output shows the manifest file name and the feature gate annotation value. For example: + [source,terminal] ---- 0000_30_cluster-api_11_deployment.yaml: release.openshift.io/feature-gate: ClusterAPIMachineManagement @@ -108,25 +102,25 @@ If you disable a feature gate after enabling it, you must manually remove any or CVO logs information about excluded and included gated manifests. You can view these messages to understand which manifests are affected by your feature gate configuration. -.Procedure +To view CVO logs for feature gate manifest exclusion messages, run the following command: -. View CVO logs for feature gate manifest exclusion messages: -+ [source,terminal] ---- $ oc logs -n openshift-cluster-version deployment/cluster-version-operator \ | grep "feature gate" ---- -+ -.Example output for excluded manifests + +When a manifest is excluded because its required gate is not enabled, CVO logs a message similar to the following: + [source,terminal] ---- I0601 04:02:58.501447 1 payload.go:217] excluding Filename: "0000_30_cluster-api_11_deployment.yaml" Group: "apps" Kind: "Deployment" Namespace: "openshift-cluster-api" Name: "capi-controllers": feature gate ClusterAPIMachineManagement is required but not enabled ---- -+ -.Example output when a gated manifest is included because the gate is enabled + +When a manifest is annotated with a negated gate (for example, `-ClusterAPIMachineManagement`) and that gate is enabled, CVO excludes the manifest and logs a message similar to the following: + [source,terminal] ---- I0601 04:54:27.510394 1 payload.go:217] excluding Filename: "0000_30_cluster-api_04_crd.core-cluster-api.yaml" @@ -134,5 +128,5 @@ I0601 04:54:27.510394 1 payload.go:217] excluding Filename: "0000_30_cluster-ap Name: "ipaddressclaims.ipam.cluster.x-k8s.io": feature gate ClusterAPIMachineManagement is enabled but manifest requires it to be disabled ---- -+ -The second message indicates a manifest that is annotated with `-ClusterAPIMachineManagement` (negated) and is being excluded because the gate is now enabled. This is expected behavior for legacy resources that are superseded by the feature. + +This is expected behavior for legacy resources that are superseded by the feature. diff --git a/nodes/clusters/nodes-cluster-enabling-features.adoc b/nodes/clusters/nodes-cluster-enabling-features.adoc index 68d318bdca99..6ba81b47aa9b 100644 --- a/nodes/clusters/nodes-cluster-enabling-features.adoc +++ b/nodes/clusters/nodes-cluster-enabling-features.adoc @@ -38,3 +38,5 @@ include::modules/nodes-cluster-enabling-features-manifest-inclusion.adoc[levelof * xref:../../storage/container_storage_interface/persistent-storage-csi-sc-manage.adoc#persistent-storage-csi-sc-manage[Managing the default storage class] * xref:../../authentication/understanding-and-managing-pod-security-admission.adoc#understanding-and-managing-pod-security-admission[Pod security admission enforcement] + +* xref:../../nodes/clusters/nodes-cluster-enabling-features.adoc#nodes-cluster-enabling-features-manifest-inclusion_nodes-cluster-enabling[Understanding feature-gate-based manifest inclusion]