diff --git a/modules/nodes-cluster-enabling-features-manifest-inclusion.adoc b/modules/nodes-cluster-enabling-features-manifest-inclusion.adoc new file mode 100644 index 000000000000..98c95d2fa6b4 --- /dev/null +++ b/modules/nodes-cluster-enabling-features-manifest-inclusion.adoc @@ -0,0 +1,132 @@ +// Module included in the following assemblies: +// +// * nodes/clusters/nodes-cluster-enabling-features.adoc + +:_mod-docs-content-type: CONCEPT +[id="nodes-cluster-enabling-features-manifest-inclusion_{context}"] += Understanding feature-gate-based manifest inclusion + +[role="_abstract"] +Starting in {product-title} 4.22, the Cluster Version Operator (CVO) uses the `release.openshift.io/feature-gate` annotation on release payload manifests to conditionally include or exclude cluster resources based on which feature gates are enabled. When you enable or disable a feature gate, CVO automatically re-evaluates all annotated manifests and applies or stops managing the affected resources. + +[id="nodes-cluster-enabling-features-manifest-inclusion-how_{context}"] +== How feature-gate-based manifest inclusion works + +Each release payload contains manifests for all cluster components, including components that are gated behind specific feature gates. CVO reads the `release.openshift.io/feature-gate` annotation on each manifest to determine whether to include it during reconciliation: + +* If the annotation specifies a feature gate that is **enabled** on the cluster, CVO includes the manifest and applies the resource. +* If the annotation specifies a feature gate that is **not enabled**, CVO excludes the manifest and does not apply the resource. +* If a manifest has no `release.openshift.io/feature-gate` annotation, CVO always includes it. + +When you toggle a feature gate at runtime using the `FeatureGate` custom resource (CR), CVO detects the change, re-reads the payload, and re-evaluates all gated manifests. Resources whose gate conditions are now met are applied. Resources whose gate conditions are no longer met are no longer managed by CVO. + +[id="nodes-cluster-enabling-features-manifest-inclusion-syntax_{context}"] +== Annotation syntax + +The `release.openshift.io/feature-gate` annotation supports the following formats: + +.Annotation syntax for feature-gate-based manifest inclusion +[cols="1a,2a,2a",options="header"] +|=== +|Format |Meaning |Example + +|`GateName` +|Include this manifest only when `GateName` is enabled. +|`release.openshift.io/feature-gate: ClusterAPIMachineManagement` + +|`-GateName` +|Include this manifest only when `GateName` is disabled. This is used for legacy resources that should be removed when a new feature replaces them. +|`release.openshift.io/feature-gate: -ClusterAPIMachineManagement` + +|`-Gate1,-Gate2` +|Include this manifest only when **all** listed gates are disabled (AND logic). The manifest is excluded if any of the listed gates is enabled. +|`release.openshift.io/feature-gate: -ClusterAPIMachineManagement,-MachineAPIMigration` +|=== + +[NOTE] +==== +The annotation name is singular: `feature-gate` (not `feature-gates`). Using the plural form will have no effect. +==== + +[id="nodes-cluster-enabling-features-manifest-inclusion-gated_{context}"] +== Identifying gated manifests in a release + +You can discover which manifests in a release payload are gated behind feature gates by extracting the payload and searching for the annotation. You must have the {oc-first} installed and access to the release image for your cluster version. + +To extract the release payload manifests to a local directory, run the following command: + +[source,terminal] +---- +$ oc adm release extract --to=/tmp/release-manifests \ + $(oc get clusterversion version -o jsonpath='{.status.desired.image}') +---- + +To search for manifests with the `release.openshift.io/feature-gate` annotation, run the following command: + +[source,terminal] +---- +$ grep -rl "release.openshift.io/feature-gate" /tmp/release-manifests/ +---- + +To see which feature gates each manifest requires, display the annotation values: + +[source,terminal] +---- +$ grep -r "release.openshift.io/feature-gate" /tmp/release-manifests/ \ + | sed 's|/tmp/release-manifests/||' +---- + +The output shows the manifest file name and the feature gate annotation value. For example: + +[source,terminal] +---- +0000_30_cluster-api_11_deployment.yaml: release.openshift.io/feature-gate: ClusterAPIMachineManagement +0000_30_cluster-api_04_crd.core-cluster-api.yaml: release.openshift.io/feature-gate: -ClusterAPIMachineManagement +0000_20_crd-compatibility-checker_11_deployment.yaml: release.openshift.io/feature-gate: CRDCompatibilityRequirementOperator +---- + +[id="nodes-cluster-enabling-features-manifest-inclusion-orphaned_{context}"] +== Orphaned resources when a gate is disabled + +When CVO stops managing a manifest because its feature gate condition is no longer met, the previously applied resources **remain on the cluster**. CVO does not delete them. + +For example, if you enable the `CRDCompatibilityRequirementOperator` feature gate, CVO creates the associated namespace, deployment, CRD, and RBAC resources. If you later remove `CRDCompatibilityRequirementOperator` from `customNoUpgrade.enabled`, CVO stops managing those resources but does not delete them. The namespace, deployment, and other resources continue to exist. + +[IMPORTANT] +==== +If you disable a feature gate after enabling it, you must manually remove any orphaned resources that are no longer needed. Use `oc delete` to remove the namespace and other resources that were created by the gated manifests. +==== + +[id="nodes-cluster-enabling-features-manifest-inclusion-logs_{context}"] +== Viewing CVO manifest inclusion logs + +CVO logs information about excluded and included gated manifests. You can view these messages to understand which manifests are affected by your feature gate configuration. + +To view CVO logs for feature gate manifest exclusion messages, run the following command: + +[source,terminal] +---- +$ oc logs -n openshift-cluster-version deployment/cluster-version-operator \ + | grep "feature gate" +---- + +When a manifest is excluded because its required gate is not enabled, CVO logs a message similar to the following: + +[source,terminal] +---- +I0601 04:02:58.501447 1 payload.go:217] excluding Filename: "0000_30_cluster-api_11_deployment.yaml" + Group: "apps" Kind: "Deployment" Namespace: "openshift-cluster-api" + Name: "capi-controllers": feature gate ClusterAPIMachineManagement is required but not enabled +---- + +When a manifest is annotated with a negated gate (for example, `-ClusterAPIMachineManagement`) and that gate is enabled, CVO excludes the manifest and logs a message similar to the following: + +[source,terminal] +---- +I0601 04:54:27.510394 1 payload.go:217] excluding Filename: "0000_30_cluster-api_04_crd.core-cluster-api.yaml" + Group: "apiextensions.k8s.io" Kind: "CustomResourceDefinition" + Name: "ipaddressclaims.ipam.cluster.x-k8s.io": + feature gate ClusterAPIMachineManagement is enabled but manifest requires it to be disabled +---- + +This is expected behavior for legacy resources that are superseded by the feature. diff --git a/nodes/clusters/nodes-cluster-enabling-features.adoc b/nodes/clusters/nodes-cluster-enabling-features.adoc index 1c0d2e71a8d0..6ba81b47aa9b 100644 --- a/nodes/clusters/nodes-cluster-enabling-features.adoc +++ b/nodes/clusters/nodes-cluster-enabling-features.adoc @@ -17,6 +17,8 @@ include::modules/nodes-cluster-enabling-features-console.adoc[leveloffset=+1] include::modules/nodes-cluster-enabling-features-cli.adoc[leveloffset=+1] +include::modules/nodes-cluster-enabling-features-manifest-inclusion.adoc[leveloffset=+1] + [role="_additional-resources"] [id="additional-resources_nodes-cluster-enabling"] == Additional resources @@ -36,3 +38,5 @@ include::modules/nodes-cluster-enabling-features-cli.adoc[leveloffset=+1] * xref:../../storage/container_storage_interface/persistent-storage-csi-sc-manage.adoc#persistent-storage-csi-sc-manage[Managing the default storage class] * xref:../../authentication/understanding-and-managing-pod-security-admission.adoc#understanding-and-managing-pod-security-admission[Pod security admission enforcement] + +* xref:../../nodes/clusters/nodes-cluster-enabling-features.adoc#nodes-cluster-enabling-features-manifest-inclusion_nodes-cluster-enabling[Understanding feature-gate-based manifest inclusion]