OCPBUGS-112550, OCPBUGS-112549: Release 4.22 GCD and KMS support - #10783
OCPBUGS-112550, OCPBUGS-112549: Release 4.22 GCD and KMS support#10783barbacbd wants to merge 25 commits into
Conversation
Updates GCP client initialization to support alternate universe domains. option.WithCredentialsJSON is used to handle most auth cases because it utilizes a different authentication flow using a self-signed JWT which is sent directly to GCP/GCD, rather than oauth2. The oauth2 flow of option.WithCredentials is problematic for GCD because service account keys (and other JSON creds), for some reason, set their token uri to https://oauth2.apis-berlin-build0.goog/token, an endpoint that doesn't exist; it is clearly documented in GCD docs that oauth2 should still use googleapis. Nevertheless, the option.WithCredentials will always use the token uri in the JSON even when it is correct. To get around this issue we use option.WithCredentialsJSON, which does not need the oauth2 token endpoint at all. We fallback to option.WithCredentials for authenticating via a service account attached to a VM, in which case there is no JSON to parse. Note that option.WithCredentialsJSON is deprecated in the GCP package because if credentials are accepted from an external source, there are potential vulnerabilities. See: https://docs.cloud.google.com/docs/authentication/client-libraries#external-credentials The installer is built to work under any circumstances as it serves a diverse set of needs. Clients built on top of the installer may accept external credentials, but it is up to those clients to validate the inputs.
Until CAPG has built in universe domain support, we can explicitly set the universe domain as an environment variable.
GCP domain-scoped projects (e.g. sovereign clouds) use a different service account email domain than standard GCP. Standard GCP uses: name@project-id.iam.gserviceaccount.com Domain-scoped projects reverse the prefix into a dot-separated format: name@project-id.prefix.iam.gserviceaccount.com For example, a project ID of "eu0:my-project" produces service account emails with the domain "my-project.eu0.iam.gserviceaccount.com". Previously, GetDefaultServiceAccount hardcoded the standard format, producing incorrect service account emails for domain-scoped project installations. Update GetDefaultServiceAccount to detect the colon-separated format and reverse it into the dot-separated email domain. Any prefix following the "prefix:project" pattern is handled generically rather than whitelisting specific prefixes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Configure default instance and disk types for sovereign cloud regions. Sovereign clouds have limited machine type availability compared to public GCP. Use c3-standard-4 instances and hyperdisk-balanced disks as defaults for sovereign cloud regions (those with 'u-' prefix), as these types are available in sovereign clouds while the standard defaults (n2-standard-4 and pd-ssd) are not.
Sovereign cloud project IDs contain a colon delimiter that distinguishes them from standard GCP projects. PSC endpoint overrides generate URLs targeting the googleapis.com universe, which is incompatible with sovereign cloud environments.
Sovereign cloud environments (identified by a domain-scoped project ID containing ":") require an OS image to be specified. This adds install-config validation ensuring that each compute and control plane machine pool has an OS image configured, either directly or via the default machine platform.
By setting the token-url to nil, we cause cloud-provider-gcp to use the mounted credentials provisioned by the credentialsrequest.
Project IDs are confirmed to always contain colons. This is based on a conversation with Alfred Chung. The two examples we have right now are: eu0 - Germany s3ns - France
Disk type validation was previously entirely static, relying on hardcoded maps that encode which disk types are compatible with which instance families. This works for public GCP where all standard disk types are available in every region, but fails for sovereign clouds where regional availability differs. This change adds an API-backed availability check that queries each zone individually via DiskTypes.Get, following the existing pattern established by GetMachineTypeWithZones for machine type validation. The two validation layers are complementary: 1. Static check (validateInstanceAndDiskType): Is this disk type compatible with this instance family? Catches combinations that never work anywhere. 2. API check (validateDiskTypeAvailability): Is this disk type available in this specific region and zones? Catches disk types that are valid in general but unavailable in the target region. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change the deprecated WithCredentialsJSON func to WithAuthCredentialsJSON which takes a "type" argument. That parameter is used to indicate the intended credentials type to accept, which can be useful to limit choice if accepting credentials from 3rd party users. As the installer considers its users as first parties, we want to accept all credentials transparently for GCP. Any layered products utililzing the installer can limit provided credentials if needed.
Google's docs indicate that WIF credentials should be validated by the caller. See: https://docs.cloud.google.com/docs/authentication/client-libraries#external-credentials The validation requires that token_url equal https://sts.googleapis.com/v1/token In a non-default universe domain, the token_url would utilize that domain instead of googleapis.com. But universe domain is an arbitrary string that cannot be validated; therefore we cannot accept WIF credentials for the installer with custom universe domains.
Domain-scoped project IDs were unconditionally applying the domain prefix to service agent email addresses, but sovereign cloud service agents use a "-system" suffix (e.g. "eu0-system") rather than just the bare domain prefix. Restrict the domain-scoped logic to sovereign cloud environments using GetCloudEnvironment and append the correct suffix.
* Update the component CRD for CAPG to allow/validate the service account(s) and project ID for sovereign clouds. The Project ID can now include a colon and the service accounts can include a period in places other than the expected suffix.
The test for invalid format should have been failing and the valid was set to true. The new crypto package may have caused this to finally come up. setting the validity to false to make sure the test is correct.
Add GetDefaultEncryptionKey to pkg/types/gcp/platform.go, which returns the KMS key from defaultMachinePlatform.osDisk.encryptionKey.kmsKey if configured. This key is used for GCS bucket encryption (bootstrap ignition and image registry). Related: CORS-4406
Add validation for customer-managed KMS keys used to encrypt GCS buckets for Ignition bootstrap and internal image registry. Changes: - Extract common KMS key validation logic into validateKMSKeyReference helper function for reuse across disk and storage encryption validation - Add validateStorageEncryptionKeys to validate Ignition and Registry storage encryption keys - Handle permission failures gracefully with warnings instead of hard errors, allowing installations to proceed even when key ring verification fails due to insufficient permissions The validation reuses the existing GetKeyRing API call pattern from machine pool disk encryption validation, ensuring consistency across all KMS key validation in the installer. Format validation for key ring and key names is handled by the GCP API itself. Location validation is implicit through the key ring lookup.
GCS bucket encryption does not support global keys. Add a check in validateKMSKeyReference to reject KMS keys with a global location before attempting the key ring lookup. Add test cases covering global location rejection for control plane, compute, and defaultMachinePlatform KMS keys.
…CS storage - Bootstrap ignition bucket: set BucketEncryption.DefaultKMSKeyName when a KMS key is configured via defaultMachinePlatform - ImageRegistry CR: generate manifest populating spec.storage.gcs.keyID with the KMS key resource path Both use the shared GetDefaultEncryptionKey and FormatKMSKeyResourcePath helpers. Vendor the imageregistry/v1 API types for typed manifest generation.
Add tests for GetBootstrapStorageName and FormatKMSKeyResourcePath covering nil keys, default/custom project IDs, and regional configurations.
…dation Validate that Google-managed service agents (Compute Engine and Cloud Storage) have the CryptoKey Encrypter/Decrypter role on configured KMS keys before provisioning begins. Without this, installs fail late with opaque permission-denied errors from GCE or GCS. The Compute Engine agent is checked on every KMS key (controlPlane, compute, defaultMachinePlatform) since it encrypts disks. The Cloud Storage agent is checked only on the defaultMachinePlatform key, which is used for bootstrap ignition bucket encryption. When the IAM policy cannot be retrieved (e.g. the caller lacks cloudkms.cryptoKeys.getIamPolicy), the check degrades gracefully to a warning rather than blocking the install.
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
openshift-ci-robot
added
jira/valid-reference
|
@barbacbd: This pull request references Jira Issue OCPBUGS-112140, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. This pull request references Jira Issue OCPBUGS-112139, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this: Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: openshift/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
|
@barbacbd: This pull request references Jira Issue OCPBUGS-112140, which is invalid:
Comment This pull request references Jira Issue OCPBUGS-112139, which is invalid:
Comment DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
all of the GCD and KMS key 5.0 pull requests.
bd65708 to
f5dfff6
Compare
|
/jira refresh |
|
@barbacbd: This pull request references Jira Issue OCPBUGS-112140, which is invalid:
Comment This pull request references Jira Issue OCPBUGS-112139, which is invalid:
Comment DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/cc @rochacbruno @sadasu |
|
/retest-required |
barbacbd
changed the title
|
@barbacbd: This pull request references Jira Issue OCPBUGS-112550, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. This pull request references Jira Issue OCPBUGS-112549, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/jira refresh |
|
@barbacbd: This pull request references Jira Issue OCPBUGS-112550, which is invalid:
Comment This pull request references Jira Issue OCPBUGS-112549, which is invalid:
Comment DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/jira refresh |
|
@barbacbd: This pull request references Jira Issue OCPBUGS-112550, which is invalid:
Comment This pull request references Jira Issue OCPBUGS-112549, which is invalid:
Comment DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
/jira refresh |
openshift-ci-robot
added
jira/valid-bug
|
@barbacbd: This pull request references Jira Issue OCPBUGS-112550, which is valid. 7 validation(s) were run on this bug
This pull request references Jira Issue OCPBUGS-112549, which is valid. 7 validation(s) were run on this bug
DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@barbacbd: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: sadasu The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
5 participants
This set of commits are used to backport the GCP KMS Key support and GCD installer support to 4.22