Instruction
{
"KubernetesVersion": "1.33.5",
"RemovedFeatureGates": ["AdmissionWebhookMatchConditions", "AggregatedDiscoveryEndpoint", "APIListChunking", "AppArmor", "AppArmorFields", "CPUManager", "DisableCloudProviders", "DisableKubeletCloudCredentialProviders", "EfficientWatchResumption", "JobPodFailurePolicy", "KubeProxyDrainingTerminatingNodes", "PDBUnhealthyPodEvictionPolicy", "PersistentVolumeLastPhaseTransitionTime", "RemainingItemCount", "VolumeCapacityPriority", "WatchBookmark"],
"K8sCSIAncillaryImages":{
"csi-provisioner": "v5.3.0",
"csi-attacher": "v4.9.0",
"csi-livenessprobe": "v2.16.0",
"csi-node-driver-registrar": "v2.14.0",
"csi-snapshotter": "v8.3.0",
"csi-snapshot-controller": "v8.3.0",
"csi-resizer": "v1.14.0",
"azuredisk-csi": "v1.33.4",
"addon-resizer": "1.8.23",
"metrics-server": "v0.8.0",
"kube-addon-manager": "v9.1.8",
"cluster-autoscaler": "v1.33.1"
},
"AzureCloudManagerImages":{
"azure-cloud-controller-manager":"v1.33.1",
"azure-cloud-node-manager":"v1.33.0"
}
}
Input Validation
- Retrieve the Kubernetes version from the "KubernetesVersion" field in the JSON and assign it to the
$KubernetesVersion variable.
- Retrieve the removed feature gates from the "RemovedFeatureGates" array in the JSON and assign it to the
$RemovedFeatureGates variable.
- Retrieve the CSI and ancillary component images from the "K8sCSIAncillaryImages" field in the JSON and assign it to the
$K8sCSIAncillaryImages variable.
- Retrieve the CSI and ancillary component images from the "AzureCloudManagerImages" field in the JSON and assign it to the
$AzureCloudManagerImages variable.
Processing Rule
- Process exactly one level-2 section (## ...) at a time.
- Within that section, handle impacted files strictly one file at a time, in the order listed.
- For each impacted file:
- Use only the variables ($KubernetesVersion, $RemovedFeatureGates) from the JSON block.
- Open/read the current file only; do not open/read any other impacted files yet.
- Apply the current section's "Implementation Guidance" to this file and save.
- Close the file and move to the next impacted file.
- Remove loaded file from current context
- After processing each impacted file, clear all loaded context/state for that file
- Do not pre-scan, open, or read all impacted files up front; operate strictly file-by-file.
- Finish all edits for the current section before starting the next.
- After completing a section:
- Keep only the variables; clear the context and discard everything else.
- Proceed to the next level-2 section in order.
Update supported and default Version
Description
Supported Kubernetes versions are declared in versions.go as version-to-bool maps, where true means allowed for new cluster creation.
Impacted Files
- pkg/api/common/versions.go
- pkg/api/common/const.go
Implementation Guidance
$KubernetesVersion must follow the format [MAJOR].[MINOR].[REVISION] (for example, 1.32.10).
- In
pkg/api/common/versions.go, update the following maps:
AllKubernetesSupportedVersions
AllKubernetesSupportedVersionsAzureStack
AllKubernetesWindowsSupportedVersionsAzureStack
- In
pkg/api/common/const.go, update the following constants:
KubernetesDefaultRelease
KubernetesDefaultReleaseWindows
KubernetesDefaultReleaseAzureStack
KubernetesDefaultReleaseWindowsAzureStack
- Logic: 0. Pre-check: If the exact
$KubernetesVersion key ([MAJOR].[MINOR].[REVISION]) already exists in the target map(s), skip all updates (no changes to versions.go or const.go).
- For each map, check whether a placeholder key for the same minor exists:
[MAJOR].[MINOR].X.
- If it exists (same minor):
- In
versions.go:
- Set the existing
[MAJOR].[MINOR].X entry to false.
- Insert the exact
$KubernetesVersion entry directly below it and set to true.
- Keep entries grouped by minor and sorted in ascending version order;
- In
const.go:
- No changes (defaults do not change when there is no new minor).
- If it does not exist (new minor):
- In
versions.go:
- Add the exact
$KubernetesVersion entry (in source order) and set to true.
- Set the oldest currently creatable minor
[MAJOR].[MINOR-2].X to false so only the latest two minors remain creatable.
- In
const.go:
- Set the default release constants to
[MAJOR].[MINOR-1] (major.minor only, no patch).
Remove invalid feature gates
Keep only the variables; clear the context and discard everything else.
Description
Feature gates may be removed or graduate to GA in new Kubernetes minor versions. For the target version ($KubernetesVersion), remove any gates listed in $RemovedFeatureGates that are no longer valid from all component defaults to prevent invalid-flag errors and warnings.
Impacted Files
- `pkg/api/defaults-kubelet.go`
- `pkg/api/defaults-controller-manager.go`
- `pkg/api/defaults-cloud-controller-manager.go`
- `pkg/api/defaults-apiserver.go`
- `pkg/api/defaults-scheduler.go`
Implementation Guidance
-
$KubernetesVersion must follow the format [MAJOR].[MINOR].[REVISION].
-
Iterate through the feature gates listed in $RemovedFeatureGates, and check whether a guard like common.IsKubernetesVersionGe(o.OrchestratorVersion, "[MAJOR].[MINOR].0") already exists in the source code.
- If such a guard exists, do not change the source code.
- If it does not exist, add the guard for
"[MAJOR].[MINOR].0" and remove the feature gate inside that guard.
-
Example:
(Note: The revision is always 0 in the comparison)
if common.IsKubernetesVersionGe(o.OrchestratorVersion, "[MAJOR].[MINOR].0") {
// Remove the invalid feature gate here
}
-
When you change the source code:
Add or update k8s CSI and ancillary images
Keep only the variables; clear the context and discard everything else.
Description
Container Storage Interface (CSI) and ancillary component images need to be updated for new Kubernetes versions. These images include CSI drivers (provisioner, attacher, liveness probe, node driver registrar, snapshotter, snapshot controller, resizer), Azure-specific CSI drivers (Azure Disk, Azure File), and other supporting components that are version-specific.
Impacted Files
- `pkg/api/k8s_versions.go`
Implementation Guidance
$KubernetesVersion must follow the format [MAJOR].[MINOR].[REVISION] (for example, 1.32.10).
- In
pkg/api/k8s_versions.go, update the kubernetesImageBaseVersionedImages map for both common.KubernetesImageBaseTypeGCR and common.KubernetesImageBaseTypeMCR sections.
- Logic:
- Extract the major.minor version from
$KubernetesVersion (e.g., 1.32 from 1.32.10).
- Check if an entry for
"[MAJOR].[MINOR]" already exists in the kubernetesImageBaseVersionedImages map.
- If it exists:
- Review and update CSI component versions if newer versions are available.
- Update Azure-specific components (Azure Disk CSI) to compatible versions.
- If it does not exist (new minor version):
- Copy the most recent minor version's entries as a template.
- Add a new entry for
"[MAJOR].[MINOR]" in both GCR and MCR sections.
- Update all CSI component image versions based on
$K8sCSIAncillaryImages for the new Kubernetes version:
common.CSIProvisionerContainerName: use "csi-provisioner" value from $K8sCSIAncillaryImages
common.CSIAttacherContainerName: use "csi-attacher" value from $K8sCSIAncillaryImages
common.CSILivenessProbeContainerName: use "csi-livenessprobe" value from $K8sCSIAncillaryImages
common.CSILivenessProbeWindowsContainerName: use "csi-livenessprobe" value from $K8sCSIAncillaryImages
common.CSINodeDriverRegistrarContainerName: use "csi-node-driver-registrar" value from $K8sCSIAncillaryImages
common.CSINodeDriverRegistrarWindowsContainerName: use "csi-node-driver-registrar" value from $K8sCSIAncillaryImages
common.CSISnapshotterContainerName: use "csi-snapshotter" value from $K8sCSIAncillaryImages
common.CSISnapshotControllerContainerName: use "csi-snapshot-controller" value from $K8sCSIAncillaryImages
common.CSIResizerContainerName: use "csi-resizer" value from $K8sCSIAncillaryImages
common.CSIAzureDiskContainerName: use "azuredisk-csi" value from $K8sCSIAncillaryImages
- Update other ancillary components using corresponding values from
$K8sCSIAncillaryImages:
common.AddonResizerComponentName: use "addon-resizer" value
common.MetricsServerAddonName: use "metrics-server" value
common.AddonManagerComponentName: use "kube-addon-manager" value
common.ClusterAutoscalerAddonName: use "cluster-autoscaler" value
- For GCR images, prefix with appropriate registry path (e.g., "oss/kubernetes-csi/", "addon-resizer:", etc.)
- For MCR images, prefix with "oss/kubernetes/" or "oss/kubernetes-csi/" as appropriate
- Add a new case for
"[MAJOR].[MINOR]" in the getK8sVersionComponents function:
- Copy the most recent minor version's case as a template
- Update the following Azure Cloud Manager images using values from
$AzureCloudManagerImages:
common.CloudControllerManagerComponentName: use "azure-cloud-controller-manager" value
common.CloudNodeManagerAddonName: use "azure-cloud-node-manager" value (if present)
- Ensure all CSI component references use
k8sComponent[common.CSIComponentName] pattern to leverage the versioned images from the map
Update VHD install dependencies for k8s CSI and ancillary images
Keep only the variables; clear the context and discard everything else.
Description
The VHD build script install-dependencies.sh pre-pulls container images and installs binaries for Kubernetes versions and their associated CSI/ancillary components. When adding support for a new Kubernetes version, the script needs to be updated to include the appropriate component versions and container images that will be available in the VHD.
Impacted Files
- `vhd/packer/install-dependencies.sh`
Implementation Guidance
$KubernetesVersion must follow the format [MAJOR].[MINOR].[REVISION] (for example, 1.32.10).
- In
vhd/packer/install-dependencies.sh, update the version arrays and container image pull sections to include components for the new Kubernetes version.
- Logic:
-
Get the supported Kubernetes versions from pkg/api/common/versions.go:
- Read the
AllKubernetesSupportedVersionsAzureStack map
- Extract all keys where the value is
true into an array and assign to variable $KubernetesSupportedVersionsAzureStack
- These represent the currently supported Kubernetes versions for Azure Stack
-
Update the K8S_VERSIONS array:
- Replace the existing array content with values from
$KubernetesSupportedVersionsAzureStack
- Sort versions in descending order (newest first) by semantic version
-
Update CSI component version arrays using versions from the kubernetesImageBaseVersionedImages map:
- For each Kubernetes version in
$KubernetesSupportedVersionsAzureStack, extract the major.minor (e.g., "1.32" from "1.32.10")
- From
pkg/api/k8s_versions.go, read the kubernetesImageBaseVersionedImages map for the corresponding major.minor version
- Extract the image versions (removing registry paths and prefixes and "v" prefix) and collect them for the respective arrays:
AZUREDISK_CSI_VERSIONS: extract version from common.CSIAzureDiskContainerName image
CSI_ATTACHER_VERSIONS: extract version from common.CSIAttacherContainerName image
CSI_NODE_DRIVER_REGISTRAR_VERSIONS: extract version from common.CSINodeDriverRegistrarContainerName image
CSI_PROVISIONER_VERSIONS: extract version from common.CSIProvisionerContainerName image
LIVENESSPROBE_VERSIONS: extract version from common.CSILivenessProbeContainerName image
CSI_RESIZER_VERSIONS: extract version from common.CSIResizerContainerName image
CSI_SNAPSHOTTER_VERSIONS: extract version from common.CSISnapshotterContainerName image
SNAPSHOT_CONTROLLER_VERSIONS: extract version from common.CSISnapshotControllerContainerName image
METRICS_SERVER_VERSIONS: extract version from common.MetricsServerAddonName image
KUBE_ADDON_MANAGER_VERSIONS: extract version from common.AddonManagerComponentName image
CLUSTER_AUTOSCALER_VERSIONS: extract version from common.ClusterAutoscalerAddonName image
CLUSTER_PROPORTIONAL_AUTOSCALER_VERSIONS: extract version from common.AddonResizerComponentName image
- Important: Replace the entire array content with only the extracted versions (do not append to existing versions)
- Before updating the file, remove duplications in each array to ensure unique versions only
- Sort versions in descending order (newest first) by semantic version
-
Update Azure Cloud Manager version arrays using versions from the getK8sVersionComponents function:
- For each Kubernetes version in
$KubernetesSupportedVersionsAzureStack, extract the major.minor (e.g., "1.32" from "1.32.10")
- From
pkg/api/k8s_versions.go, read the getK8sVersionComponents function for the corresponding major.minor version case
- Extract the image versions (removing registry paths and prefixes and "v" prefix) and collect them for the respective arrays:
CLOUD_CONTROLLER_MANAGER_VERSIONS: extract version from common.CloudControllerManagerComponentName image
CLOUD_NODE_MANAGER_VERSIONS: extract version from common.CloudNodeManagerAddonName image (if present)
- Important: Replace the entire array content with only the extracted versions (do not append to existing versions)
- Before updating the file, remove duplications in each array to ensure unique versions only
- Sort versions in descending order (newest first) by semantic version
Update Windows VHD install dependencies for k8s CSI and ancillary images
Keep only the variables; clear the context and discard everything else.
Description
The Windows VHD build script configure-windows-vhd-phase2.ps1 pre-pulls container images for Kubernetes versions and their associated CSI/ancillary components. When adding support for a new Kubernetes version, the script needs to be updated to include the appropriate component versions and container images that will be available in the Windows VHD.
Impacted Files
- `vhd/packer/configure-windows-vhd-phase2.ps1`
Implementation Guidance
$KubernetesVersion must follow the format [MAJOR].[MINOR].[REVISION] (for example, 1.32.10).
- In
vhd/packer/configure-windows-vhd-phase2.ps1, update the $imagesToPull array in the Get-ContainerImages function to include container images for the new Kubernetes version.
- Logic:
-
Get the supported Kubernetes versions from pkg/api/common/versions.go:
- Read the
AllKubernetesSupportedVersionsAzureStack map
- Extract all keys where the value is
true into an array and assign to variable $KubernetesSupportedVersionsAzureStack
- These represent the currently supported Kubernetes versions for Azure Stack
-
Extract CSI and ancillary component images from pkg/api/k8s_versions.go:
- For each Kubernetes version in
$KubernetesSupportedVersionsAzureStack, extract the major.minor (e.g., "1.32" from "1.32.10")
- From the
kubernetesImageBaseVersionedImages map for common.KubernetesImageBaseTypeMCR, collect the following Windows-compatible images:
- Azure Disk CSI images:
common.CSIAzureDiskContainerName (both regular and -windows-hp variants)
- CSI Node Driver Registrar images:
common.CSINodeDriverRegistrarWindowsContainerName
- CSI Liveness Probe images:
common.CSILivenessProbeWindowsContainerName
- From the
getK8sVersionComponents function for the corresponding major.minor version case:
- Azure Cloud Node Manager images:
common.CloudNodeManagerAddonName
-
Update the $imagesToPull array in the Get-ContainerImages function:
- Important: Replace the entire array content with a complete set of images (do not append to existing images)
- Include Windows base images (these should remain constant):
"mcr.microsoft.com/windows/servercore:ltsc2019"
"mcr.microsoft.com/windows/nanoserver:1809"
"mcr.microsoft.com/oss/kubernetes/pause:3.8"
"mcr.microsoft.com/oss/kubernetes/windows-host-process-containers-base-image:v1.0.0"
- Include all extracted CSI and ancillary component images from step 2
- For Azure Disk CSI images, include both regular and Windows Host Process (
-windows-hp) variants when available
- Before updating the file, remove duplications in the array to ensure unique images only
- Sort images alphabetically for consistency
-
Update the Kubernetes binaries download URLs in the Get-FilesToCacheOnVHD function:
- In the
"c:\akse-cache\win-k8s\" section of the $map variable
- Important: Replace the entire URL list with URLs for supported Kubernetes versions
- For each version in
$KubernetesSupportedVersionsAzureStack, add the corresponding download URL:
- Format:
"https://packages.aks.azure.com/kubernetes/v{VERSION}/windowszip/v{VERSION}-1int.zip"
- Where
{VERSION} is the full Kubernetes version (e.g., "1.32.10")
- Sort URLs in descending order by version (newest first)
-
Update the ACR Credential Provider URLs in the $acrCredentialProviderUrls array:
- For each Kubernetes version in
$KubernetesSupportedVersionsAzureStack, extract the major.minor (e.g., "1.32" from "1.32.10")
- From the
getK8sVersionComponents function, find the Azure Cloud Controller Manager version for the corresponding major.minor
- Important: Replace the entire array content with entries for supported Kubernetes versions
- Format each entry as:
@{ Url = "https://github.com/kubernetes-sigs/cloud-provider-azure/releases/download/v{ACR_VERSION}/azure-acr-credential-provider-windows-amd64.exe"; K8sVersion = "v{MAJOR}.{MINOR}" }
- Where
{ACR_VERSION} is the Azure Cloud Controller Manager version and {MAJOR}.{MINOR} is the Kubernetes major.minor version
- Sort entries in descending order by Kubernetes version (newest first)
Update Windows VHD main script for k8s CSI and ancillary images
Keep only the variables; clear the context and discard everything else.
Description
The main Windows VHD build script configure-windows-vhd.ps1 also contains container image configuration and Kubernetes binary caching logic that needs to be updated when adding support for new Kubernetes versions. This script differs from the phase2 script as it has conditional logic based on Windows Server version (2019, 2004) and supports both containerd and docker container runtimes.
Impacted Files
- `vhd/packer/configure-windows-vhd.ps1`
Implementation Guidance
$KubernetesVersion must follow the format [MAJOR].[MINOR].[REVISION] (for example, 1.32.10).
- In
vhd/packer/configure-windows-vhd.ps1, update the Get-ContainerImages function and Get-FilesToCacheOnVHD function to include components for the new Kubernetes version.
- Logic:
-
Get the supported Kubernetes versions from pkg/api/common/versions.go:
- Read the
AllKubernetesSupportedVersionsAzureStack map
- Extract all keys where the value is
true into an array and assign to variable $KubernetesSupportedVersionsAzureStack
- These represent the currently supported Kubernetes versions for Azure Stack
-
Extract CSI and ancillary component images from pkg/api/k8s_versions.go:
- For each Kubernetes version in
$KubernetesSupportedVersionsAzureStack, extract the major.minor (e.g., "1.32" from "1.32.10")
- From the
kubernetesImageBaseVersionedImages map for common.KubernetesImageBaseTypeMCR, collect the following Windows-compatible images:
- Azure Disk CSI images:
common.CSIAzureDiskContainerName (both regular and -windows-hp variants)
- CSI Node Driver Registrar images:
common.CSINodeDriverRegistrarWindowsContainerName
- CSI Liveness Probe images:
common.CSILivenessProbeWindowsContainerName
- From the
getK8sVersionComponents function for the corresponding major.minor version case:
- Azure Cloud Node Manager images:
common.CloudNodeManagerAddonName
-
Update the $imagesToPull arrays in the Get-ContainerImages function:
- Important: Update images for Windows Server 2019 case only (do not modify 2004 case unless specifically needed)
- In the
'2019' switch case, replace the entire $imagesToPull array content with:
- Windows base images (these should remain constant):
"mcr.microsoft.com/windows/servercore:ltsc2019"
"mcr.microsoft.com/windows/nanoserver:1809"
"mcr.microsoft.com/oss/kubernetes/pause:3.8"
"mcr.microsoft.com/oss/kubernetes/windows-host-process-containers-base-image:v1.0.0"
- All extracted CSI and ancillary component images from step 2
- For Azure Disk CSI images, include both regular and Windows Host Process (
-windows-hp) variants when available
- Before updating the file, remove duplications in the array to ensure unique images only
- Sort images alphabetically for consistency
-
Update the Kubernetes binaries download URLs in the Get-FilesToCacheOnVHD function:
- In the
"c:\akse-cache\win-k8s\" section of the $map variable
- Important: Replace the entire URL list with URLs for supported Kubernetes versions
- For each version in
$KubernetesSupportedVersionsAzureStack, add the corresponding download URL:
- Format:
"https://packages.aks.azure.com/kubernetes/v{VERSION}/windowszip/v{VERSION}-1int.zip"
- Where
{VERSION} is the full Kubernetes version (e.g., "1.32.10")
- Sort URLs in descending order by version (newest first)
-
Note: Unlike the phase2 script, the main script does not have ACR Credential Provider URLs, so no updates are needed for that section.
Miscellaneous Changes
Keep only the variables; clear the context and discard everything else.
Description
Impacted Files
Implementation Guidance
$KubernetesVersion must follow the format [MAJOR].[MINOR].[REVISION] (for example, 1.32.10).
- Logic:
- Get the supported Kubernetes versions from
pkg/api/common/versions.go:
- Read the AllKubernetesSupportedVersionsAzureStack map
- Extract all keys where the value is true into an array and assign to variable $KubernetesSupportedVersionsAzureStack
- Sort the versions in descending order (newest first)
- Read
examples/azure-stack/kubernetes-azurestack.json and examples/azure-stack/kubernetes-windows.json, set the orchestratorRelease and orchestratorVersion values to the second last (penultimate) supported Kubernetes version in $KubernetesSupportedVersionsAzureStack (e.g., if supported versions are 1.32.10 and 1.31.11, use 1.31.11):
"orchestratorRelease": "1.31",
"orchestratorVersion": "1.31.11",
- Read
hack/tools/Makefile and set KUBECTL_VERSION to the second last supported Kubernetes version in $KubernetesSupportedVersionsAzureStack with "v" prefix (e.g., "v1.31.11")
- Create release example API model if not exist:
- Extract the major.minor version from
$KubernetesVersion (e.g., 1.32 from 1.32.10)
- Check if the file
examples/kubernetes-releases/kubernetes[MAJOR].[MINOR].json already exists
- If the file does not exist, create it with the following JSON structure (replace
[MAJOR].[MINOR] with actual version like 1.32):
{
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"orchestratorRelease": "[MAJOR].[MINOR]"
},
"masterProfile": {
"count": 1,
"dnsPrefix": "",
"vmSize": "Standard_D2_v3"
},
"agentPoolProfiles": [
{
"name": "agentpool1",
"count": 3,
"vmSize": "Standard_D2_v3"
}
],
"linuxProfile": {
"adminUsername": "azureuser",
"ssh": {
"publicKeys": [
{
"keyData": ""
}
]
}
}
}
}
- If the file already exists, skip creation (no changes needed)
- Update generate test examples for supported versions:
- Read
cmd/generate_test.go and locate the TestExampleAPIModels function
- In the
tests slice, find the entries that reference ../examples/kubernetes-releases/kubernetes*.json files
- Important: Replace the entire set of kubernetes-releases test entries with entries for all supported Kubernetes versions
- For each major.minor version extracted from
$KubernetesSupportedVersionsAzureStack, add a test entry in the format:
{
name: "[MAJOR].[MINOR] example",
apiModelPath: "../examples/kubernetes-releases/kubernetes[MAJOR].[MINOR].json",
setArgs: defaultSet,
},
- Example: if supported versions are 1.32.10 and 1.31.11, add entries for "1.32 example" and "1.31 example"
- Sort entries in descending order by version (newest first)
- Ensure all other test entries in the slice remain unchanged
Instruction
{ "KubernetesVersion": "1.33.5", "RemovedFeatureGates": ["AdmissionWebhookMatchConditions", "AggregatedDiscoveryEndpoint", "APIListChunking", "AppArmor", "AppArmorFields", "CPUManager", "DisableCloudProviders", "DisableKubeletCloudCredentialProviders", "EfficientWatchResumption", "JobPodFailurePolicy", "KubeProxyDrainingTerminatingNodes", "PDBUnhealthyPodEvictionPolicy", "PersistentVolumeLastPhaseTransitionTime", "RemainingItemCount", "VolumeCapacityPriority", "WatchBookmark"], "K8sCSIAncillaryImages":{ "csi-provisioner": "v5.3.0", "csi-attacher": "v4.9.0", "csi-livenessprobe": "v2.16.0", "csi-node-driver-registrar": "v2.14.0", "csi-snapshotter": "v8.3.0", "csi-snapshot-controller": "v8.3.0", "csi-resizer": "v1.14.0", "azuredisk-csi": "v1.33.4", "addon-resizer": "1.8.23", "metrics-server": "v0.8.0", "kube-addon-manager": "v9.1.8", "cluster-autoscaler": "v1.33.1" }, "AzureCloudManagerImages":{ "azure-cloud-controller-manager":"v1.33.1", "azure-cloud-node-manager":"v1.33.0" } }Input Validation
$KubernetesVersionvariable.$RemovedFeatureGatesvariable.$K8sCSIAncillaryImagesvariable.$AzureCloudManagerImagesvariable.Processing Rule
Update supported and default Version
Description
Supported Kubernetes versions are declared in versions.go as version-to-bool maps, where true means allowed for new cluster creation.
Impacted Files
Implementation Guidance
$KubernetesVersionmust follow the format[MAJOR].[MINOR].[REVISION](for example,1.32.10).pkg/api/common/versions.go, update the following maps:AllKubernetesSupportedVersionsAllKubernetesSupportedVersionsAzureStackAllKubernetesWindowsSupportedVersionsAzureStackpkg/api/common/const.go, update the following constants:KubernetesDefaultReleaseKubernetesDefaultReleaseWindowsKubernetesDefaultReleaseAzureStackKubernetesDefaultReleaseWindowsAzureStack$KubernetesVersionkey ([MAJOR].[MINOR].[REVISION]) already exists in the target map(s), skip all updates (no changes toversions.goorconst.go).[MAJOR].[MINOR].X.versions.go:[MAJOR].[MINOR].Xentry tofalse.$KubernetesVersionentry directly below it and set totrue.const.go:versions.go:$KubernetesVersionentry (in source order) and set totrue.[MAJOR].[MINOR-2].Xtofalseso only the latest two minors remain creatable.const.go:[MAJOR].[MINOR-1](major.minor only, no patch).Remove invalid feature gates
Keep only the variables; clear the context and discard everything else.
Description
Feature gates may be removed or graduate to GA in new Kubernetes minor versions. For the target version ($KubernetesVersion), remove any gates listed in $RemovedFeatureGates that are no longer valid from all component defaults to prevent invalid-flag errors and warnings.
Impacted Files
Implementation Guidance
$KubernetesVersionmust follow the format[MAJOR].[MINOR].[REVISION].Iterate through the feature gates listed in
$RemovedFeatureGates, and check whether a guard likecommon.IsKubernetesVersionGe(o.OrchestratorVersion, "[MAJOR].[MINOR].0")already exists in the source code."[MAJOR].[MINOR].0"and remove the feature gate inside that guard.Example:
(Note: The revision is always
0in the comparison)When you change the source code:
Add a new unit test at the bottom of the corresponding test file to verify that the removed feature gates are sanitized correctly.
Test[Component]FeatureGates[Major][Minor]. For example: forpkg/api/defaults-apiserver.go, the test file ispkg/api/defaults-apiserver_test.go, and[Component]isAPIServer.Add or update k8s CSI and ancillary images
Keep only the variables; clear the context and discard everything else.
Description
Container Storage Interface (CSI) and ancillary component images need to be updated for new Kubernetes versions. These images include CSI drivers (provisioner, attacher, liveness probe, node driver registrar, snapshotter, snapshot controller, resizer), Azure-specific CSI drivers (Azure Disk, Azure File), and other supporting components that are version-specific.
Impacted Files
Implementation Guidance
$KubernetesVersionmust follow the format[MAJOR].[MINOR].[REVISION](for example,1.32.10).pkg/api/k8s_versions.go, update thekubernetesImageBaseVersionedImagesmap for bothcommon.KubernetesImageBaseTypeGCRandcommon.KubernetesImageBaseTypeMCRsections.$KubernetesVersion(e.g.,1.32from1.32.10)."[MAJOR].[MINOR]"already exists in thekubernetesImageBaseVersionedImagesmap."[MAJOR].[MINOR]"in both GCR and MCR sections.$K8sCSIAncillaryImagesfor the new Kubernetes version:common.CSIProvisionerContainerName: use "csi-provisioner" value from$K8sCSIAncillaryImagescommon.CSIAttacherContainerName: use "csi-attacher" value from$K8sCSIAncillaryImagescommon.CSILivenessProbeContainerName: use "csi-livenessprobe" value from$K8sCSIAncillaryImagescommon.CSILivenessProbeWindowsContainerName: use "csi-livenessprobe" value from$K8sCSIAncillaryImagescommon.CSINodeDriverRegistrarContainerName: use "csi-node-driver-registrar" value from$K8sCSIAncillaryImagescommon.CSINodeDriverRegistrarWindowsContainerName: use "csi-node-driver-registrar" value from$K8sCSIAncillaryImagescommon.CSISnapshotterContainerName: use "csi-snapshotter" value from$K8sCSIAncillaryImagescommon.CSISnapshotControllerContainerName: use "csi-snapshot-controller" value from$K8sCSIAncillaryImagescommon.CSIResizerContainerName: use "csi-resizer" value from$K8sCSIAncillaryImagescommon.CSIAzureDiskContainerName: use "azuredisk-csi" value from$K8sCSIAncillaryImages$K8sCSIAncillaryImages:common.AddonResizerComponentName: use "addon-resizer" valuecommon.MetricsServerAddonName: use "metrics-server" valuecommon.AddonManagerComponentName: use "kube-addon-manager" valuecommon.ClusterAutoscalerAddonName: use "cluster-autoscaler" value"[MAJOR].[MINOR]"in thegetK8sVersionComponentsfunction:$AzureCloudManagerImages:common.CloudControllerManagerComponentName: use "azure-cloud-controller-manager" valuecommon.CloudNodeManagerAddonName: use "azure-cloud-node-manager" value (if present)k8sComponent[common.CSIComponentName]pattern to leverage the versioned images from the mapUpdate VHD install dependencies for k8s CSI and ancillary images
Keep only the variables; clear the context and discard everything else.
Description
The VHD build script
install-dependencies.shpre-pulls container images and installs binaries for Kubernetes versions and their associated CSI/ancillary components. When adding support for a new Kubernetes version, the script needs to be updated to include the appropriate component versions and container images that will be available in the VHD.Impacted Files
Implementation Guidance
$KubernetesVersionmust follow the format[MAJOR].[MINOR].[REVISION](for example,1.32.10).vhd/packer/install-dependencies.sh, update the version arrays and container image pull sections to include components for the new Kubernetes version.Get the supported Kubernetes versions from
pkg/api/common/versions.go:AllKubernetesSupportedVersionsAzureStackmaptrueinto an array and assign to variable$KubernetesSupportedVersionsAzureStackUpdate the
K8S_VERSIONSarray:$KubernetesSupportedVersionsAzureStackUpdate CSI component version arrays using versions from the
kubernetesImageBaseVersionedImagesmap:$KubernetesSupportedVersionsAzureStack, extract the major.minor (e.g., "1.32" from "1.32.10")pkg/api/k8s_versions.go, read thekubernetesImageBaseVersionedImagesmap for the corresponding major.minor versionAZUREDISK_CSI_VERSIONS: extract version fromcommon.CSIAzureDiskContainerNameimageCSI_ATTACHER_VERSIONS: extract version fromcommon.CSIAttacherContainerNameimageCSI_NODE_DRIVER_REGISTRAR_VERSIONS: extract version fromcommon.CSINodeDriverRegistrarContainerNameimageCSI_PROVISIONER_VERSIONS: extract version fromcommon.CSIProvisionerContainerNameimageLIVENESSPROBE_VERSIONS: extract version fromcommon.CSILivenessProbeContainerNameimageCSI_RESIZER_VERSIONS: extract version fromcommon.CSIResizerContainerNameimageCSI_SNAPSHOTTER_VERSIONS: extract version fromcommon.CSISnapshotterContainerNameimageSNAPSHOT_CONTROLLER_VERSIONS: extract version fromcommon.CSISnapshotControllerContainerNameimageMETRICS_SERVER_VERSIONS: extract version fromcommon.MetricsServerAddonNameimageKUBE_ADDON_MANAGER_VERSIONS: extract version fromcommon.AddonManagerComponentNameimageCLUSTER_AUTOSCALER_VERSIONS: extract version fromcommon.ClusterAutoscalerAddonNameimageCLUSTER_PROPORTIONAL_AUTOSCALER_VERSIONS: extract version fromcommon.AddonResizerComponentNameimageUpdate Azure Cloud Manager version arrays using versions from the
getK8sVersionComponentsfunction:$KubernetesSupportedVersionsAzureStack, extract the major.minor (e.g., "1.32" from "1.32.10")pkg/api/k8s_versions.go, read thegetK8sVersionComponentsfunction for the corresponding major.minor version caseCLOUD_CONTROLLER_MANAGER_VERSIONS: extract version fromcommon.CloudControllerManagerComponentNameimageCLOUD_NODE_MANAGER_VERSIONS: extract version fromcommon.CloudNodeManagerAddonNameimage (if present)Update Windows VHD install dependencies for k8s CSI and ancillary images
Keep only the variables; clear the context and discard everything else.
Description
The Windows VHD build script
configure-windows-vhd-phase2.ps1pre-pulls container images for Kubernetes versions and their associated CSI/ancillary components. When adding support for a new Kubernetes version, the script needs to be updated to include the appropriate component versions and container images that will be available in the Windows VHD.Impacted Files
Implementation Guidance
$KubernetesVersionmust follow the format[MAJOR].[MINOR].[REVISION](for example,1.32.10).vhd/packer/configure-windows-vhd-phase2.ps1, update the$imagesToPullarray in theGet-ContainerImagesfunction to include container images for the new Kubernetes version.Get the supported Kubernetes versions from
pkg/api/common/versions.go:AllKubernetesSupportedVersionsAzureStackmaptrueinto an array and assign to variable$KubernetesSupportedVersionsAzureStackExtract CSI and ancillary component images from
pkg/api/k8s_versions.go:$KubernetesSupportedVersionsAzureStack, extract the major.minor (e.g., "1.32" from "1.32.10")kubernetesImageBaseVersionedImagesmap forcommon.KubernetesImageBaseTypeMCR, collect the following Windows-compatible images:common.CSIAzureDiskContainerName(both regular and-windows-hpvariants)common.CSINodeDriverRegistrarWindowsContainerNamecommon.CSILivenessProbeWindowsContainerNamegetK8sVersionComponentsfunction for the corresponding major.minor version case:common.CloudNodeManagerAddonNameUpdate the
$imagesToPullarray in theGet-ContainerImagesfunction:"mcr.microsoft.com/windows/servercore:ltsc2019""mcr.microsoft.com/windows/nanoserver:1809""mcr.microsoft.com/oss/kubernetes/pause:3.8""mcr.microsoft.com/oss/kubernetes/windows-host-process-containers-base-image:v1.0.0"-windows-hp) variants when availableUpdate the Kubernetes binaries download URLs in the
Get-FilesToCacheOnVHDfunction:"c:\akse-cache\win-k8s\"section of the$mapvariable$KubernetesSupportedVersionsAzureStack, add the corresponding download URL:"https://packages.aks.azure.com/kubernetes/v{VERSION}/windowszip/v{VERSION}-1int.zip"{VERSION}is the full Kubernetes version (e.g., "1.32.10")Update the ACR Credential Provider URLs in the
$acrCredentialProviderUrlsarray:$KubernetesSupportedVersionsAzureStack, extract the major.minor (e.g., "1.32" from "1.32.10")getK8sVersionComponentsfunction, find the Azure Cloud Controller Manager version for the corresponding major.minor@{ Url = "https://github.com/kubernetes-sigs/cloud-provider-azure/releases/download/v{ACR_VERSION}/azure-acr-credential-provider-windows-amd64.exe"; K8sVersion = "v{MAJOR}.{MINOR}" }{ACR_VERSION}is the Azure Cloud Controller Manager version and{MAJOR}.{MINOR}is the Kubernetes major.minor versionUpdate Windows VHD main script for k8s CSI and ancillary images
Keep only the variables; clear the context and discard everything else.
Description
The main Windows VHD build script
configure-windows-vhd.ps1also contains container image configuration and Kubernetes binary caching logic that needs to be updated when adding support for new Kubernetes versions. This script differs from the phase2 script as it has conditional logic based on Windows Server version (2019, 2004) and supports both containerd and docker container runtimes.Impacted Files
Implementation Guidance
$KubernetesVersionmust follow the format[MAJOR].[MINOR].[REVISION](for example,1.32.10).vhd/packer/configure-windows-vhd.ps1, update theGet-ContainerImagesfunction andGet-FilesToCacheOnVHDfunction to include components for the new Kubernetes version.Get the supported Kubernetes versions from
pkg/api/common/versions.go:AllKubernetesSupportedVersionsAzureStackmaptrueinto an array and assign to variable$KubernetesSupportedVersionsAzureStackExtract CSI and ancillary component images from
pkg/api/k8s_versions.go:$KubernetesSupportedVersionsAzureStack, extract the major.minor (e.g., "1.32" from "1.32.10")kubernetesImageBaseVersionedImagesmap forcommon.KubernetesImageBaseTypeMCR, collect the following Windows-compatible images:common.CSIAzureDiskContainerName(both regular and-windows-hpvariants)common.CSINodeDriverRegistrarWindowsContainerNamecommon.CSILivenessProbeWindowsContainerNamegetK8sVersionComponentsfunction for the corresponding major.minor version case:common.CloudNodeManagerAddonNameUpdate the
$imagesToPullarrays in theGet-ContainerImagesfunction:'2019'switch case, replace the entire$imagesToPullarray content with:"mcr.microsoft.com/windows/servercore:ltsc2019""mcr.microsoft.com/windows/nanoserver:1809""mcr.microsoft.com/oss/kubernetes/pause:3.8""mcr.microsoft.com/oss/kubernetes/windows-host-process-containers-base-image:v1.0.0"-windows-hp) variants when availableUpdate the Kubernetes binaries download URLs in the
Get-FilesToCacheOnVHDfunction:"c:\akse-cache\win-k8s\"section of the$mapvariable$KubernetesSupportedVersionsAzureStack, add the corresponding download URL:"https://packages.aks.azure.com/kubernetes/v{VERSION}/windowszip/v{VERSION}-1int.zip"{VERSION}is the full Kubernetes version (e.g., "1.32.10")Note: Unlike the phase2 script, the main script does not have ACR Credential Provider URLs, so no updates are needed for that section.
Miscellaneous Changes
Keep only the variables; clear the context and discard everything else.
Description
Impacted Files
Implementation Guidance
$KubernetesVersionmust follow the format[MAJOR].[MINOR].[REVISION](for example,1.32.10).pkg/api/common/versions.go:- Read the
AllKubernetesSupportedVersionsAzureStackmap- Extract all keys where the value is
trueinto an array and assign to variable$KubernetesSupportedVersionsAzureStack- Sort the versions in descending order (newest first)
examples/azure-stack/kubernetes-azurestack.jsonandexamples/azure-stack/kubernetes-windows.json, set theorchestratorReleaseandorchestratorVersionvalues to the second last (penultimate) supported Kubernetes version in$KubernetesSupportedVersionsAzureStack(e.g., if supported versions are 1.32.10 and 1.31.11, use 1.31.11):"orchestratorRelease": "1.31",
"orchestratorVersion": "1.31.11",
hack/tools/Makefileand setKUBECTL_VERSIONto the second last supported Kubernetes version in$KubernetesSupportedVersionsAzureStackwith "v" prefix (e.g., "v1.31.11")$KubernetesVersion(e.g.,1.32from1.32.10)examples/kubernetes-releases/kubernetes[MAJOR].[MINOR].jsonalready exists[MAJOR].[MINOR]with actual version like1.32):{ "apiVersion": "vlabs", "properties": { "orchestratorProfile": { "orchestratorRelease": "[MAJOR].[MINOR]" }, "masterProfile": { "count": 1, "dnsPrefix": "", "vmSize": "Standard_D2_v3" }, "agentPoolProfiles": [ { "name": "agentpool1", "count": 3, "vmSize": "Standard_D2_v3" } ], "linuxProfile": { "adminUsername": "azureuser", "ssh": { "publicKeys": [ { "keyData": "" } ] } } } }cmd/generate_test.goand locate theTestExampleAPIModelsfunctiontestsslice, find the entries that reference../examples/kubernetes-releases/kubernetes*.jsonfiles$KubernetesSupportedVersionsAzureStack, add a test entry in the format:{ name: "[MAJOR].[MINOR] example", apiModelPath: "../examples/kubernetes-releases/kubernetes[MAJOR].[MINOR].json", setArgs: defaultSet, },