fix(e2e): replicate gallery images to a fixed region set so concurrent writers converge

[ganeshkumarashok]

Problem

publishingProfile.targetRegions is full desired state, not an atomic append. Each E2E writer did:

GET version -> append only its own region -> CreateOrUpdate

Two writers that read before either wrote compute different desired states:

	reads	writes
A (southeastasia)	[eastus]	[eastus, southeastasia]
B (uaenorth)	[eastus]	[eastus, uaenorth]

Whichever lands last wins outright, and the other region is silently dropped — surfacing later as GalleryImageNotFound on VMSS create.

vhdbuilder/packer/replicate-captured-sig-image-version.sh already documents the same constraint: "SIG API requires specifying a complete set of replication targets".

#8823 wrapped this in a sync.Mutex. That correctly serialises goroutines within one process, but the writers that actually collide are separate E2E pipelines in separate processes, which a mutex cannot reach.

Approach

A lost update is only harmful when writers disagree. So make them agree.

Every writer replicates to the same fixed set of E2E regions, so concurrent writes submit an identical desired state. Whichever lands last is still correct — no mutex, no barrier, no cross-process coordination, no ETag (the gallery image version API exposes none).

merged := existing ∪ E2ERegions ∪ {location}   // same for every writer

Steady state is free: once a version has every region, missing is empty, no write is issued, and the run is a GET plus the regional readiness wait.

What's here

• e2e/config/regions.go — the six E2E regions as constants, plus E2EReplicationRegions, IsE2ERegion, NormalizeRegion (ARM returns both "West US 2" and "westus2").
• Scenarios reference those constants instead of region literals, so the list is the definition rather than a copy that can drift. A scenario pinned to an unlisted region now fails fast with "add it to e2eRegions" instead of timing out waiting for replication that will never happen.
• Batched write — all missing regions go in one update instead of one update per region, with a 409/412 re-read-and-merge retry for the case where a writer slips in between the read and the write.
• Failure is scoped — replication is only fatal when the caller's own region is missing, so contention or a quota on a region a test never uses cannot fail that test.
• Image.Ephemeral — image versions captured at runtime for a single test stay single-region. They have exactly one writer and are deleted on cleanup, so there is no lost update to defend against and no reason to fan a throwaway captured OS image out to six regions.
• Retains the regional readiness wait from fix(e2e): wait on RegionalReplicationStatus before declaring SIG image usable #8374: a version appears in targetRegions before its regional replica actually serves traffic, and that gap is the direct cause of the 404s.

Scope

This only affects the standalone E2E pipelines that resolve images by tag (e2e.yaml, e2e-gpu.yaml, e2e-gpu-azurelinux.yaml, e2e-windows.yaml, e2e-tme.yaml, e2e-rcv1p-not-opted-in.yaml).

The VHD builder pipelines already set useVhdMetadataArtifacts: true and consume published VHD metadata (#8893); on that path GetVHDResourceID returns early and E2E never writes to the gallery at all. That is the strategic fix — this PR is a safety net for the pipelines not yet on it.

Related: if the REPLICATIONS pipeline variable is extended to cover the six E2E regions, images arrive pre-replicated, missing is always empty, and this code never issues a write. Worth doing independently.

Trade-off

Shared image versions get replicated to all six E2E regions rather than on demand. That is the price of writer agreement, and it is what makes the race disappear without locking. Ephemeral per-test images are exempt.

Known limitation

Two pipelines running different commits with different region lists can still disagree. That is inherent to a full-state API with no optimistic concurrency, and it self-heals on the next run.

Testing

go build ./..., go vet ./..., go test ./config (incl. -race), and the e2e test binary compiles. New unit tests cover the merge, the convergence property itself (two writers from different snapshots produce the same set), the ephemeral carve-out, region-set membership, and target-region lookup.

[Copilot]

Pull request overview

This PR refactors AgentBaker’s E2E image-replication flow to avoid per-scenario TargetRegions read/modify/write races by introducing a discovery-only “preflight” pass that computes a per-image region manifest, performs at most one replication update per selected image-version, and persists metadata for the subsequent scenario execution pass to use read-only lookups.

Changes:

• Add a discovery-only E2E preflight coordinator that batches desired/required regions per image and writes VHD metadata for the main run.
• Update Azure SIG replication logic to merge TargetRegions, retry on concurrent update conflicts, and wait on RegionalReplicationStatus.State == Completed for required regions.
• Improve resilience around GalleryImageNotFound by consulting replication state during VMSS creation retries, and stabilize random VHD selection across both passes.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
e2e/vmss.go	Classifies GalleryImageNotFound separately and consults replication state before retrying VMSS creation.
e2e/vhd_preflight.go	Adds preflight coordinator to batch desired/required regions per image and emit metadata.
e2e/vhd_preflight_test.go	Unit test for region batching/selection behavior and metadata output.
e2e/validators.go	Makes Ubuntu 22.04 FIPS detection resilient to VHD pointer copies by comparing distro identity.
e2e/test_helpers.go	Splits “scenario selection + normalization” from execution; supports preflight registration-only runs.
e2e/scenario_rcv1p_test.go	Skips subscription feature probing in preflight; lazily resolves branch CSE package URL in mutator.
e2e/scenario_helpers_test.go	Writes preflight metadata after tests complete when E2E_VHD_PREFLIGHT_OUTPUT is set.
e2e/config/vhd.go	Adds preflight entrypoints and metadata lookup semantics; adds VHDResourceID.Version() and random index override.
e2e/config/vhd_metadata_test.go	Tests metadata parsing for “missing image” entries and VHDResourceID.Version().
e2e/config/azure.go	Implements batched replication: merge desired regions, conflict retry, and per-region replication-status waiting.
e2e/config/azure_replication_test.go	Adds tests for replication-status lookup and target-region merge behavior.
.pipelines/scripts/e2e_run.sh	Runs discovery-only preflight pass, persists metadata, and sets a stable random VHD index for both passes.
CHANGELOG.md	Documents the high-level behavior shift and rationale.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

e2e/vmss.go:372

• The special GalleryImageNotFound handling is gated on s.Runtime.Cluster.Model.Location != nil. If that pointer is unexpectedly nil, we fall back to the generic retry loop and lose the intended fast-fail / wait-for-replication behavior. Since Scenario.Location is already available and is what the run is targeting, use it as a fallback (and only override it when the cluster model location is present).

		if isGalleryImageMissing && s.VHD != nil && s.Runtime != nil && s.Runtime.Cluster != nil && s.Runtime.Cluster.Model != nil && s.Runtime.Cluster.Model.Location != nil {
			location := *s.Runtime.Cluster.Model.Location
			toolkit.Logf(ctx, "GalleryImageNotFound on VMSS create in %s; consulting regional replication status before retrying", location)
			if waitErr := config.Azure.WaitForImageVersionReplicatedToRegion(ctx, s.VHD, location); waitErr != nil {
				// Surface the precise infra cause instead of a generic "failed after N retries".

e2e/config/azure.go:670

• If ensureTargetRegions fails, scoping the failure to “did my own region make it?” currently depends on a single follow-up GET (getImageVersion). If that GET fails transiently, we immediately return the original error even if the region update actually succeeded, which can add avoidable flakiness. Consider retrying the GET once (or a couple times) with a short delay before failing.

		live, getErr := a.getImageVersion(ctx, image, *version.Name)
		if getErr != nil || !hasTargetRegion(live.Properties.PublishingProfile.TargetRegions, location) {
			return err

e2e/config/azure.go:746

• The error message in updateImageVersion says "create a new images client", but this code is creating a gallery image versions client. Also prefer %w to preserve the wrapped error for callers.

	client, err := armcompute.NewGalleryImageVersionsClient(image.Gallery.SubscriptionID, a.Credential, a.ArmOptions)
	if err != nil {
		return fmt.Errorf("create a new images client: %v", err)
	}