test(e2e): schema-upgrade coverage — auto, rollback, HA, unpullable version

[WentingWu666666]

What

Adds e2e specs to the upgrade area covering the gaps tracked in #439 (deferred from #426).

Included in this PR

• schemaVersion: "auto" single-step path (upgrade_schema_auto_test.go) — a single spec.documentDBVersion bump migrates both the binary and the extension schema in one step (operator runs ALTER EXTENSION documentdb UPDATE automatically, no finalize patch). Asserts status.schemaVersion advances on its own and data is retained.
• Rollback within the safe window (upgrade_schema_rollback_test.go) — in two-phase mode the installed schema stays at the old version, so reverting spec.documentDBVersion before finalize is admitted by the image-rollback webhook. Asserts the cluster returns to Ready on the old binary, the schema is still old, and data survives the down-hop.
• Multi-instance / HA rollout (upgrade_schema_ha_test.go) — runs the two-phase upgrade on a 3-instance cluster, asserts the cluster returns to 3 ready instances after the binary roll and the schema finalize, and execs psql on a replica pod to confirm pg_extension.extversion converged to the new version via WAL streaming replication (the operator computes status.schemaVersion from the primary only, so the replica is verified directly).
• Invalid / unpullable documentDBVersion (2nd ordered spec in upgrade_schema_test.go) — reuses the existing cluster (no extra DocumentDB). Patches to a version above the installed schema (0.999.0, so the rollback webhook admits it but no image exists); asserts the schema does not advance and seeded data stays readable while pods can't pull, then rolls back and asserts recovery.

Supporting changes:

• New schema_version mixin so specs can set spec.schemaVersion at creation.
• replicaInstalledSchemaVersion helper (reads the extension version on a replica via CNPG EventuallyExecCommand).
• Hoisted shared cluster creation in upgrade_schema_test.go into BeforeAll so the happy-path and unpullable-version specs share one cluster.

Deferred to a follow-up PR (needs operator change)

Per the design discussion on #439, these pair with a proposed fail-fast update-path preflight in the operator and will land together:

• Skipping more than one minor (jump >1 version)
• Migration failure (ALTER EXTENSION UPDATE fails) — realistically triggered by a broken update-path, which the preflight introduces
• Multi-version chain (folded into the jump/preflight work)

Verification

go build ./..., go vet, gofmt, and Ginkgo --dry-run all clean; all new specs discovered.

Coverage tracked in #439

• spec.schemaVersion: "auto" single-step path
• Rollback within the safe window (binary downgrade before finalize)
• Multi-instance / HA rollout ordering
• Invalid / unpullable documentDBVersion
• Migration failure (ALTER EXTENSION UPDATE fails) — follow-up with preflight
• Skipping more than one minor (jump >1 version) — follow-up with preflight
• Multi-version chain — follow-up with preflight

Refs #439.

[documentdb-triage-tool]

🤖 Auto-triaged by documentdb-triage-tool.

Applied: test, enhancement, performance
Project fields suggested: Component test · Priority P2 · Effort M · Status In Progress
Confidence: 0.88 (mixed)

Reasoning

component from path globs (test); effort from diff stats (373+0 LOC, 3 files); LLM: Adds new e2e test specs covering schema-upgrade auto-mode and safe-window rollback paths tracked in an existing coverage gap issue, with a reusable mixin; purely test-layer changes across a couple of files.

If a label is wrong, remove it manually and ping @patty-chow so the rules can be tuned. The bot will not re-label items that already have component labels.

[Copilot]

Pull request overview

Expands the e2e upgrade test suite to cover additional DocumentDB schema-upgrade scenarios (auto mode, rollback window, HA behavior, and an unpullable version failure path), and adds supporting helpers/manifests to drive spec.schemaVersion at creation time and to verify replica schema convergence via psql.

Changes:

• Add new upgrade e2e specs covering: schemaVersion: auto, rollback before finalize, HA (3 instances) migration, and unpullable documentDBVersion recovery.
• Add a schema_version manifest mixin to set spec.schemaVersion during initial resource creation.
• Add a helper to read pg_extension.extversion on replica pods to assert schema propagation via WAL streaming replication.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
test/e2e/tests/upgrade/upgrade_schema_test.go	Shares a single cluster across ordered specs and adds an unpullable-version recovery test.
test/e2e/tests/upgrade/upgrade_schema_auto_test.go	New spec for single-step schema migration when schemaVersion is "auto".
test/e2e/tests/upgrade/upgrade_schema_rollback_test.go	New spec validating rollback-safe window before schema finalize.
test/e2e/tests/upgrade/upgrade_schema_ha_test.go	New HA spec validating 3-instance rollout and replica schema convergence.
test/e2e/tests/upgrade/helpers_test.go	Adds replica schema version helper using CNPG exec to query pg_extension.
test/e2e/manifests/mixins/schema_version.yaml.template	Adds mixin to set spec.schemaVersion via templated manifests.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[xgerman]

Additional test coverage concerns

The new scenarios cover the primary happy paths well, but these gaps could leave false confidence:

• Validate all installed DocumentDB extensions. The HA helper checks only documentdb. A dependent extension could remain stale or broken while that check passes. Verify every installed DocumentDB-related extension on the primary and each replica.

• Add version-skew and patch-version cases. The tests cover a clean adjacent-version transition only. Please cover an installed schema newer than the requested target, patch versions without an exact migration path, and targets below the installed schema. The expected behavior should be explicit: safely skip/reject the change or recover without corrupting state.

• Test migration failure and recovery. The PR currently defers ALTER EXTENSION failure coverage. Add a failure-path test that verifies the error is surfaced in status/events, the cluster does not falsely report Ready, the recorded schema version remains truthful, and retry/rollback recovers safely.

• Test multi-version jumps. Verify that skipping multiple minor versions either follows a supported chain or fails fast without partially changing the binary/schema state.

• Avoid masking status-read failures. schemaVersionGetter returns the last successful value when a status fetch fails. Consequently, Consistently can pass during a prolonged API/status outage. The assertion should surface fetch errors or track them separately.

• Strengthen the unpullable-image test. It currently verifies that data remains readable, but does not prove that the requested image was attempted and failed. Assert that the desired version is persisted, the workload targets it, pods expose the expected image-pull failure, and status reports the failure while the existing primary remains available.

• Validate migrated objects, not only extversion. Checking the catalog version does not prove that migration-created or altered functions, tables, indexes, permissions, triggers, and other objects work. Add representative post-migration operations for objects affected by the release.

• Exercise failover around schema migration. After migration, promote a replica and verify the promoted primary has the complete extension/object state and continues serving traffic. This would cover rollout/failover ordering that the current end-state checks do not exercise.