[Cosmos] Move live tests to rotation-friendly fixed accounts

[tvaron3]

Is your feature request related to a problem? Please describe.

Python Cosmos key-auth live-test matrix jobs currently deploy sdk/cosmos/test-resources.bicep through the shared eng/common/TestResources flow, export ACCOUNT_HOST/ACCOUNT_KEY, run azpysdk whl/pytest marker lanes, and remove the account afterward (sdk/cosmos/tests.yml, eng/pipelines/templates/jobs/live.tests.yml). This per-run central provisioning model does not fit an ephemeral tenant that rotates roughly every three months, and it makes overlapping-run safety difficult to validate independently from provisioning.

The package has a broad sync/async test surface sharing TestConfig.TEST_DATABASE_ID; session hooks in tests/conftest.py create/delete a database and common containers. Specialized pytest lanes include split, query/long, multi-region/multi-write, circuit breaker, per-partition automatic failover, AAD, semantic reranker, and GSI. Some tests already use UUIDs and owned-resource cleanup, but fixed shared accounts require a deliberate cross-run interference audit before provisioning is disabled.

Prior art: Azure/azure-sdk-for-java#49735. Python should capture its lessons without assuming an identical implementation. Fixed accounts exposed global feed/count assumptions, weak random names, broad stale cleanup, non-thread-safe async collectors, destructive retries that reused resources, unit-only lifecycle hooks that accidentally reached live/emulator endpoints, and control-plane timeouts.

Describe the solution you'd like

Migrate only key-based Python Cosmos live tests to fixed, team-owned accounts that can be recreated after tenant rotation. Keep overlapping runs logically isolated; do not add a pipeline-wide lease or global build serialization.

Discovery and account inventory

• Inventory every account shape/topology/capability required by sync and async tests and map each key-auth pytest marker/matrix leg in sdk/cosmos/live-platform-matrix.json to a logical account selector.
• At minimum validate the current shapes implied by the matrix/Bicep: standard single-region Session, multi-write multi-region Session, single-write multi-region Session, circuit-breaker variants, per-partition automatic failover, split/throughput workloads, vector/full-text capabilities, and the separately configured GSI account.
• Determine whether semantic-reranker, integrated-cache, fault-injection, control-plane, or other specialized tests run in another pipeline/config and need dedicated accounts/selectors.
• Keep AAD/service-principal lanes (COSMOS_TEST_DATA_AUTH_MODE=aad) that require a different tenant or identity configuration outside this key-based migration unless their design is explicitly included.

Fixed-account provisioning and secret resolution

• Add a checked-in declarative account definition plus idempotent provisioning/reconciliation script under an appropriate Cosmos pipeline/test-resource directory; deploy into a team-owned resource group.
• Support dry-run/what-if and reconciliation so the same definitions recreate accounts after tenant rotation without embedding subscription or tenant details.
• Store all logical-account endpoints plus primary and secondary keys in one schema-versioned JSON secret sourced from Key Vault/ADO variable group. Commit only a schema and sanitized sample.
• Add resolver unit tests and a reusable pipeline pre-step that validates the JSON/schema, selects the matrix leg's logical account, masks secrets, and exports the existing Python variables (ACCOUNT_HOST, ACCOUNT_KEY, and any explicitly introduced secondary-key variable).
• Preserve matrix-controlled non-secret settings such as auth mode and account behavior flags rather than duplicating them in the secret.
• Disable the per-run deploy/remove path only after all fixed accounts and the versioned secret exist and the resolver succeeds. The exact Python template hook/bypass is an implementation decision; today tests.yml extends archetype-sdk-tests.yml, which invokes live.tests.yml deployment before BeforeTestSteps.

Concurrency hardening

• Make database, container, lease/processor, item, and other mutable resource names run-unique, preferably using UUID plus pipeline/run identity. Never rely on short/random-only names for isolation.
• Track ownership and delete only resources created by the current test/run. Remove broad stale-database/container cleanup from suites that may overlap.
• Replace account-global exact-count/feed assertions with semantic containment/ID assertions or queries scoped to the run-owned database/container.
• Audit tests/conftest.py, tests/test_config.py, class/function fixtures, cleanup finalizers, sync/async parametrization, retries, and client lifecycle. Ensure unit-only marker selections cannot accidentally connect to a fixed live account.
• Make async result collection and shared mutable state concurrency-safe; close async clients deterministically.
• Retry only bounded, known-transient operations. Do not retry destructive lifecycle sequences against a partially reused database/container.
• Isolate irreversible/heavy workloads with dedicated selectors/accounts where required: partition splits/throughput changes, change feed/processor leases, fault injection, control-plane operations, multi-region/session/consistency profiles, circuit breaker/failover, GSI, and other capability-gated tests.
• Diagnose logical name/state conflicts separately from account-level 429/capacity contention; account sizing or matrix throttling must not hide isolation defects.

Validation, recordings, and operations

• Add targeted key-auth sync and async live validation for each logical account selector, followed by overlapping matrix runs without a global lease.
• Document recording/playback implications. The live job currently sets AZURE_RUN_MODE=Live and AZURE_SKIP_LIVE_RECORDING=true, and the package has no checked-in tests/recordings tree; ensure resolver values never enter recordings/logs and playback/unit selections remain network-isolated.
• Add CI acceptance coverage for missing/malformed secret JSON, unknown selector, missing endpoint/key fields, masking/export behavior, provisioning dry-run/reconciliation, and failure before pytest when resolution is invalid.
• Add a tenant-rotation runbook covering account recreation, identity/RBAC prerequisites, secret regeneration/versioning, smoke validation, rollback, and ownership/on-call expectations.
• Prove per-run account creation and teardown are no longer used by migrated key-auth lanes, while unaffected AAD/emulator/specialized lanes retain their intended provisioning path.
• Prove overlapping runs do not delete or observe each other's resources and do not require global pipeline serialization.

No secrets, keys, account endpoints, or concrete subscription/tenant IDs belong in this issue or the repository.

Describe alternatives you've considered

1. Keep centrally provisioned accounts per run. This preserves isolation but does not meet rotation/self-ownership goals and retains the central provisioning dependency.
2. Serialize the entire Cosmos live pipeline or acquire a pipeline-wide lease. This reduces collisions but lowers throughput, risks abandoned leases, and masks unsafe shared-account assumptions.
3. Use one fixed account with broad stale-resource cleanup. This is unsafe for overlapping runs because one run can delete or count another run's resources.
4. Move AAD and key-auth tests together. This couples unrelated tenant/identity requirements; key-auth migration should land independently unless AAD account ownership is explicitly designed.

Additional context

Verified Python-specific surfaces:

• sdk/cosmos/tests.yml: live-test matrix, max parallelism 8, user-administered variable group, GSI secret mapping.
• sdk/cosmos/live-platform-matrix.json: pytest marker legs and ARM topology parameters.
• sdk/cosmos/test-resources.bicep: current account, regions, capabilities, role assignment, and ACCOUNT_HOST/ACCOUNT_KEY outputs.
• sdk/cosmos/azure-cosmos/pytest.ini: Cosmos live marker taxonomy.
• sdk/cosmos/azure-cosmos/tests/conftest.py and tests/test_config.py: session-wide database/container lifecycle, key/AAD client selection, UUID-based shared IDs, sync/async helpers, and split control-plane operations.
• test_change_feed.py/test_change_feed_async.py, split tests, and GSI tests show representative owned-resource, empty-feed/count, control-plane, and specialized-account behaviors that need auditing.

Unknowns to resolve during design: the complete set of live pipelines outside sdk/cosmos/tests.yml; whether GSI's existing dedicated secret should be folded into the versioned account map; secondary-key consumers; required account capacity/quotas under overlap; and which heavy tests need exclusive logical accounts rather than merely unique database/container names.