[Feature]: Add end-to-end cloud tests for Python SDK on Lambda Managed Instances

[zhongkechen]

What would you like?

Add an end-to-end cloud test suite that deploys the Python Durable Execution SDK to Lambda Managed Instances (LMI) and validates real service/runtime behavior, including multiple concurrent invocations in one execution environment, invocation deadlines, thread cleanup, and checkpoint/replay.

This should provide cloud regression coverage for #741 (missing invocation deadline handling and unbounded branch cleanup), as well as a reusable LMI test deployment and runner. Local mocks cannot establish whether a timed-out invocation releases its runtime worker, whether an old attempt continues producing side effects, or whether other workers in the same environment remain healthy.

The repository already has:

• A cloud example workflow that builds the checkout, deploys with SAM, runs cloud-mode pytest, and produces JUnit summaries.
• A cloud test runner supporting real invocations, asynchronous starts, callbacks, and execution-history inspection.
• Example test fixtures that select cloud mode and resolve deployed function names.

The inspected deployment/workflow has no LMI capacity-provider configuration or LMI-specific assertions. Reuse these foundations and add dedicated LMI fixtures and deployment support.

Possible Implementation

1. Add an explicit LMI deployment target.

• Deploy functions with durability enabled, an LMI capacity provider, explicit PerExecutionEnvironmentMaxConcurrency, and a published version/qualified invocation target. Build and package the core SDK from the exact checkout being tested.
• Use Python 3.14 as the single LMI test runtime. Test environment concurrency 1 as a baseline and at least one explicit value greater than 1 (for example 2 or 4). Vary SDK map/parallel concurrency independently; it is a separate limit.
• Accept region, capacity-provider configuration, networking, runtime, and concurrency through the deployment/test configuration. Verify the resulting function version is active and actually uses LMI before running assertions.
• Make prerequisites explicit: the selected account/region/runtime must support the required LMI + durability configuration. Unsupported provisioning or unavailable capacity must be reported distinctly; never fall back silently to standard Lambda or count an unexecuted LMI scenario as passing.
• Use one persistent deployment for the complete Python 3.14 suite, following Java PR ci: skip integration/otel tests for external pull requests #728. Minimize functions by deploying only two shared handlers: one with native environment concurrency 1 and one with native environment concurrency 2. Reuse each handler for ordinary, deadline, replay, and cleanup cases with a common 60-second invocation timeout; wait for the actual service timeout in deadline tests. Do not duplicate functions per scenario or deploy separately per concurrency setting. Keep each function limited to one environment and avoid extra published versions. Use fixed stack/function/bucket names and create them only when absent; later runs update the same functions. Use per-run prefixes for data and serialize deployment/testing across runs. Before an update, verify persistent ownership tags and wait for earlier durable executions to finish. Failed stacks are retained for diagnosis; do not delete and recreate them automatically. Keep content-addressed code artifacts for updates and rollback.

2. Add the following scenarios.

Scenario	Required evidence/assertions
Basic cloud execution and replay	Exercise steps, nested child contexts, map/parallel, durable waits, callback completion, and retry. Check results and execution history. Completed operations must reuse stored successes/failures without rerunning completed user code.
Actual LMI multi-concurrency	Drive overlapping invocations with distinct inputs at environment concurrency > 1. Collect environment identity, process identity, request ID, execution ARN, and start/end events. Prove at least two overlapping invocations used different Python worker processes in the same execution environment. Verify results/checkpoints are attributed to the correct execution. Sending requests in parallel from pytest alone is insufficient evidence.
Warm process reuse and normal cleanup	Observe invocation entry/exit around the decorated handler, plus per-process thread/resource snapshots. Across repeated success, failure, and suspension/resume cases, verify invocation-owned handler/checkpoint/branch threads do not remain after the wrapper returns. Identify reused workers explicitly; do not assume the next request lands on the same process. Track thread count, file descriptors, and memory over bounded repeated runs against a warmed baseline, allowing for legitimate caches.
Invocation deadline during a step (#741)	Use a short configured Lambda invocation timeout and a controlled step that crosses it. Record platform timeout evidence, attempt identity, cancellation/exit, checkpoint activity, and external side effects. For cooperative work, assert no new user work is started after the cancellation cutoff defined by the fix. Verify worker capacity is reclaimed within the agreed bound; a caller receiving a timeout alone does not satisfy the test.
Early parallel/map completion (#741)	A fast branch meets first_successful/min_successful while a losing branch is held in controlled I/O inside a step. Distinguish winner selection, user result computation, and actual wrapper return. Assert the cancellation/cleanup contract chosen in #741, including bounded recovery for a non-cooperative fixture. Include nested pools and a branch waiting for a real in-flight synchronous checkpoint so cleanup ordering is exercised.
Isolation and capacity recovery	Keep known healthy work active in the same environment while another invocation times out or enters slow cleanup. Verify healthy workers continue, then observe the affected worker recover or be replaced according to the supported policy. Do not accept apparent recovery caused solely by autoscaling to unrelated environments.
Timeout/retry overlap and side effects	Exercise a configured service retry path, or an explicitly labeled driver-controlled retry when appropriate. Correlate attempts and write uniquely identified records to a test-owned external sink. Verify completed steps are skipped on replay, stored failures remain stable, and stale attempts follow the cancellation policy. Record repeated interrupted attempts separately: at-least-once steps may rerun, so the test must not incorrectly demand exactly-once execution.

Include the relevant lifecycle/progress guards from Java PR #728 using Python public APIs: root finally must complete before PENDING; two concurrent invocations of a shared decorated handler must progress independently; nested child/map/parallel operations must progress with single-worker branch pools; normal return and failure must settle in-flight work; and an abandoned child must reject a subsequent durable operation before executing its body. Python has no public shared-executor injection or stepAsync API, so use idiomatic equivalents and document that distinction.

For the #741 scenarios, land the infrastructure and failing regressions with explicit linkage to the fix. Any temporary expected failure must have a narrow condition, an issue reference, and strict unexpected-pass handling; it must remain visible in the report and be removed when the fix lands.

3. Make placement, timing, and side effects observable.

• Collect structured records containing run ID, execution ARN, request ID, environment identifier, per-process initialization identifier, PID, operation name, attempt, timestamp, and lifecycle phase. Use a documented environment identifier where available, or a test-only environment marker with safe cross-process initialization; PID alone is not globally unique.
• Keep infrastructure diagnostics outside the durable orchestration decision path. Use stable operation names. Perform test I/O and external barrier operations inside steps so the handlers still exercise the real public SDK model and remain replay-correct.
• Coordinate overlap and blocked I/O using a test-owned control endpoint or external store with explicit release signals and bounded emergency timeouts. Do not use arbitrary sleeps as the sole evidence of concurrency, and do not replace durable waits with blocking sleeps in ordinary workflow scenarios.
• Use before/after-wrapper diagnostics to distinguish user handler completion from SDK resource cleanup. For a wrapper that never returns, collect evidence from the driver/control endpoint and worker health probes. Post-timeout diagnostics must not depend solely on the timed-out invocation's response.
• Inspect real service execution history and checkpoint outcomes. Combine this with external side-effect records and runtime logs; absence of a late checkpoint is not proof that an old step stopped executing.
• Account for log/metric propagation delays with bounded polling. A run that cannot establish same-environment overlap or worker identity is inconclusive and must not pass its placement-dependent assertions.

4. Separate the three different timeouts.

Configure and report these independently:

• Lambda invocation timeout, which exercises LMI's non-terminating timeout behavior.
• Durable execution timeout, covering the logical execution across invocations.
• Test-driver/polling deadline, bounding how long CI waits.

The current cloud runner's result-wait timeout is a client-side polling limit, not a way to configure the deployed Lambda invocation timeout. Use actual function configuration and service/runtime evidence for timeout assertions. Give the driver enough additional time to observe cleanup/recovery after the invocation deadline without allowing an unbounded test run.

5. Integrate with CI and retain actionable artifacts.

• Add a dedicated LMI workflow/job using the existing repository's OIDC role, test-account verification, Hatch build/test commands, and cloud function mapping conventions. Preserve the current restrictions on privileged tests for forked PRs and Dependabot.
• Run the complete LMI e2e runtime/concurrency matrix automatically whenever a same-repository PR is opened, updated, or reopened (including Draft PRs), and on every push to main, including every merged change. Do not require labels, ready-for-review status, or path filters. Retain workflow_dispatch for manual reruns; do not add scheduled jobs. Preserve the existing privileged-test restrictions for forked PRs and Dependabot, while running the unprivileged harness checks on those PRs.
• Give every triggered commit isolated invocation, event, and control data while sharing the persistent functions. Serialize all LMI cloud jobs across PRs, main pushes, and manual runs with one repository-wide concurrency group that covers deployment, testing, diagnostics, and cleanup. Use cancel-in-progress: false and queue: max so pending jobs queue instead of replacing another commit's job; GitHub retains up to 100 pending jobs and cancels additional arrivals once full. Keep harness jobs independent. Preserve the existing provider capacity; do not increase MaxVCpuCount. This group only coordinates jobs in this repository, so manual local deployments and other repositories using the provider still need capacity coordination.
• Run all 28 cases in one cloud job against that deployment, preserving both native concurrency configurations. After each case's assertions, release its controls, stop remaining logical executions, and verify all observed wrappers exit before reusing its function. Cleanup must not turn a deadline/recovery regression into a pass. If case retirement fails, the next case must retire the outstanding work before invoking the shared function.
• Upload JUnit results and a concise summary, deployed SDK commit/version, runtime and LMI configuration, qualified function targets, execution histories, lifecycle/placement records, side-effect ledger, and relevant CloudWatch logs/metrics. Failure output should identify the scenario and affected requests/workers.
• Bound each scenario and the overall job. Run cleanup releases only that run's blocked fixtures, stops its remaining test executions, and waits for observed wrappers to exit; stopping a durable execution alone is not proof that LMI code stopped. Keep the persistent functions, stack, bucket, and code artifacts after successful, failed, or canceled tests. Use isolated runs/RUN_ID/events/ and runs/RUN_ID/control/ prefixes with one-day expiry; code artifacts do not expire automatically. Infrastructure retirement remains under the test-account owner's explicit lifecycle policy.
• Ensure failures and cancellations still produce diagnostics and run-scoped cleanup. Remove the automatic infrastructure janitor. After an interrupted run, wait for prior executions before updating; if they do not settle within the budget, fail setup without changing code. Re-running the suite must reuse the same two functions and avoid cross-run test-state collisions.

Acceptance criteria:

• The full LMI cloud matrix runs automatically on every same-repository PR update (including Draft PRs) and every push to main, without labels or path filters; fork/Dependabot credential restrictions remain explicit and another commit cannot replace a pending run.
• At most one LMI cloud job from this repository holds provider capacity at a time, through cleanup; pending jobs use the GitHub maximum queue without changing provider capacity.
• One persistent deployment with only two shared LMI functions runs every scenario at native concurrency 1 and 2. Consecutive runs update the same names/ARNs; tests never delete the functions, stack, bucket, or code artifacts.
• CI can deploy the checkout as durable functions on a verified LMI target and invoke a published version.
• The suite uses Python 3.14 only, with a concurrency-1 baseline and concurrency > 1; unsupported provisioning is explicitly reported.
• A test proves overlapping invocations in different Python processes within the same execution environment.
• Deadline and early-completion scenarios reproduce the failures tracked by [Bug]: Missing invocation deadlines and unbounded branch cleanup can pin LMI workers #741 and become passing regression tests when its cancellation/cleanup contract is implemented.
• Worker recovery and unaffected concurrent work are verified without autoscaling or unrelated placement masking the result.
• Success, failure, suspension, retry, replay, and nested branch cleanup preserve checkpoint semantics and have observable resource-lifetime assertions.
• Reports distinguish invocation timeout, execution timeout, driver timeout, assertion failure, and provisioning/placement failure.
• No test can remain blocked indefinitely; CI failure/cancellation run cleanup, persistent create/update, prior-execution waiting, and cross-run isolation are exercised.
• A documented command allows maintainers to run the suite against the configured test account and inspect retained artifacts.

Is this a breaking change?

No. This adds test infrastructure, fixtures, and CI coverage. Any core SDK behavior changes belong to #741 or separate implementation issues.

Does this require an RFC?

No for the test harness itself. Any new runtime cancellation/worker-retirement contract discovered while implementing #741 should be designed and reviewed separately; these tests should validate that agreed contract.

Additional Context

• Related core SDK bug: [Bug]: Missing invocation deadlines and unbounded branch cleanup can pin LMI workers #741
• Python runtime for Lambda Managed Instances: each worker process handles one invocation at a time; process concurrency differs from SDK branch-thread concurrency.
• LMI execution-environment lifecycle: no freeze between invocations and no forced code termination at invocation timeout.

The initial evidence behind #741 was local source inspection and reproducible mocked-service tests. PR #743 now records the real-LMI validation, including the newly discovered late-operation defect added to #741. Runtime-specific tests should initially live in this Python repository. Any new language-neutral conformance requirements or shared runner changes should be coordinated with the conformance repository separately.