test: add LMI cloud lifecycle regressions - #743
zhongkechen wants to merge 10 commits into
Conversation
Add opt-in cloud deployment and lifecycle coverage for Python LMI. Assert invocation deadlines, early branch cleanup, replay, and worker recovery; keep issue 741 regressions red until the SDK fix.
The root only coordinates workspace packages. Disable its implicit editable install so the LMI test directory does not trigger setuptools' multiple top-level package discovery error when starting examples. Exercise examples environment initialization in the unprivileged LMI harness job and verify that it imports the checked-out SDK.
Run the complete cloud matrix on every trusted PR update, including drafts, and every main push without labels or path filters. Preserve existing cloud credential restrictions for forks and Dependabot. Use run-owned resources without a cross-run concurrency group so newer commits cannot replace another commit's pending cloud job.
Initialize explicit hold/release objects before invoking fixtures and report missing or forbidden controls as infrastructure failures. Signal the winner only after the losing step has entered held I/O, and reject cleanup assertions when that loser exited before the winner was ready. Poll events by case with bounded parallel reads, correlate checkpoint holds to their request, and check stale effects before waiting for a retry on a potentially pinned worker. Cover the observed false positives and full fixture handshakes with regression tests.
Keep LMI on Python 3.14 with environment concurrency one and two. Exercise root finally before PENDING, bounded nested progress, concurrent shared-handler invocations, in-flight return/failure cleanup, and late operations from abandoned children using Python public APIs. Document the Java/Python mapping and preserve the newly exposed late-step scope defect as an explicit issue 741 regression.
Collect each case's own executions during teardown and leave full-run history/log collection to the final workflow step. Pace history reads and bound throttle-only retries, including reads through the public cloud runner, without retrying invocation or callback writes. Cover pagination, throttling exhaustion, permission errors, and scoped versus full-run collection with focused tests.
Checkpoint a victim's placement decision before starting fault work. If a runtime environment rotates between anchor and victim admission, replace the pair with a bounded retry. Never retry an admitted lifecycle assertion. Keep the admission decision stable during replay.
Hold a repository-wide concurrency slot through deployment, testing, and cleanup without increasing shared provider capacity. Retain up to 100 pending cloud jobs with queue: max instead of evicting older jobs. Keep the harness parallel and document the queue and repository scope.
Deploy one function per native concurrency setting and reuse it for ordinary and deadline scenarios. Run all 28 cases in one cloud job without changing provider capacity or losing regression assertions. Release controls, stop remaining executions, and observe wrapper exit before reusing each function. Extend fault observation for the shared 60-second timeout and retain a bounded emergency release.
Create the persistent stack once and update the same two functions after previous durable executions finish. Preserve failed stacks, code artifacts, and infrastructure after every test run. Isolate controls and events by run ID, expire only run data, and scope cleanup to the current run without deleting functions or buckets. Remove the infrastructure janitor and verify create/update behavior.
Per-test analysis of LMI run 35918433206Scope: workflow run, attempt 1, PR head Evidence reviewed: job/step results, the two Overall: 28 cloud cases — 16 PASS, 12 assertion FAILURES, 0 ERRORS, 0 SKIPS. All 72 harness tests passed. Both cloud jobs successfully built, deployed/read back their functions, collected final evidence, deleted their temporary resources, and published artifacts. Only the cloud assertion steps failed. The ledgers contain no
Configuration and interpretation
Passing casesEach row passed on both c1 and c2 (eight cases per configuration).
The last two passes do not contradict the bounded-cleanup failures below: they verify that work settles after an explicit release, while the failing early-completion tests require bounded cleanup without that release. Failing casesEarly completion —
|
| Scenario | c1: wrapper return after winner | c2: wrapper return after winner | Effects recorded after winner +5 s (c1 / c2) |
|---|---|---|---|
parallel |
7.108 s | 7.709 s | 7 / 8 |
map |
7.209 s | 7.509 s | 7 / 8 |
nested |
6.908 s | 7.011 s | 6 / 6 |
These measured delays are terminated by test cleanup; they are not measurements of an infinite production hang. The source-level unbounded wait is the shutdown(wait=True) cleanup path described in #741 problem 2. Canceling queued futures does not stop an already-running branch blocked in user I/O. In affected workloads this can retain runtime capacity and allow unwanted downstream work after the winning result is selected.
test_invocation_deadline_isolation_and_capacity_recovery (one failure per configuration)
Scenario / expected: Hold a step across the actual 10-second Lambda invocation timeout, submit recovery demand around the deadline, and recover all configured worker slots in the original environment within deadline +5 s. c2 first establishes a healthy companion in that environment.
Actual: Both cases reached and failed #741: full original-environment worker capacity did not recover. Service history independently records InvocationCompleted with Sandbox.Timedout / Task timed out after 10.00 seconds for each original request. The external ledger still records the original request's effects well beyond that timeout:
| Configuration | Fault marker | Last observed effect after invocation deadline | Effects after deadline +5 s |
|---|---|---|---|
| c1 | deadline-baa254d57873 |
+16.043 s | 39 |
| c2 | deadline-76b8fd56a328 |
+16.233 s | 40 |
The original wrappers eventually raised after driver cleanup. Later execution completion cannot satisfy the earlier recovery bound. This directly supports #741 problem 1: a service timeout alone does not stop the LMI Python work or restore full worker capacity.
Scope limit: The c2 companion was released and completed before the deadline; this run establishes that early healthy execution, not continuous healthy-peer progress throughout timeout recovery. The full-capacity assertion still requires both slots, so a probe using only the already-released healthy slot cannot make recovery pass. The test aborts at the recovery assertion; subsequent in-test timing assertions were not reached. The service timeout and continued effects above were checked independently in the retained evidence.
test_service_timeout_retry_does_not_repeat_completed_step (one failure per configuration)
Scenario / expected: After a real invocation timeout, stale-attempt effects must cease within the cleanup bound, and a service retry of the same durable execution must reuse the completed step rather than repeat its body.
Actual: Both fail first at #741: work continued after invocation deadline + cleanup grace. Original-attempt effects continue until +7.549 s (c1) and +8.065 s (c2) after the deadline. In c2 (deadline-158ca58c11ed), a retry of the same execution enters another process in the same environment at deadline +1.078 s, while the original process continues producing effects. This supplies direct evidence of stale-attempt/retry overlap, within #741 problem 1.
Scope limit: The formal retry-wait and completed-step assertions occur after the failing stale-effect assertion and were not executed. Final evidence nevertheless records one success body and one service StepSucceeded for that completed step in each configuration, together with the later retry. That supplementary observation does not turn these cases into passes or establish exactly-once execution for interrupted at-least-once work. No completed-step duplication was observed in these records.
test_abandoned_child_rejects_late_durable_operation (one failure per configuration)
Scenario / expected: After race-parallel has completed, release its losing branch and attempt a new child.step(name="late-work") from that branch's finally. The new operation should raise OrphanedChildException before executing its body or creating a service operation.
Actual: Both fail at #741: abandoned child executed a late user side effect. The ledger records LATE_ATTEMPT, a late-work body, and LATE_ACCEPTED; service history independently confirms admission after ancestor completion:
| Configuration / marker | race-parallel ContextSucceeded |
late-work StepStarted and StepSucceeded |
|---|---|---|
c1 — late-operation-10904bc0b80a |
21:02:31.849 UTC | 21:02:33.321 UTC |
c2 — late-operation-c04640fcc324 |
21:13:27.221 UTC | 21:13:28.819 UTC |
All times are on 2026-09-23. This reproduces #741 problem 3 on both configurations, extending the earlier concurrency-2 evidence already recorded in that issue. The orphan guard checks the new operation's own ID, but that ID does not inherit the already-completed ancestor's state. This creates a risk of admitting new downstream actions after the enclosing operation has completed, distinct from merely allowing an existing I/O call to continue.
The demonstrated boundary is after ancestor completion. These late bodies execute before their outer wrapper's final return; the records do not establish new work beginning after wrapper return.
What this run establishes, and what remains pending
- All 12 cloud assertion failures in this run correspond to the three defects now tracked in [Bug]: Missing invocation deadlines and unbounded branch cleanup can pin LMI workers #741. No additional untracked SDK defect is established by this run. They remain ordinary failures, not
xfailor expected-success results. Passing cases and supplementary observations do not satisfy the portions of failed cases that were never reached. - The earlier run 35904960112 had a c1 capacity-provider provisioning failure; its c1 tests did not execute. That was an infrastructure failure, not an expected SDK regression. In the run analyzed here, both deployments and both cleanups succeeded.
- The tested revision used temporary
normal/deadlinedeployments with different timeouts. Current head7bd33b18c5320effbacb86206c8ab12ed9c8ddc3instead retains two persistent shared functions, runs all 28 cases in one job, uses a common 60-second invocation timeout, isolates data per run, and settles case work before reuse. Those deployment/lifecycle changes are not cloud-validated by the older run above. - At the time of posting, the current persistent-deployment run is queued. Current-head local validation has 93 passing harness tests, successful packaging/Ruff, and all 28 cloud cases collected. Cloud validation must still establish first creation, subsequent updates of the same functions, and case/run isolation with the new layout.
The SDK fixes remain in #741; this PR supplies the regressions and stays Draft.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
| - name: Run cloud regressions (red until issue 741 is fixed) | ||
| timeout-minutes: 20 | ||
| run: hatch run lmi:cloud --junitxml=lmi_tests/artifacts/cloud.xml |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_fultsmqjug3q6e3j44muf27nms
[P1] Do not enable known-failing regressions on every PR and main push. The base SDK still joins blocked losing branches, so the early-completion cases necessarily fail, making this workflow red for every trusted PR and main build. Keep the cloud regressions manual or non-blocking until #741 is fixed, or land the tests and fix together before enabling these triggers.
| fi | ||
| - name: Retain diagnostics | ||
| if: always() | ||
| uses: actions/upload-artifact@v7 |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_u2u3julqacr3i2xyjwmrhegldi
[P2] Pin both upload-artifact steps to an immutable commit. The mutable @v7 reference can change executed code, and this occurrence runs after AWS credentials are configured in a job with id-token: write. Replace both @v7 references with the repository-approved SHA; existing workflows pin v7.0.1 to 043fb46d1a93c77aae656e7c1c64a875d1fc6a0a.
| provider = os.environ["CAPACITY_PROVIDER_ARN"] | ||
| account = client("sts").get_caller_identity()["Account"] |
There was a problem hiding this comment.
Codex AI review · Finding arf_v1_junjdxjxbbvuhm6fpa5ys7zb2f
[P2] Clear or namespace prior run metadata before deployment can fail. Local runs reuse lmi_tests/artifacts, but manifest.json is replaced only after account, provider, capacity, and build validation. If any earlier check fails, the documented exit trap’s collect/cleanup commands load the previous manifest, potentially releasing or stopping the previous run and reporting its XML/cleanup status as current. Remove prior run-scoped artifacts before these fallible checks or store each run under its run ID while preserving build.json.
Codex AI reviewFound one blocking CI regression plus two operational/security issues in the new harness. Reviewed commit |
This branch was successfully deployed
1 participant
Adds real-LMI regression coverage for invocation deadlines, branch cleanup, and abandoned-child operations in #741, following aws/aws-durable-execution-sdk-java#728. One persistent deployment runs all 28 Python 3.14 cloud cases using two shared functions: native environment concurrency 1 and 2. Ordinary and deadline scenarios reuse the same function for their concurrency setting.
Closes #742. This PR does not fix or close #741 and remains Draft.
python-lmi-e2estack/bucket and fixed-namepython-lmi-e2e-c1/python-lmi-e2e-c2functions. Both use the same handler/artifact, a 60-second invocation timeout, exactly one execution environment, and$LATEST.PUBLISHED. Verify native concurrency, durability, scaling, qualified target, and artifact identity. Later runs retain names/ARNs, use content-addressed code, and refresh runtime run/commit identity as in Java PR ci: skip integration/otel tests for external pull requests #728. Before updating, verify ownership and wait for previous durable executions; busy or failed stacks block updates and remain available for diagnosis. No extra versions or provider-capacity changes are made.cancel-in-progress: falseandqueue: maxpreserve active work and up to 100 pending jobs. Harness jobs remain parallel. The group does not coordinate other repositories or local deployments sharing the provider.runs/RUN_ID/and expire after one day. Final cleanup releases/stops only the current run's work and observes wrapper exit; it keeps the functions, stack, bucket, and code artifacts after success or failure. There is no automatic infrastructure deletion or janitor. Content-addressed code remains available for rollback, and the test-account owner controls final infrastructure retirement.Validation:
7bd33b1), queued for GitHub runner allocation. Cloud create/update results are not yet available.git diff --checkis clean. Actionlint 1.7.12 passes with only its known unsupportedqueuekey diagnostic excluded (upstream issue); GitHub's schema and service support that property.The five-second recovery/cleanup bound is a proposed test acceptance criterion to review with #741, not a new SDK guarantee. Core cancellation/worker-retirement behavior belongs to that fix.