Problem
The cloud example tests for retry behavior can fail after the SAM stacks are retained between workflow runs.
Failed run: https://github.com/aws/aws-durable-execution-sdk-python/actions/runs/28961742737
In that run, the retry examples failed in the cloud test matrix. One representative job:
https://github.com/aws/aws-durable-execution-sdk-python/actions/runs/28961742737/job/85934728847
Observed failures included:
packages/aws-durable-execution-sdk-python-examples/test/step/test_step_with_retry.py::test_step_with_retry
- Expected the step to succeed on durable attempt
2.
- Actual operation succeeded on attempt
1.
packages/aws-durable-execution-sdk-python-examples/test/step/test_steps_with_retry.py::test_steps_with_retry
- Expected
pollsRequired == 2.
- Actual result was
pollsRequired == 1.
Why This Happens
The examples currently simulate transient retry behavior with module-level counters:
step_with_retry.py uses _attempts = count(1).
steps_with_retry.py uses _attempts = count(1).
That works in a fresh local process, but it is not reliable in deployed Lambda functions. Because the cloud test workflow now keeps the SAM stacks after completion for troubleshooting, Lambda execution environments can stay warm across workflow runs. If a warm environment already advanced the module-level counter in a previous run, the next run may skip the intended first failure and succeed immediately.
That makes the tests assert stale assumptions:
test_step_with_retry expects one retry, but the function may succeed on attempt 1.
test_steps_with_retry expects the first poll to retry and the second poll to find the item, but the function may find the item on poll 1.
Notes
We briefly explored exposing the durable step attempt on StepContext, but that changes the public SDK interface and should not be done casually as part of the SAM workflow PR.
Potential follow-up directions:
- Find a deterministic way for examples to inspect durable retry attempt metadata without changing public API.
- Move these examples/tests away from module-level state.
- Add test-only input or example logic that deterministically fails once per durable execution rather than once per Lambda execution environment.
Acceptance Criteria
- The retry example tests remain deterministic when Lambda execution environments are reused.
- The fix does not add a new public
StepContext field unless that API change is intentionally designed and reviewed.
- Cloud tests can retain stacks after completion without causing these retry examples to become order-dependent.
Problem
The cloud example tests for retry behavior can fail after the SAM stacks are retained between workflow runs.
Failed run: https://github.com/aws/aws-durable-execution-sdk-python/actions/runs/28961742737
In that run, the retry examples failed in the cloud test matrix. One representative job:
https://github.com/aws/aws-durable-execution-sdk-python/actions/runs/28961742737/job/85934728847
Observed failures included:
packages/aws-durable-execution-sdk-python-examples/test/step/test_step_with_retry.py::test_step_with_retry2.1.packages/aws-durable-execution-sdk-python-examples/test/step/test_steps_with_retry.py::test_steps_with_retrypollsRequired == 2.pollsRequired == 1.Why This Happens
The examples currently simulate transient retry behavior with module-level counters:
step_with_retry.pyuses_attempts = count(1).steps_with_retry.pyuses_attempts = count(1).That works in a fresh local process, but it is not reliable in deployed Lambda functions. Because the cloud test workflow now keeps the SAM stacks after completion for troubleshooting, Lambda execution environments can stay warm across workflow runs. If a warm environment already advanced the module-level counter in a previous run, the next run may skip the intended first failure and succeed immediately.
That makes the tests assert stale assumptions:
test_step_with_retryexpects one retry, but the function may succeed on attempt1.test_steps_with_retryexpects the first poll to retry and the second poll to find the item, but the function may find the item on poll1.Notes
We briefly explored exposing the durable step attempt on
StepContext, but that changes the public SDK interface and should not be done casually as part of the SAM workflow PR.Potential follow-up directions:
Acceptance Criteria
StepContextfield unless that API change is intentionally designed and reviewed.