Skip to content

Fix flaky cloud retry example tests with retained SAM stacks #520

Description

@zhongkechen

Problem

The cloud example tests for retry behavior can fail after the SAM stacks are retained between workflow runs.

Failed run: https://github.com/aws/aws-durable-execution-sdk-python/actions/runs/28961742737

In that run, the retry examples failed in the cloud test matrix. One representative job:
https://github.com/aws/aws-durable-execution-sdk-python/actions/runs/28961742737/job/85934728847

Observed failures included:

  • packages/aws-durable-execution-sdk-python-examples/test/step/test_step_with_retry.py::test_step_with_retry
    • Expected the step to succeed on durable attempt 2.
    • Actual operation succeeded on attempt 1.
  • packages/aws-durable-execution-sdk-python-examples/test/step/test_steps_with_retry.py::test_steps_with_retry
    • Expected pollsRequired == 2.
    • Actual result was pollsRequired == 1.

Why This Happens

The examples currently simulate transient retry behavior with module-level counters:

  • step_with_retry.py uses _attempts = count(1).
  • steps_with_retry.py uses _attempts = count(1).

That works in a fresh local process, but it is not reliable in deployed Lambda functions. Because the cloud test workflow now keeps the SAM stacks after completion for troubleshooting, Lambda execution environments can stay warm across workflow runs. If a warm environment already advanced the module-level counter in a previous run, the next run may skip the intended first failure and succeed immediately.

That makes the tests assert stale assumptions:

  • test_step_with_retry expects one retry, but the function may succeed on attempt 1.
  • test_steps_with_retry expects the first poll to retry and the second poll to find the item, but the function may find the item on poll 1.

Notes

We briefly explored exposing the durable step attempt on StepContext, but that changes the public SDK interface and should not be done casually as part of the SAM workflow PR.

Potential follow-up directions:

  • Find a deterministic way for examples to inspect durable retry attempt metadata without changing public API.
  • Move these examples/tests away from module-level state.
  • Add test-only input or example logic that deterministically fails once per durable execution rather than once per Lambda execution environment.

Acceptance Criteria

  • The retry example tests remain deterministic when Lambda execution environments are reused.
  • The fix does not add a new public StepContext field unless that API change is intentionally designed and reviewed.
  • Cloud tests can retain stacks after completion without causing these retry examples to become order-dependent.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions