Skip to content

test(runner): make the lease-expiry test observe the clock, not race it - #119

Merged
abrichr merged 1 commit into
mainfrom
fix/desktop-flaky-lease-expiry
Aug 19, 2026
Merged

test(runner): make the lease-expiry test observe the clock, not race it#119
abrichr merged 1 commit into
mainfrom
fix/desktop-flaky-lease-expiry

Conversation

@abrichr

@abrichr abrichr commented Aug 19, 2026

Copy link
Copy Markdown
Member

What

Remove the wall-clock race in
TestLeaseDiscipline::test_expired_unrenewed_lease_cannot_report_false_success
(added in #116).

Why

main failed on ubuntu-latest / Python 3.11:

FAILED tests/test_engine/test_runner_loop.py::TestLeaseDiscipline::test_expired_unrenewed_lease_cannot_report_false_success - assert 0 == 1

The test gave the lease an 80 ms life and slept 120 ms inside the run. But
RunnerService._handle_job stages the bundle and binds the effective policy
between reading the lease and the pre-start check
if datetime.now(timezone.utc) >= lease_deadline: raise Refusal(...). When that
work took longer than 80 ms on a loaded runner the dispatch was refused before
execution, so flow.run was never called and len(flow.calls) == 1 failed
with 0.

This is a flaky test, not a product defect: refusing a dispatch whose lease is
already dead is the correct fail-closed behaviour.

How

Drive the sequence by observation instead of by guessed durations:

  • Patch engine.runner_loop.datetime with a real clock plus a test-controlled
    offset, and give the lease 30 s so the pre-start check always passes.
  • The patched run body waits for the renew loop's first extend attempt, then
    moves the clock 60 s past the deadline, then holds the run open until the
    renew loop stops attempting extends — which proves it observed the expiry.

Every assertion is unchanged: the run executed exactly once, at least one extend
was attempted, the ack outcome is uncertain, no run_summary reached the
wire, and the journal records uncertain. Nothing is skipped or relaxed.

Tests

  • -k expired_unrenewed: 5 consecutive runs passed, ~0.16 s each.
  • Full suite: 951 passed, 6 skipped.
  • ruff check engine/ tests/ scripts/: passed.

🤖 Generated with Claude Code

`test_expired_unrenewed_lease_cannot_report_false_success` gave the lease an
80 ms life and slept 120 ms inside the run. The runner refuses a dispatch whose
lease is already dead at the pre-start check, and bundle staging plus policy
binding sit between the lease read and that check. When those took longer than
80 ms the job was refused before execution, `flow.calls` stayed empty, and the
test failed with `assert 0 == 1`. It did exactly that on `main` for
ubuntu-latest / Python 3.11.

Drive the sequence by observation instead:

- Patch `engine.runner_loop.datetime` with a real clock plus a test-controlled
  offset, and give the lease 30 s so the pre-start check always passes.
- The patched run body waits for the renew loop's first extend attempt, then
  moves the clock past the deadline, then holds the run open until the renew
  loop stops attempting extends, which proves it observed the expiry.

Every assertion is unchanged: the run executed once, at least one extend was
attempted, the ack is `uncertain`, no `run_summary` reached the wire, and the
journal records `uncertain`. The test now takes about 0.16 s instead of 0.12 s
of fixed sleeps, and passed five consecutive local runs.

Tests: full suite 951 passed, 6 skipped. Ruff passed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@abrichr
abrichr merged commit 8bebe4b into main Aug 19, 2026
17 checks passed
@abrichr
abrichr deleted the fix/desktop-flaky-lease-expiry branch August 19, 2026 22:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant