fix(automation): keep a vanished job lock contended instead of stale - #739
Merged
Conversation
Root cause of the Windows flake in jobs::concurrent_manual_job_triggers_do_not_double_execute (seen on the 2026-08-04 master run, Test Windows shard 2): when a manual trigger loses the create_new race and the winner finishes and releases the lock before the loser's staleness reads complete, lock_created_at reports no timestamp and lock_is_stale treated that as stale. The loser then "reclaimed" the released lock and executed the job a second time. A lock with no readable creation timestamp is now aged by its on-disk mtime and reported stale only when provably old; a vanished lock stays contended. Crash leftovers still get reclaimed: a dead-pid lock with payload ages by created_at, and an empty crash leftover ages by mtime (both covered by new regression tests). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A2BarcJk3iuv77aJQwHvnx
|
ScriptedAlchemy
added a commit
that referenced
this pull request
Aug 27, 2026
Absorbs the master-side CI stabilization. Delete-side conflicts resolve to this branch: the V2 resolver (lifecycle/identity.rs reject_split_identity_cutover) already fails closed on populated duplicate shards, path_layout.rs is already restructured past the clippy lint, and the automation lock code is rewritten here (the vanished-lock reclaim guard from #739 still needs a V2 port — see the PR #707 comment from 2026-08-27). Tree is identical to the previous branch head. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A2BarcJk3iuv77aJQwHvnx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Windows double-execution flake
jobs::concurrent_manual_job_triggers_do_not_double_executeflaked on the 2026-08-04 master run (Test Windows shard 2):backend.callswas 2, and nextest is configured fail-on-flaky.Race: a manual trigger loses the
create_newrace; the winner finishes the job and releases (deletes) the lock before the loser's staleness reads complete.lock_created_atthen finds no file,lock_is_stalereturnedtruefor the missing timestamp, and the loser "reclaimed" the just-released lock and ran the job a second time. Windows' slower file I/O widens the window (the test's backend also blocks the runtime for 200ms, so the loser's checks resume exactly when the winner is finishing).Fix: a lock with no readable creation timestamp is aged by its on-disk mtime and reported stale only when provably older than the stale window; a vanished lock stays contended. Crash recovery is preserved: dead-pid locks with payload age by
created_atas before, and an empty crash leftover ages by mtime — both pinned by new regression tests (task_lock_does_not_steal_fresh_empty_lock_file,task_lock_reclaims_empty_lock_file_older_than_stale_window).Verification: full
automation_runner_testsuite 154/154 locally; scoped clippy-D warningsclean on both changed targets;cargo fmt --checkclean.Note for #707:
crates/tracedecay-agent-hosts/src/automation/scheduler.rson the V2 branch has the samecreate_new+ stale-reclaim pattern and needs the same port.🤖 Generated with Claude Code
https://claude.ai/code/session_01A2BarcJk3iuv77aJQwHvnx