Skip to content

feat: recover local workers on macOS and Windows - #1305

Draft
kvlonge wants to merge 2 commits into
feat/orchestration-backendfrom
feat/non-linux-recovery
Draft

kvlonge wants to merge 2 commits into
feat/orchestration-backendfrom
feat/non-linux-recovery

Conversation

@kvlonge

@kvlonge kvlonge commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Why

Scheduler recovery is only proven on Linux. After a scheduler restart on macOS or Windows, local-worker claims are skipped, so a surviving worker keeps its run and write claims with nothing able to stop it. This PR makes recovery work on both platforms and tests it on their runners.

Changes

  • New crate sqlbuild-process-ownership, the only crate allowed unsafe code (item-level allows with SAFETY comments behind a safe prepare/start/recover/observe/stop API). Every other crate still forbids it.
  • Linux: unchanged proof (inherited claim lock, /proc scan, pidfd).
  • macOS: same inherited lock, checked through libproc; descriptor and start time are rechecked just before kill. Only a PID reused within those microseconds could be signalled.
  • Windows: each worker runs in a private named Job object (Local\ name with location and claim, owner and DACL limited to the server account), joined while suspended; recovery opens it by name and never adopts a job another account owns.
  • The CLI's local launcher and recovery use the crate on all three platforms.
  • Verify job "Scheduler recovery (macOS, Windows)", change-selected and counted in the aggregate.

Verification

Work in progress. The crate's recovery cases pass on Linux, and mutating out the marker or descriptor check fails them. Clippy is clean for the crate on Linux, Windows and macOS targets; make check-ci and the public-repository scan pass. The first runner results are pending. A temporary diagnostics step for the Windows lease failures is included and will be removed before review. No DuckShop case: this is process supervision, not a project shape.

Adds a change-selected Verify job that runs the control store, scheduler
and worker supervision tests on macOS and Windows runners against SQLite
and a native PostgreSQL server, and counts its result in the aggregate.
Work in progress. Adds sqlbuild-process-ownership, the only crate allowed
unsafe code, with a safe API to prepare, start, recover, observe and stop
an owned worker. Linux keeps its inherited lock, /proc and pidfd proof.
macOS checks the lock through libproc and rechecks the start time just
before signalling. Windows runs each worker in a private named Job object
owned by, and open only to, the server account. The CLI's local launcher
and recovery now use it on all three platforms.

Validated so far: the crate's recovery cases pass on Linux, and clippy is
clean for the crate on Linux, Windows and macOS targets. The Verify job
also carries a temporary diagnostics step for the Windows lease failures,
to be removed before review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant