Skip to content

Prevent worker from certifying a stale runtime during extension drop/recreate - #336

Draft
pinodeca with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-stale-runtime-certification
Draft

Prevent worker from certifying a stale runtime during extension drop/recreate#336
pinodeca with Copilot wants to merge 4 commits into
mainfrom
copilot/fix-stale-runtime-certification

Conversation

Copilot AI commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

The background worker could initialize a duroxide runtime against one extension epoch, then publish readiness and write its epoch sentinel after a concurrent DROP EXTENSION / CREATE EXTENSION had replaced that epoch. Because the sentinel landed in the new df schema, the stale runtime—bound to provider objects that no longer existed—was certified as current and never restarted, causing readiness timeouts.

Root cause

run_duroxide_runtime() wrote the epoch sentinel after initialization and readiness publication, so the sentinel identified whichever epoch existed at the end of init rather than the epoch the runtime was initialized against. A drop/recreate mid-init routed a stale-runtime sentinel into the new epoch's schema, defeating run_until_extension_dropped_or_shutdown's replacement detection.

Changes

  • Epoch identity: capture_extension_epoch() reads pg_extension.oid, which is fresh for every CREATE EXTENSION and therefore distinguishes epochs even when the extension appears continuously present between polls.
  • Capture before init, revalidate after: the epoch is captured before initialize_duroxide_runtime, then rechecked (a) after init before publishing readiness, and (b) again after the readiness/sentinel writes before entering the processing loop. A mismatch (or a failed pre-init capture) tears down the just-created runtime and retries against the current epoch. The second check closes the narrow window where a drop/recreate would otherwise write the sentinel into the new epoch's schema.
  • teardown_runtime helper: extracted from the existing shutdown tail (behavior unchanged) and reused for stale-runtime teardown.
  • Test hook: PG_DURABLE_TEST_PAUSE_BEFORE_READY_MS inserts a pause between init and readiness publication—no-op in production—to make the race deterministic.
  • Regression test: scripts/test-epoch-race.sh starts the worker with the pause enabled, drops/recreates the extension mid-init, and asserts the stale runtime is torn down, the worker becomes ready, and a durable function completes against the current epoch. Wired into CI.

The revalidation runs only at initialization time, so shutdown latency is unaffected.

Observed log with the fix:

pg_durable: TEST hook — pausing 4000ms before readiness publication
pg_durable: extension epoch changed during initialization (before=Some(16384), after=Some(16728)) — tearing down stale runtime and retrying
pg_durable: initializing duroxide runtime...
pg_durable: duroxide runtime started
pg_durable: epoch sentinel written (...)
pg_durable: processing durable functions...

Copilot AI and others added 3 commits August 5, 2026 17:04
…recreate

Co-authored-by: pinodeca <32303022+pinodeca@users.noreply.github.com>
Co-authored-by: pinodeca <32303022+pinodeca@users.noreply.github.com>
Co-authored-by: pinodeca <32303022+pinodeca@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix worker certification of stale runtime during extension lifecycle Prevent worker from certifying a stale runtime during extension drop/recreate Aug 5, 2026
Copilot AI requested a review from pinodeca August 5, 2026 17:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Worker can certify a stale runtime during extension drop/recreate

2 participants