Draft
Prevent worker from certifying a stale runtime during extension drop/recreate#336
Conversation
…recreate Co-authored-by: pinodeca <32303022+pinodeca@users.noreply.github.com>
Co-authored-by: pinodeca <32303022+pinodeca@users.noreply.github.com>
Co-authored-by: pinodeca <32303022+pinodeca@users.noreply.github.com>
Copilot
AI
changed the title
[WIP] Fix worker certification of stale runtime during extension lifecycle
Prevent worker from certifying a stale runtime during extension drop/recreate
Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The background worker could initialize a duroxide runtime against one extension epoch, then publish readiness and write its epoch sentinel after a concurrent
DROP EXTENSION/CREATE EXTENSIONhad replaced that epoch. Because the sentinel landed in the newdfschema, the stale runtime—bound to provider objects that no longer existed—was certified as current and never restarted, causing readiness timeouts.Root cause
run_duroxide_runtime()wrote the epoch sentinel after initialization and readiness publication, so the sentinel identified whichever epoch existed at the end of init rather than the epoch the runtime was initialized against. A drop/recreate mid-init routed a stale-runtime sentinel into the new epoch's schema, defeatingrun_until_extension_dropped_or_shutdown's replacement detection.Changes
capture_extension_epoch()readspg_extension.oid, which is fresh for everyCREATE EXTENSIONand therefore distinguishes epochs even when the extension appears continuously present between polls.initialize_duroxide_runtime, then rechecked (a) after init before publishing readiness, and (b) again after the readiness/sentinel writes before entering the processing loop. A mismatch (or a failed pre-init capture) tears down the just-created runtime and retries against the current epoch. The second check closes the narrow window where a drop/recreate would otherwise write the sentinel into the new epoch's schema.teardown_runtimehelper: extracted from the existing shutdown tail (behavior unchanged) and reused for stale-runtime teardown.PG_DURABLE_TEST_PAUSE_BEFORE_READY_MSinserts a pause between init and readiness publication—no-op in production—to make the race deterministic.scripts/test-epoch-race.shstarts the worker with the pause enabled, drops/recreates the extension mid-init, and asserts the stale runtime is torn down, the worker becomes ready, and a durable function completes against the current epoch. Wired into CI.The revalidation runs only at initialization time, so shutdown latency is unaffected.
Observed log with the fix: