ci: gate mongodb test healthcheck on a writable primary - #17816
Open
DanRibbens wants to merge 1 commit into
Open
ci: gate mongodb test healthcheck on a writable primary#17816DanRibbens wants to merge 1 commit into
DanRibbens wants to merge 1 commit into
Conversation
The e2e/int MongoDB container's healthcheck ran `db.adminCommand('ping')`,
which succeeds as soon as mongod accepts connections — including during the
replica-set election gap that follows the entrypoint's auth restart. Because
CI brings the service up with `docker compose ... up -d --wait`, the container
was reported ready before a writable PRIMARY existed, so early transactions
were rejected with:
MongoServerError: Only servers in a sharded cluster can start a new
transaction at the active transaction number
That surfaced as intermittent, all-retries e2e failures (most visibly on the
slower tanstack-start variant) in suites that write on load — e.g. versions
autosave and localization — where the aborted transaction cascades into a
failed RSC render and "element not found" assertions.
Gate the healthcheck on `db.hello().isWritablePrimary || quit(1)` so the
container stays unhealthy (and `--wait` keeps blocking) until an election
completes. Verified locally against a fresh first-run: `up -d --wait` now
returns only once `rs.status().myState === 1`, and a start+commit transaction
succeeds immediately after.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
📦 esbuild Bundle Analysis for payloadThis analysis was generated by esbuild-bundle-analyzer. 🤖
Largest pathsThese visualization shows top 20 largest paths in the bundle.Meta file: packages/next/meta_index.json, Out file: esbuild/index.js
Meta file: packages/payload/meta_index.json, Out file: esbuild/index.js
Meta file: packages/payload/meta_shared.json, Out file: esbuild/exports/shared.js
Meta file: packages/richtext-lexical/meta_client.json, Out file: esbuild/exports/client_optimized/index.js
Meta file: packages/ui/meta_client.json, Out file: esbuild/exports/client_optimized/index.js
Meta file: packages/ui/meta_shared.json, Out file: esbuild/exports/shared_optimized/index.js
DetailsNext to the size is how much the size has increased or decreased compared with the base branch of this PR.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Intermittent e2e failures across suites (most visibly on the slower
tanstack-startvariant) with this root error:It cascades: the transaction aborts → the write (e.g. autosave) fails → the Server Components render throws → the page never renders → follow-on assertions fail with "Test ended" / "element not found". Seen recently on run 31960129016 (
versions [tanstack-start]), earlier onlocalization [tanstack-start], and confirmed pre-existing onmain(the exact error string appears 6–18× per run across several recent main tanstack-start jobs).Root cause
The
mongodbtest service (test/docker-compose.yml) is a single-member replica set (rs0) initialized by a custom entrypoint whose final step restartsmongodwith auth. Its healthcheck randb.adminCommand('ping'), which succeeds the instant mongod accepts connections — including during the replica-set election gap right after that auth restart, before a writable PRIMARY exists.CI starts the service with
docker compose … --profile mongodb up -d --wait(.github/actions/start-services/action.yml).--waitonly blocks until the container ishealthy, so it returned during the election gap and tests began issuing transactions against a node that couldn't start them.Fix
Gate the healthcheck on a writable primary instead of a bare ping:
--eval 'db.hello().isWritablePrimary || quit(1)'quit(1)keeps the container unhealthy (and--waitblocking) until an election completes. One-line change plus an explanatory comment; no application or test-code changes.Verification (local, Docker)
true, exit 0 (healthy); non-primary →quit(1), exit 1 (unhealthy).down -vthenup -d --wait):--waitreturned Healthy in ~12s, and at that momentrs.status().myState === 1(PRIMARY) anddb.hello().isWritablePrimary === true.--waitreturned — the exact operation that was being rejected in CI.Notes
versions,localization) and checking that theOnly servers in a sharded cluster…error count drops to 0.directConnection=true&replicaSet=rs0connection string and the internalmongodb:27017RS member host are left as-is; they weren't the trigger and changing them is riskier. Flagging for follow-up if the flake persists.🤖 Generated with Claude Code