fix(bootstrap): first-boot FK race + single-flight

[mmcintosh]

fix(bootstrap): document-type seeding must precede its FK consumers on a fresh DB

Fixes #1017

TL;DR

On a fresh database's first boot, bootstrapMiddleware seeds system data with a
flat Promise.all that races a foreign-key dependency: two steps create
document_types rows while two other steps insert documents that FK-reference
those rows. When a consumer insert wins the race, D1 throws FOREIGN KEY constraint failed; the per-step .catch() swallows it, so RBAC roles/verbs and every core
plugin are silently left unseeded. A second facet compounds it: there is no
single-flight guard, so two concurrent requests on a cold isolate both enter bootstrap
and seed in parallel — re-opening the same FK window across the two runs. The bootstrap
then writes an unconditional KV skip-marker (24h TTL), which latches the broken
state: every subsequent request/isolate takes the KV fast-path, skips D1 entirely,
and the site runs with no roles and no plugins until the marker expires or is cleared.

This PR (1) makes the two document-type producers run and finish before the consumer
inserts, (2) adds a single-flight in-flight promise so concurrent cold-isolate requests
await one bootstrap instead of racing parallel ones, and (3) makes the "bootstrap
complete" latch + KV marker conditional on success. It's a surgical change to
middleware/bootstrap.ts plus a real-DB regression test. Two files, +166/−49.

---

Sequencing — this is PR 3 of a coordinated set of 4

These four PRs were split from overlapping work so each is single-concern. Land PR-1
first; after that the remaining three touch disjoint files and may land in any order.

1. PR-1 — fix(core): beta.25 test-infra (per-DB caches + executionCtx guard).
   Prerequisite — merge first. Fixes ~25 unit failures present on current main.
2. PR-2 — feat: views plugin port. Re-grounded onto PR-1.
3. PR-3 — this PR — fix(bootstrap): first-boot FK race + single-flight. Touches
   only middleware/bootstrap.ts (+ its test); already independent of the PR-1 files.
4. PR-4 — feat: forms document-model port. Re-grounds onto PR-1 + PR-3.

Why this PR shows failing unit tests until PR-1 lands: this branch is cut from
current main, so npm test reports the same ~25 pre-existing failures as pristine
main — in 7 files, none of which import bootstrap.ts. They are fixed by PR-1.
This PR's own regression test and tsc are green; the FK-ordering + single-flight
behavior is proven by the tests and reality boots below.

Relationship to PR-4 (forms): not code-coupled — a form document type is a
producer in the seed order, not an FK-race victim, so forms works on an already-booted
DB regardless. But a greenfield deploy wants this PR in first, or the very first boot
still leaves RBAC roles / plugins unseeded (admin 403s).

---

How we hit it (real, not theoretical)

We were spinning up a clean local greenfield stack (fresh D1, empty KV) on
beta.25 to validate an incoming feature branch. The very first request logged:

[PluginBootstrap] Installing plugin: Authentication System
✘ [ERROR] [PluginBootstrap] Error ensuring plugin Authentication System:
    Error: D1_ERROR: FOREIGN KEY constraint failed: SQLITE_CONSTRAINT_FOREIGNKEY
  [cause]: Error: FOREIGN KEY constraint failed: SQLITE_CONSTRAINT_FOREIGNKEY
    at PluginService.installPlugin (services/plugin-service.ts:103)
    at PluginBootstrapService.ensurePluginInstalled (services/plugin-bootstrap.ts:131)
    at PluginBootstrapService.bootstrapCorePlugins (services/plugin-bootstrap.ts:80)
    at async Promise.all (index 4)
✘ [ERROR] [Bootstrap] Error seeding RBAC documents:
    Error: D1_ERROR: FOREIGN KEY constraint failed: SQLITE_CONSTRAINT_FOREIGNKEY

Two things stood out:

1. Every failing step is inside the await Promise.all([...]) in
   bootstrap.ts (at async Promise.all (index 4)), and the failures are exactly
   the steps that write documents rows (RBAC seed at index 3, plugin bootstrap at
   index 4). The steps that create document_types are indices 0–1 of the same
   array.
2. After that first boot, the errors never appeared again — but the site was broken:
   admin routes returned 403 Insufficient permissions (no RBAC roles), and core
   plugins were inactive. Restarting the worker didn't help. Only clearing the KV
   bootstrap marker (rm -rf .wrangler/state/v3/kv locally) and re-booting fixed it —
   because by the second boot the document_types rows happened to already exist, so
   the race didn't reproduce.

That second-boot-heals-it behavior is the tell: the bug is a write-ordering race on
a cold DB, and the KV marker turns a transient race into a sticky outage.

Root cause

documents.type_id is NOT NULL REFERENCES document_types(id)
(migrations/0002_documents.sql:30). The seed phase in bootstrap.ts runs five
steps in one Promise.all:

Step	Function	Role
0	bootstrapDocumentTypes	producer — inserts document_types rows
1	autoRegisterCollectionDocumentTypes	producer — inserts document_types rows
2	repairMissingCredentialAccounts	independent (auth account table)
3	rbacService.ensureSystemRbacSeed	consumer — INSERT INTO documents (rbac_role/verb/grant), FK → document_types
4	PluginBootstrapService.bootstrapCorePlugins	consumer — installPlugin → INSERT INTO documents, FK → document_types

Because all five are launched concurrently, on a cold DB a consumer's documents
insert can execute before the producer has committed the matching document_types
row → FOREIGN KEY constraint failed. D1 enforces this FK; local SQLite in the test
harness disables it (which is why unit tests never caught it — see Testing).

Second facet: no single-flight guard

bootstrapComplete (module-level) only flips to true at the end of a successful
run. On a cold isolate, two requests that arrive close together both read
bootstrapComplete === false, both miss the KV marker (not written yet), and both run
the full seed concurrently. Even with the ordering fix applied per-run, two
concurrent runs can interleave (run A creating types while run B inserts documents) and
duplicate every seed write. There is no promise that a second caller can await.

Third defect: the marker latches failures

After the batch:

bootstrapComplete = true;                        // in-memory latch — unconditional
await cacheKv.put(BOOTSTRAP_KV_KEY(), '1', { expirationTtl: 86400 })  // KV marker — unconditional

Both are written even when steps in the batch failed. The KV marker is the cold-start
fast-path (if (kvDone === '1') { … return next() } near the top of the middleware),
so once it's set, no isolate re-runs the D1 seed for 24h — the partial bootstrap
is cached as if it were complete.

Who this affects

Any environment whose first-ever request lands on a fresh, unseeded database:

• New Cloudflare deploys — the first request after a fresh D1 is provisioned.
• Preview/branch deploys with their own D1.
• Local dev on a clean .wrangler state.
• CI that boots against a freshly-migrated DB.

Warm/existing databases are unaffected (the document_types already exist, so the
race can't lose). That's precisely why it's easy to miss in day-to-day dev against an
already-bootstrapped DB, and why it bites new installs specifically.

The fix

packages/core/src/middleware/bootstrap.ts — the seed phase becomes two phases:

• Phase A (await, sequential): the two producers —
  bootstrapDocumentTypes then autoRegisterCollectionDocumentTypes — so all
  document_types rows are committed before any documents insert references them.
  These are two fast, idempotent registrations; awaiting them adds negligible latency.
• Phase B (parallel, as before): the two consumers (RBAC seed, plugin bootstrap)
  plus the independent credential-account repair, kept in a Promise.all so
  cold-start latency is preserved now that the FK precondition holds.

Single-flight guard. A module-level bootstrapInFlight: Promise<void> | null holds
the running bootstrap. A request that arrives mid-bootstrap awaits it and proceeds
rather than starting a second seed. The check-and-set is synchronous (no await
between the if (bootstrapInFlight) test and the assignment), so exactly one request
wins the latch; it is released in a finally. This closes the concurrency facet and
also removes duplicate cold-start work.

A small runStep(label, fn) wrapper records whether any step threw. The
bootstrapComplete latch and the KV skip-marker are now written only when every
step succeeded. If something still fails transiently (a D1 hiccup, say), nothing is
cached, and the next request retries the idempotent bootstrap and converges — instead
of locking a broken state for 24h.

Net behavior on the happy path is unchanged: types get registered, consumers seed,
the marker is written. The only difference is ordering (so the FK holds) and that a
failed boot no longer poisons the cache.

Why not just widen the KV TTL / bump the version key

c99a5673 ("bump version to beta.25 — invalidates KV bootstrap key") already showed
the marker can cache a bad state — but a version bump only busts the key once per
release; the next fresh DB re-enters the same race and re-latches. The self-host path
633023e4 ("entrypoint-based seed prevents concurrent SQLite corruption") applies the
right idea — seed sequentially before serving — but only to the Docker/SQLite runtime.
This PR brings that ordering guarantee to the Workers/D1 cold-start path, which is
where new cloud installs boot.

Testing

packages/core/src/__tests__/middleware/bootstrap-fk-ordering.test.ts (new, real
SQLite):

• The shared d1-sqlite harness disables foreign keys (D1 doesn't reliably enforce
  them and the services delete derived rows explicitly). Here the FK is the
  subject, so the test re-enables foreign_keys = ON on the DB under test — matching
  what production D1 enforced when it threw.
• Reproduces the failure mode: a documents insert (via the real
  DocumentsService.create) for a type that isn't registered yet rejects with
  FOREIGN KEY constraint failed and writes nothing — the "roles missing after boot"
  symptom.
• Proves the fix: after bootstrapDocumentTypes runs (Phase A), the same consumer
  inserts land cleanly; a spread of real system types (rbac_role, rbac_verb,
  plugin) all persist under FK enforcement.

Reality-tested end-to-end on fresh greenfield boots (clean D1 + empty KV) against the
beta.25 runtime:

• Single first request: clean boot, zero FK errors, DB fully seeded (21
  document types, 4 RBAC roles, 6 verbs, 4 plugins) — no KV-clear-and-reboot needed.
• Four concurrent cold-isolate requests: the bootstrap body runs exactly once
  (Starting system initialization logged once), completes cleanly, zero FK errors
  — proving the single-flight guard.

Before the fix, the same first boot always FK-failed on Installing plugin: Authentication System + RBAC seed and required a KV-clear + second boot to recover.

Gates: tsc --noEmit 0 errors. The full unit suite shows 25 pre-existing failures on
this branch identical to pristine origin/main (verified: same 7 files, none of
which import bootstrap.ts); they are the separately-flagged beta.25 unit-suite
breakage (per-DB cache + executionCtx guard) and are fixed by that repair. Green CI
for this PR requires that repair to land first.

Scope / not in scope

• No change to the KV fast-path read, the infra-path skips, or plugin gating.
• The self-host entrypoint seed path (633023e4) is unchanged.

[mmcintosh]

Note on the red CI. The failing check here is the unit-test step (about 25 tests), and those failures are pre-existing on main — this branch is cut straight from current main, and none of them touch bootstrap.ts. They are the known beta.25 test-harness issue (a per-database cache leaks generated columns across in-memory test DBs, producing "no such column: q_*" errors) and are fixed by the test-infra PR (#1018, Fixes #1016). Type-check passes here, and this PR's purpose is to fix the E2E against preview failure: the loginAsAdmin timeout caused by the first-boot FK race that leaves the RBAC roles and core plugins unseeded (#1017). The two are a pair: #1018 fixes this PR's unit suite, and this PR fixes #1018's E2E preview. Recommend landing #1018 first, then this one — after both, CI is green across the stack.