Build searchable-driven search-doc generation + parity harness + baseline

[habdelra]

Builds the searchable-driven search-doc generator and the parity tooling, without making it authoritative — store-driven searchDoc stays in charge of production indexing until the cutover (CS-11724). Stacked on #5350 (CS-11721).

Generator — searchDocFromFields(instance) (packages/base/card-api.gts)

Parallel to searchDoc. Derives link depth from the explicit searchable annotation instead of from what the render loaded, and loads the named link targets itself (targeted loading) rather than relying on store residency.

• Routes are dotted paths rooted at the indexed card's link fields. Depth is governed entirely by the annotations on the card being indexed — a card pulled in as a link target does not re-consult its own searchable. true makes the immediate ("self") link searchable; a dotted path makes a deeper (n+1) link searchable; arrays combine.
• The owner's store is threaded through the recursion (a contained FieldDef value may not be store-associated, but a link reached through it must still load against the owner's store). Targeted loading degrades a missing/broken/unloadable target to { id }.
• Cycle clipping, { id } for unfollowed/broken/not-loaded links, linksToMany id normalization, and the query-backed-field skip are preserved from the store-driven path. Link targets enumerate their declared type, dropping unqueryable polymorphic-subtype bloat.

Parity

• Differential test (packages/host/tests/integration/searchable-search-doc-test.gts): the searchable-driven path follows a searchable link to the same target, with the same contained data, the store-driven render loaded. Whole-doc byte-equality is intentionally not asserted here — the new spec keeps { id } for every relationship while the store-driven path omits unused links via usedLinksToFieldsOnly; reconciling that to an identical doc is the cutover's gate, after the migration (CS-11723) reproduces today's depth.
• searchable-parity-diff.ts (packages/realm-server/scripts): the realm-scale post-migration validator — diffs a realm's live store-driven search docs (boxel_index) against the searchable-driven output per card, ignoring _cardType and (optionally) the intended shallow-link difference. Data-gathering documented inline.
• Baseline: ctse/military-pigeon render-vs-searchDoc split captured from staging (≈ 933:1) and recorded on the ticket for the follow-on (prerender off the hot path) to measure against.

Tests (host integration, run in CI)

searchable-search-doc-test.gts — 9 cases: the routes-come-only-from-the-indexed-card trio (target's own searchable dormant when pulled in; honored only when itself indexed; a dotted route on the indexer drives the deeper expansion), self link, n+1 route, contains-routing through a contained value, cycle clip → { id }, missing/broken link → { id }, unannotated link → { id }, declared-type enumeration drops subtype bloat, and the differential expansion-parity check.

Verified: glint + tsc clean (base/host/runtime-common/realm-server), eslint clean, and all 9 integration tests green on a live test-services:host stack.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1b054e4c39

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[github-actions]

Preview deployments

• Host staging preview

• Host production preview

Host Test Results

    1 files      1 suites   2h 32m 37s ⏱️
3 319 tests 3 304 ✅ 15 💤 0 ❌
3 338 runs  3 323 ✅ 15 💤 0 ❌

Results for commit 0f3ce37.

Realm Server Test Results

    1 files  ±0      1 suites  ±0   9m 48s ⏱️ +32s
1 666 tests ±0  1 666 ✅ ±0  0 💤 ±0  0 ❌ ±0 
1 745 runs  ±0  1 745 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 0f3ce37. ± Comparison against earlier commit 41f434e.

[Copilot]

Pull request overview

Adds a new searchable-driven search-doc generator (searchDocFromFields) alongside a parity harness (host integration tests + realm-scale diff script) to validate behavior before the eventual cutover from the existing store-driven searchDoc.

Changes:

• Introduces searchDocFromFields(instance) in packages/base/card-api.gts, generating search docs by following field.searchable routes and targeted-loading link targets.
• Adds a host integration test suite covering route semantics, cycle clipping, missing/broken links, declared-type link enumeration, and a differential parity check.
• Adds a realm-server script to diff live store-driven docs against generated docs, with an option to ignore known shallow-link differences.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
packages/base/card-api.gts	Adds searchDocFromFields and the searchable-driven recursion + targeted loading logic
packages/host/tests/integration/searchable-search-doc-test.gts	Adds integration coverage for searchable-driven search-doc generation and parity expectations
packages/host/tests/helpers/base-realm.ts	Exposes searchDoc and searchDocFromFields from the base realm test helper
packages/realm-server/scripts/searchable-parity-diff.ts	Adds a CLI script to diff live boxel_index.search_doc output against generated docs

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.