Skip to content

Move .realm.json properties into database#4623

Open
backspace wants to merge 33 commits intomainfrom
routing-db-cs-10053
Open

Move .realm.json properties into database#4623
backspace wants to merge 33 commits intomainfrom
routing-db-cs-10053

Conversation

@backspace
Copy link
Copy Markdown
Contributor

@backspace backspace commented May 1, 2026

This moves the publishable and showAsCatalog properties left behind in the “sidecar” .realm.json into the database:

CleanShot 2026-05-01 at 16 05 49@2x

backspace and others added 14 commits April 29, 2026 18:16
Adds a generic `sameRealm: boolean` option to `linksTo()` that, when
set, makes `LinksTo.validate` reject a linked card whose realm does
not match the containing card's realm. The check uses the existing
`getRealmURLString` helper (now exported from field-support) so it
works whether the parent is a FieldDef (via [realmContext]) or a
CardDef (via meta.realmURL); it skips silently when either realm is
unknown so synthetic in-memory construction is unaffected.

Apply the option to `RoutingRuleField.instance` so RealmConfig
hostRoutingRules can only reference cards within the same realm —
the constraint called out in the Host Mode Routing MVP spec.

Tests for the validation path land in a follow-up; CI will exercise
the existing instance-of failure case to confirm no regression in
the surrounding LinksTo.validate logic.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a unit test in card-basics-test that exercises the new
sameRealm option on linksTo across three cases:

- same-realm assignment is accepted
- cross-realm assignment throws the expected validation error
- assignment is allowed when the parent has no realm meta (the
  silently-skip-if-unknown safety valve for synthetic in-memory
  construction)

Realm meta is injected on the synthetic test cards by setting the
exported `[meta]` symbol directly, mirroring the pattern used in
serialization-test.gts. Endpoint-level coverage of the same path
will land with the planned defense-in-depth /_config 400 commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror the linksTo plumbing on LinksToMany so any future card that
needs a same-realm constraint on a multi-link gets it from the same
RelationshipOptions flag. Each element in the array runs through the
same realm check; nulls and not-loaded references are skipped (as
they already are for the existing instance-of and FileDef-id checks).

No current consumer — RoutingRuleField uses linksTo, not linksToMany —
but this keeps the option behaviour symmetric across the two
relationship field types.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Renders the rule as `<path> → <instance.cardTitle>`, using
@fields.instance @Format='atom' to delegate the linked card's
display. Edit / embedded / fitted templates are intentionally
deferred to the realm-config UI work in CS-10056.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surface sameRealm on the Field interface and pass it through to the
chooseCard call as preselectConsumingRealm so the card picker opens
with the consuming realm's filter on by default. The user can still
deselect the filter and pick a card from another realm — schema-level
validate.same-realm rejects the assignment in that case, so this is
purely a UX nudge that steers users toward valid candidates.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reverts the sameRealm option on linksTo / linksToMany. CS-10052 was
treating "routing rules must reference same-realm cards" as a
schema-level constraint, but the rule is a UX/product expectation,
not a security boundary — Boxel supports cross-realm card loads, and
the failure modes for a cross-realm rule (403s for viewers, weird
published-realm bundles) are confusing rather than dangerous.

The right home for the constraint is a custom edit template on
RealmConfig (or RoutingRuleField) that hard-scopes the picker to the
consuming realm. That work belongs with CS-10056, the realm-config
edit UI ticket.

Reverts:
- sameRealm? on RelationshipOptions / FieldConstructor / Field interface
- LinksTo / LinksToMany sameRealm field, constructor wiring, validate hook
- RoutingRuleField.instance: sameRealm: true
- LinksToEditor preselectConsumingRealm wiring
- card-basics-test sameRealm test case
- getRealmURLString export (back to private in field-support.ts)

The atom template on RoutingRuleField stays — that's an independent
rendering improvement.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CS-10053 schema-only commit. Creates a new public.realm_metadata
table keyed by realm URL with nullable show_as_catalog and
publishable boolean columns plus the standard created_at/updated_at
timestamps. No readers or writers in this commit — those land in
follow-ups. The existing readonly_role grants from the grafana
setup migration (1751981407344) auto-extend to the new table via
ALTER DEFAULT PRIVILEGES.

Picked a separate table rather than columns on realm_registry
because these are mutable per-realm settings, not identity, and
realm_registry has kind-conditional CHECK constraints we don't want
to muddy. Decoupling also lets a missing/late registry row not
block metadata reads or writes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a private getRealmMetadata() method on Realm (sibling of
getLastPublishedAt) that reads showAsCatalog and publishable from
the realm_metadata table introduced in the previous commit. Returns
null/null on missing rows or query failure, matching the pre-CS-10053
"absent in sidecar" semantics.

parseRealmInfo now fetches metadata in parallel with the sidecar read
and lastPublishedAt query, then overlays any non-null DB values on
top of the sidecar parse. While the table is empty this is a no-op —
sidecar values still drive the result for every realm. The next
commit will populate the table via boot-time backfill.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds runRealmMetadataBackfill (and an advisory-lock-wrapped variant)
that walks each realm directory at boot, copies showAsCatalog and
publishable from the legacy .realm.json sidecar into the new
realm_metadata table, and trims the migrated keys out of the
sidecar in the same pass.

Wired into main.ts after the existing registry backfill. Uses a
distinct advisory lock id (7331012) so the two backfills don't
serialize on each other in multi-instance deployments.

Idempotent: ON CONFLICT DO NOTHING on the INSERT, plus a no-op trim
on a sidecar that no longer has the migrated keys. Once a realm has
a row, the DB is authoritative and the sidecar copy never wins —
matching the read overlay shipped in the previous commit.

Tests cover source / bootstrap / published realms, the "no
migratable keys" no-op, the rerun-preserves-existing-row case, and
graceful skip on malformed or non-object sidecar JSON.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Flips the runtime over to the new DB-backed metadata store:

parseRealmInfo no longer reads showAsCatalog or publishable from the
.realm.json sidecar — DB values are authoritative. The sidecar parse
loop drops those two keys; the metadata overlay assigns them
unconditionally (a missing row reads as null, matching the prior
"absent in sidecar" semantics).

patchRealmConfig adds a third bucket — REALM_CONFIG_METADATA_PROPERTIES,
currently { publishable } — and routes those keys through a new
upsertRealmMetadata method that UPSERTs into realm_metadata. The
existing read-everything-before-writing-anything preflight is
preserved; the metadata write happens after the card and sidecar
writes succeed. showAsCatalog stays in PROTECTED_REALM_CONFIG_PROPERTIES.

RealmServer.createRealm stops writing the .realm.json sidecar and
INSERTs a realm_metadata row with publishable=true at realm
creation time. handle-publish-realm.ts stops mutating the sidecar's
publishable key and UPSERTs the published realm's metadata row to
publishable=false after the directory swap succeeds. The hostHome
rewrite (still sidecar-owned until CS-10055) is preserved, with a
guard so a missing sidecar doesn't crash the read.

Tests updated:
- realm-endpoints-test "allows any property except showAsCatalog"
  asserts the response body's publishable rather than the sidecar
  contents (which it never touches now).
- The malformed-sidecar PATCH test switches its example field from
  publishable to interactHome (publishable no longer goes to the
  sidecar, so it stopped exercising the sidecar parse path).
- realm-lifecycle-test asserts no sidecar is written by createRealm
  and that realm_metadata has the seeded publishable=true row.
- publish-unpublish-realm-test reads publishable from realm_metadata
  instead of the published sidecar across three test cases.

Existing realms continue to work via the boot-time backfill from the
previous commit; their first boot copies sidecar values into DB and
trims the sidecar. The base realm fixture (packages/base/.realm.json)
intentionally keeps its showAsCatalog: false value in git so the
backfill has something to migrate on first boot in any environment.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
createRealm no longer writes a .realm.json sidecar, so a fresh realm
returns 404 for that path. The "doesn't upload .realm.json" test was
unconditionally fetching the remote file to check it didn't contain
the local marker; gate the fetch on the file actually existing
remotely. The intent (CLI doesn't push the protected file) is still
validated either way.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 1, 2026

Preview deployments

Host Test Results

    1 files      1 suites   1h 59m 50s ⏱️
2 563 tests 2 548 ✅ 15 💤 0 ❌
2 582 runs  2 567 ✅ 15 💤 0 ❌

Results for commit a8be24f.

Realm Server Test Results

    1 files  +    1      1 suites  +1   18m 8s ⏱️ + 18m 8s
1 234 tests +1 234  1 234 ✅ +1 234  0 💤 ±0  0 ❌ ±0 
1 306 runs  +1 306  1 306 ✅ +1 306  0 💤 ±0  0 ❌ ±0 

Results for commit a8be24f. ± Comparison against earlier commit 1b4fbdc.

backspace and others added 7 commits May 1, 2026 14:22
The "host submode is available" test relied on publishable: true in
.realm.json, which the runtime no longer reads after CS-10053. Drop
the publishable key from the shared realm contents and instead
INSERT a realm_metadata row with publishable=true in the publishable
module's beforeEach, ordered before setupAcceptanceTestRealm so the
realm's first parseRealmInfo call captures the value before
#cachedRealmInfo locks it in. The "not publishable" module needs no
row — absent rows read as null, treated as not-publishable by the
host UI.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The runtime no longer reads publishable from .realm.json after
CS-10053. The tests in this file navigate directly to host-mode
URLs and don't gate on publishable, so the line was a no-op already
— remove it so future readers don't think setting it has an effect.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
# Conflicts:
#	packages/runtime-common/realm.ts
@habdelra
Copy link
Copy Markdown
Contributor

habdelra commented May 4, 2026

I think this work effects this PR. this was critical for dealing with the flaky matrix tests #4630

backspace added 4 commits May 4, 2026 11:45
is it possible to warn when you’re making it without having
run migrations? when do they run, even? confusing
@backspace backspace marked this pull request as ready for review May 4, 2026 19:21
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1234bf9b55

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/realm-server/server.ts Outdated
Comment thread packages/runtime-common/realm.ts
Two coupled fixes so a deleted realm doesn't bleed metadata into a
recreated realm at the same URL:

1. removeRealmDatabaseArtifacts (the realm-delete cleanup helper)
   now DELETEs from realm_metadata alongside the other realm tables.
   The row shouldn't outlive the realm.

2. createRealm's UPSERT now resets show_as_catalog to NULL in the
   ON CONFLICT branch (and explicitly inserts NULL on first insert),
   not just publishable. Defensive: if any other path left a row at
   this URL, createRealm starts the new realm clean.

Without these, a "delete and recreate at the same URL" sequence
inherits the prior show_as_catalog value — stale "false" surfaces
the new realm as hidden in the catalog immediately after creation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
backspace and others added 7 commits May 4, 2026 15:11
The /_config schema accepts arbitrary attribute values
(z.record(z.unknown())). Card-bound fields are validated downstream
by the card's deserialize layer; sidecar-bound fields are loosely
typed JSON. The DB-bound publishable column is the odd one out —
without an upfront type check, a non-boolean PATCH value reached
the SQL boolean column and surfaced as an opaque 500.

Add a 400 with a clean message when publishable is anything other
than boolean or null. Placed alongside the existing protected-
property check so the rejection happens before reads or writes,
preserving the all-or-nothing semantic for mixed PATCHes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
autofix didn’t work? what’s happening
# Conflicts:
#	packages/host/config/schema/1777660517426_schema.sql
#	packages/host/config/schema/1778020800000_schema.sql
This reverts commit 1b4fbdc.
@backspace backspace requested a review from a team May 5, 2026 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants