Skip to content

migrations-and-drift.md

Latest commit

 

History

History
406 lines (327 loc) · 25.4 KB

migrations-and-drift.md

File metadata and controls

406 lines (327 loc) · 25.4 KB
Raw

Migrations and drift detection

MetaObjects treats metadata as the source of truth and generated code + database schema as derived. Migration is the build-time pipeline that emits SQL DDL from metadata diffs; drift detection is the cross-cutting discipline that catches divergence between metadata, code, DB, and prompts.

There are 7 drift sources, and the toolchain has a guard for each.

The 7 drift sources

Drift source Caught by When
Code-vs-DB Codegen — the generated SQL DDL is emitted from the same metadata as the entity / table code. Build time
Code-vs-API-doc Cross-port codegen from the same metadata. Build time
DB-vs-metadata meta verify --db (TS CLI) — introspects the live DB and fails if it has drifted from metadata. Includes modeled projection view bodies (a changed CREATE VIEW is replace-view drift); a hand-authored unmodeled view is unmanaged and never flagged. A schema concern owned by the Node toolchain regardless of server language; on the JVM ports the runtime auto-create/validator path was removed (ADR-0015) and the metaobjects:verify Maven goal is not available. Cloudflare D1 has no client wire protocol, so it can't go through --db's Kysely-driver introspection — use meta verify --dialect d1 [--d1 <binding>] [--remote] instead (the same wrangler-shelled-out path meta migrate --dialect d1 uses); --remote is required to check the deployed database, not the local wrangler dev shadow copy. Pointing --db file: at wrangler's local D1 state directory (.wrangler/state/**/d1/**) still runs, but only verifies that local copy — verify warns when it detects this. CI on every PR
Migration-vs-metadata The Node meta migrate emits migrations FROM metadata diffs — they cannot drift from metadata by construction. Schema migrations for every port are owned by this Node toolchain (@metaobjectsdev/cli migrate, ADR-0015); the C# and Python migrate surfaces were removed. Build time
Generated-edited @generated headers in emitted code + three-way merge that preserves hand-edits inside non-generated regions. Code review
Prompt-vs-payload FR-004 Renderer.verify parses {{...}} references in templates and checks each one exists on the payload VO. Build time + runtime
Generated-vs-runtime Kotlin / Java MetadataStartupValidator (from Spring ApplicationReadyEvent) re-loads metadata at startup and asserts the generated table objects match. App startup

Authoring (no metadata change required)

Migration is invisible to the metadata author — the same Author declaration drives a CREATE TABLE on the first run and an ALTER TABLE on later runs.

{
  "object.entity": {
    "name": "Author",
    "children": [
      { "source.rdb": { "@table": "authors" } },
      { "field.long":   { "name": "id" } },
      { "field.string": { "name": "name", "@required": true, "@maxLength": 200 } },
      { "field.string": { "name": "bio",  "@maxLength": 2000 } },
      { "identity.primary": { "@fields": "id", "@generation": "increment" } }
    ]
  }
}

Add a new email field tomorrow — the next meta migrate emits the ALTER TABLE for the new column.

Migration commands per port

TypeScript

meta migrate                  # diff metadata vs DB → emit migration SQL
meta migrate --dry-run        # preview without writing
meta migrate --dialect d1     # Cloudflare D1 dialect (TS-only)

Dialects: postgres (default), sqlite, d1. Output lands under the path configured in metaobjects.config.ts (typically ./migrations/<timestamp>__<slug>.sql).

meta migrate apply-pending replays the committed migration files against --db in order, ledger-tracked (_metaobjects_migrations) and transactional — with no diff and no metadata load. For a project whose chain builds the schema, it is the way to provision a fresh or CI database from the committed migrations; run meta verify --replay to know you are one of those. A project adopted with migrate baseline --from-db is not: its chain starts after the schema already existed, so there is nothing for apply-pending to build. meta migrate --apply, by contrast, is diff-first — it authors a new migration from the metadata-vs-DB diff before applying it. apply-pending just runs the pending already-committed files, making it idempotent; --dry-run lists what would run. postgres/sqlite only — on D1 use wrangler d1 migrations apply.

meta verify --replay — the chain applies from empty

meta verify --replay              # the committed chain applies to an empty database
meta verify --replay-snapshot     # ...and reproduces the committed schema snapshot

A committed chain can be broken without anything saying so. meta migrate writes a DROP TABLE "x" whenever x is in the live database and absent from the metadata — including when no migration in the chain ever created x, which is what happens when another tool owns that table. Every incremental migrate keeps succeeding against the database that already has x; the chain only fails the day somebody provisions a fresh one (#313).

--replay replays the whole committed chain into an empty throwaway database and asserts it applies. Nothing is compared. This is the gate for the promise apply-pending makes.

--replay-snapshot additionally asserts the replayed schema equals the committed snapshot. That catches a different thing: hand-edited structural DDL — a committed up.sql someone changed so the chain still applies but no longer produces the schema the snapshot records.

Neither needs a database. The engine is local and disposable — real Postgres in-process via PGlite, a throwaway temp file for sqlite — so there is nothing to provision, no credentials, and no scratch database to collide with or accidentally drop. PGlite is an optional peer dependency of @metaobjectsdev/migrate-ts: install it (npm i -D @electric-sql/pglite) to replay a postgres chain. Without it the gate exits 2 and says so.

With no --db there is no URL to infer a dialect from, so: --dialect wins, else migrate.dialect from .metaobjects/config.json, else the run refuses naming --dialect. --migration-format flyway and --dialect d1 are refused — replay those with flyway migrate and wrangler d1 migrations apply against a scratch database.

Exit codes follow verify's convention: a chain that fails to apply, or a snapshot mismatch, is drift → 1; an engine that cannot start is operational → 2. An empty chain, or a missing snapshot, passes → 0, and says which, because a gate that is silent when it checked nothing cannot be told apart from one that passed.

--replay-snapshot does not apply to a project adopted with migrate baseline --from-db. Such a project's snapshot is the whole introspected database while its chain is empty, so the comparison cannot pass by construction. Use --replay there. This is documented rather than auto-detected: the only signal that could distinguish the two lives in the target database's ledger, and this gate runs against a fresh engine that has no ledger at all.

If the gate goes red, applied migrations are checksum-immutable — hand-editing a committed up.sql is rejected on any database that already applied it. The supported fix is a compensating migration: author a new one that creates the missing object, or that supersedes the bad drop.

--allow drop-unmanaged — the same problem, caught earlier

meta migrate --from-db refuses to author a drop for a table or view the committed snapshot never contained, because that is precisely the statement that cannot replay:

migrate: refusing to drop public.other_owned_table — absent from the committed schema
snapshot, so this toolchain never managed it and the migration could not replay against
a database where it never existed. Re-run with '--allow drop-unmanaged' if the drop is
intended.

Pass --allow drop-unmanaged when the drop is genuinely what you want. The refusal does not fire for brownfield projects: baseline --from-db puts the foreign table in the snapshot, and a project declaring migrate.scope carries its out-of-scope entries forward, so both read as managed. It fires precisely when nothing ever claimed the object. A project with no snapshot yet — every greenfield first run — is unaffected.

Emitted forward drops carry IF EXISTS

Every forward DROP TABLE / DROP VIEW / DROP INDEX / DROP CONSTRAINT a migration emits is now IF EXISTS, in both dialects, so an already-absent object cannot break a replay. Down statements stay bare deliberately: rollbackTo runs down.sql and the ledger delete in one transaction, so a silently-no-op down would still record the rollback as done — rollback is the one place a loud failure is load-bearing.

A chain that creates a table or view in a non-default schema now also emits CREATE SCHEMA IF NOT EXISTS "<schema>"; ahead of it. Without that, an @schema project's chain could never apply to a virgin database, because nothing ever created the schema. The down does not drop the schema: it may hold objects this tool does not own and cannot restore.

D1: rebuilding a foreign-key-referenced table

Cloudflare D1 applies each migration inside its own implicit transaction, and SQLite ignores PRAGMA foreign_keys while a transaction is open. The SQLite table-rebuild recipe (used for a CHECK, column type/nullability/default, foreign-key, or field.enum values change) relies on PRAGMA foreign_keys = OFF taking effect before it drops and recreates the table — which does not happen on remote D1.

meta migrate --dialect d1 handles this by auto-generating a cascade (#241, closing #226's residual under-refuse gap below) instead of refusing outright. When a change would rebuild a table that another table's foreign key references, the emitter rebuilds that table together with every table that transitively references it, in one pass: the affected tables are dropped referrers-first and recreated parents-first, under PRAGMA defer_foreign_keys = ON so every foreign-key check defers to the end of D1's implicit transaction instead of firing mid-rebuild. The result applies cleanly against a populated production database and re-converges — a follow-up meta verify/meta migrate sees no drift. The cascade is built over the union of the actual (live) and expected (target) schemas' foreign-key graphs, so it also covers the case a target-schema-only check would miss: a single migration that both rebuilds a referenced table and drops the referencing foreign key in the same run. A projection/view that reads a rebuilt table is dropped before the rebuild and recreated after — for the CHECK/FK/enum-values rebuild class as well as column changes — so a dependent view is never stranded mid-migration.

The one case still hand-written: a multi-table foreign-key cycle (table A references B references … references A, two or more tables). A cycle has no parents-first rebuild order, so meta migrate --dialect d1 still refuses at generation time — hand-write the migration (drop the foreign key on one side of the cycle, rebuild the tables, then restore it) or break the cycle in your metadata. A self-referencing table (a table whose own foreign key targets itself) is not a cycle in this sense and is handled by the cascade like any other rebuild.

A moved primary key (adoption-time refusal)

The diff/emit has no add-primary-key / drop-primary-key change kind, so an existing table whose live PRIMARY KEY differs from the metadata identity cannot be expressed as a migration. When adopting such a database (--from-db), meta migrate now refuses at generation time instead of emitting un-appliable SQL — detect-and-refuse, the same arc as #226→#241 for the D1 foreign-key rebuilds above. It throws a PrimaryKeyChangeError (naming the table and both PKs), the CLI catches it and exits 1 (#258).

Previously the move degraded silently into an add-column + drop-column: the old PK column and its constraint were dropped while the new column was never made primary key, leaving the table with no primary key, so every foreign key referencing it failed at apply (there is no unique constraint matching given keys). This surfaces only when adopting an existing database whose PK disagrees with the metadata — a greenfield create-table carries its primary key inline.

The check is engine-wide (postgres / sqlite / d1 — the diff is shared) and runs after rename detection, mapping live PK column names through any detected rename-column change, so a primary-key column that was merely renamed (the engine preserves the PK through RENAME COLUMN) is not mistaken for a move. The read-only meta verify / drift path does not set the refusal flag, so verify keeps reporting primary-key drift rather than throwing. Auto-migrating the move (adding the add-primary-key / drop-primary-key change kinds) is a documented future follow-up.

A legacy serial primary key (adoption-time refusal)

When migrating against a live Postgres database whose primary key is a legacy serial / bigserial column — one carrying a live nextval(...) default — and the metadata declares that identity.primary without @generation, the diff would otherwise emit ALTER COLUMN … DROP DEFAULT. That is destructive: every insert that omits the id starts failing. The missing @generation is genuinely ambiguous — it reads identically whether the author simply never declared it (and wants to keep auto-increment) or deliberately dropped it (to move the column onto app-assigned ids) — so meta migrate refuses rather than guessing, the same detect-and-refuse arc as the primary-key move above. Declare @generation: increment on the identity to keep the sequence, or pass --allow drop-identity-default if removing auto-increment is intentional. An identity that does declare @generation: increment never reaches this gate (its default diff is skipped), so only the undeclared case fires.

Java

Schema migrations for Java projects are owned by the TypeScript toolchain (@metaobjectsdev/cli migrate). The Java Maven plugin's meta:migrate goal was removed, and per ADR-0015 the OMDB runtime auto-create path was removed too — OMDB is pure data-access (CRUD/query/codec/transactions). Provision the schema by applying the TS-produced DDL/migrations to the database the Java service connects to.

Use the TS CLI against the same database the Java service connects to:

meta migrate --db postgresql://... --slug initial   # emit migration SQL
meta migrate --db postgresql://... --apply          # apply pending migrations

Kotlin

Kotlin schema migrations follow the same story as Java: the Java Maven plugin's meta:migrate mojo was removed and there is no Kotlin-specific migrate command. Schema migrations are owned by the TS toolchain:

meta migrate --db postgresql://... --slug initial
meta migrate --db postgresql://... --apply

C#

Schema migrations for C# projects are owned by the TypeScript toolchain (@metaobjectsdev/cli migrate). Per ADR-0015 the C# migrate engine and the --from-db introspection surface were removed — the C# CLI (dotnet meta) is gen / verify only. Use the TS CLI against the same database the C# service connects to:

meta migrate --db postgresql://... --slug initial   # emit migration SQL
meta migrate --db postgresql://... --apply          # apply pending migrations

Python

Schema migrations for Python projects are owned by the TypeScript toolchain (@metaobjectsdev/cli migrate). Per ADR-0015 the Python migrate module was removed — the metaobjects console script is gen / verify only (the runtime is pure data-access via ObjectManager). Use the TS CLI against the same database the Python service connects to:

meta migrate --db postgresql://... --slug initial   # emit migration SQL
meta migrate --db postgresql://... --apply          # apply pending migrations

Drift verify commands per port

Port Command What it does
TypeScript meta verify --db Introspects the live DB; reports DB-vs-metadata drift.
Java mvn metaobjects:verify -Dmeta.verify.mode=codegen|templates (metaobjects:verify Maven goal) + Renderer.verify (build-time) The codegen/template-drift metaobjects:verify Maven goal is alive and is how Java gates drift in CI (codegen mode regens + fails on drift vs committed output; templates mode drift-checks {{...}} references against the payload VO via Renderer.verify). Only the live-DB-schema metaobjects:verify goal was removed — that's TS-owned now (meta verify --db) — along with the runtime auto-create validator (ADR-0015).
Kotlin mvn metaobjects:verify -Dmeta.verify.mode=codegen|templates (metaobjects:verify Maven goal — same goal covers both Java + Kotlin) + MetadataStartupValidator (startup) Same as Java — the metaobjects:verify Maven goal remains, plus template-drift and startup validation.
C# meta verify ./metadata --templates ./prompts Drift-checks templates against their payload VOs (FR-004 prompt-drift).
Python python -m metaobjects.render.verify Same as C# verify — template-vs-payload drift.

Generated-vs-runtime: Kotlin startup validator

KotlinSpringConfigGenerator emits a @Configuration class that re-loads metadata at Spring ApplicationReadyEvent and asserts that the generated Table objects still match the metadata. If a developer hand-edited a generated table and the regen didn't catch it (or a CI race shipped a stale build), the app fails-fast at startup instead of silently serving wrong data.

// GENERATED — MetadataExposedConfig.kt
@Configuration
class MetadataExposedConfig(private val dataSource: DataSource) {
    init { Database.connect(dataSource) }

    @EventListener(ApplicationReadyEvent::class)
    fun validateMetadata() {
        val loader = loadResources("app", listOf("meta.entities.json"))
        MetadataStartupValidator.validate(loader)
    }
}

Migrating views (projections)

A projection's CREATE VIEW is generated from its origin.* children, so meta migrate owns the view. Three things are worth knowing.

Append projection fields; don't insert them. Postgres can update a view in place (CREATE OR REPLACE VIEW) only when the existing output columns are unchanged and any new ones are added at the end. A view's columns come out in projection declaration order, so:

  • adding a field at the end of a projection → non-destructive replace. Dependent views, grants, and the view's identity all survive.
  • inserting a field in the middle, reordering, renaming, or removing one → the view must be dropped and recreated, which is destructive to anything that depends on it and is therefore gated (--allow drop-view).

Body-only changes — a different join path, an origin.aggregate @filter, a changed aggregate that keeps the same result type — are always non-destructive.

A cascading drop is blocked, loudly. If dropping a view would destroy dependent objects — another application's view, a materialized view — meta migrate blocks and names every one of them. Proceeding requires --allow drop-view,drop-view-cascade, and the emitted migration carries a WARNING: CASCADE DROP banner listing what it destroys. MetaObjects does not manage those objects and cannot restore them. --allow drop-view alone never cascades.

One-time upgrade step (--allow adopt-view). Managed views carry a MetaObjects fingerprint in their COMMENT ON VIEW; that fingerprint — not the view's SQL text — is how migrate knows whether a view is current. (It cannot use the text: Postgres does not store view SQL, it stores a parse tree and regenerates the SQL in its own style, so what you wrote never comes back.) A view with no fingerprint is either hand-written or was created before fingerprinting existed, and those are indistinguishable — so migrate fails closed rather than overwrite somebody's hand-written SQL:

meta migrate --allow adopt-view      # once per environment, after upgrading

That stamps the existing views. Afterwards they converge silently. This is also what closes the loop on the doctrine in downstream-metadata-decisions.md: a hand-written view sitting where a projection expects one is now visible to meta verify --db as drift, instead of being silently invisible to it.

Genuinely-irreducible views and externally-owned objects (@sql / @unmanaged, #208). Some read models cannot be expressed as origin.* (a recursive CTE, a window function, a set operation). Rather than hand-write such a view outside the tool — where it degrades to accidentally-unmanaged — carry its body in a source.rdb @sql attribute. The tool registers, fingerprints, and drift-checks that verbatim body (never parsing it) exactly like a synthesized view: it emits CREATE VIEW … AS <your body> with a fingerprint stamp, a second migrate is a no-op, and a pre-existing hand-written view at that name is adopted via the same --allow adopt-view step above. An @sql view may carry an extends-bound identity.primary / fields (for row identity and read-model shape) without the tool mis-synthesizing a body. For a DB object whose DDL is owned entirely elsewhere (Flyway, a hand-migration) — a view or a table — mark its source @unmanaged: true: meta migrate never creates, drops, or drift-checks it, and meta verify --db reports it as external (declared). The two are mutually exclusive. See ADR-0043.

Verified by

The following conformance fixtures gate this feature's behavior across ports:

Schema migration (fixtures/persistence-conformance/migrations/)

  • bootstrap-canonical-from-empty.yaml — full-CREATE bootstrap from an empty database
  • add-nullable-column.yaml — incremental ALTER TABLE … ADD COLUMN for a new nullable field
  • drop-table-blocked-without-allow.yaml — destructive operations require an explicit allow-flag
  • noop-converged-canonical.yaml — the idempotence gate: apply the whole canonical schema, then diff the same metadata against the database it just produced. A converged schema must emit zero SQL. This is what catches any asymmetry between what emit writes, what introspect reads back, and what the expected schema models — the class of bug that makes meta migrate re-propose (and on SQLite, destructively rebuild) unchanged tables forever.

Template drift (fixtures/verify-conformance/) — the Renderer.verify engine asserts that every variable, section, partial, and required-tag in a template resolves against its declared payload. 31 fixtures, grouped:

  • Variables: verify-var-known-clean, verify-var-unknown, verify-var-implicit-iterator-clean, verify-var-unescaped-and-triple-unknown
  • Dotted paths: verify-dotted-path-clean, verify-dotted-path-head-nonfield, verify-dotted-path-tail-nonfield
  • Sections: verify-section-over-nonfield, verify-section-pushes-element-clean, verify-section-element-nonfield, verify-nested-sections-clean, verify-nested-section-nonfield, verify-parent-context-in-section-clean, verify-scalar-section-conditional-clean, verify-scalar-section-conditional-nonfield
  • Inverted sections: verify-inverted-section-clean, verify-inverted-section-over-nonfield
  • Partials: verify-partial-no-provider, verify-partial-unresolved, verify-partial-resolved-clean, verify-partial-resolved-drift, verify-partial-in-section-context
  • Required slots / tags: verify-required-slot-used-clean, verify-required-slot-unused, verify-required-slot-via-section, verify-required-tags-present, verify-required-tag-in-partial, verify-required-tag-self-closing, verify-required-tag-missing-open, verify-required-tag-missing-close, verify-required-tag-prefix-no-overmatch

Browse the full set under fixtures/verify-conformance/.

Cross-port runner coverage: TS / Java / Kotlin / C# / Python all execute these via their respective conformance runners. See docs/CONFORMANCE.md for the per-port pass/skip ledger.

See also