Skip to content

Commit f74e3d0

Browse files
authored
Merge pull request #315 from metaobjectsdev/feat/migrate-chain-replayability
feat: cross-port metadata sources (four CLIs, one config) + migration-chain replayability (#313)
2 parents 9f22175 + c9dda9b commit f74e3d0

74 files changed

Lines changed: 9902 additions & 158 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

CHANGELOG.md

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,147 @@ this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
77

88
## [Unreleased]
99

10+
### Added — `sources` is read by all four CLI surfaces, plus `meta init --config-only`
11+
12+
`.metaobjects/config.json`'s `sources` key stops being a Node-only concern. Adopter
13+
guide: [`docs/features/metadata-sources.md`](docs/features/metadata-sources.md).
14+
15+
- **`sources` is read by all four CLI surfaces**, not just the Node `meta` CLI —
16+
the C#, Python and Java/Kotlin CLIs (Kotlin has no CLI of its own; it runs
17+
through the same Maven plugin as Java) now resolve metadata from the
18+
port-neutral `.metaobjects/config.json`, so one declaration serves every port
19+
(C#'s CLI loader accepts only a single directory `path` source — see the
20+
adopter guide). Each reads a **neutral subset** (`schema_version` + `sources`) and ignores
21+
unknown top-level keys, so the TypeScript-owned keys in that file (`migrate`,
22+
`scope`, `extract`, and the rest) never become a four-port change. Precedence
23+
is a ladder — explicit CLI argument, then the port's own native surface (a
24+
pom's `<sourceDir>`/`<sources>`, Python's `metadata` key), then `sources`,
25+
then the default `metaobjects/` directory — and a config that exists but is
26+
malformed errors at its own rung rather than silently falling through. Gated
27+
by the new
28+
[`fixtures/source-resolution-conformance/`](fixtures/source-resolution-conformance/)
29+
corpus, which every port runs.
30+
- **`meta init --config-only`** writes `.metaobjects/config.json` and nothing
31+
else, so a Maven- or pip-rooted project can declare its sources for the Node
32+
CLI (which owns `migrate` and `verify --db`, ADR-0015) without acquiring a
33+
TypeScript scaffold it will not use.
34+
- **`scope` / `migrate.scope` stay Node-CLI-only.** Java's shipped `<filters>`
35+
grammar uses `*` to cross the `::` separator and `@` to match one segment —
36+
respectively `scope`'s `**` and `*`, inverted — plus `!`-prefix exclusion and
37+
a `.[attr]` predicate `scope` cannot express at all
38+
(`GeneratorUtil.createRegexFromGlob` carries a `TODO` conceding its own
39+
separator handling is wrong). Both are output filters over the same resolved
40+
file set, so reconciling them is a separate, adopter-affecting decision
41+
rather than a mechanical port. No cross-port behavior depends on `scope`.
42+
- **Resolved file order, and the malformed-config error code, are deliberately
43+
NOT cross-port contracts.** The ports' directory walks already differ and
44+
always have (Java sorts by basename, C# by full-path ordinal, Python by
45+
basename, TypeScript walks depth-first); the corpus compares file **sets**.
46+
A malformed config must raise rather than silently degrade to "no config",
47+
but which error is each port's own — verified empirically: TypeScript raises
48+
a raw `ZodError` with no code at all, Python raises
49+
`ERR_COLLECTION_NOT_FOUND`, C# and Java both raise `ERR_BAD_ATTR_VALUE`.
50+
- **Directory expansion follows symlinked directories in all four ports**
51+
including when a declared `sources` path is itself a symlink, or a symlink
52+
sits partway through a walked tree. TypeScript and C# already did; Java and
53+
Python now match (a symlinked `sources` path previously resolved to zero
54+
files in Java, silently, exit 0). A symlink cycle is a loud error rather
55+
than a hang. Gated by two new `symlinks`-bearing corpus cases.
56+
- **Behavior change (Java/Maven only): a `<loader>` naming neither
57+
`<sourceDir>` nor `<sources>`, with no `.metaobjects/config.json` `sources`
58+
and no default `metaobjects/` directory, now FAILS the build**
59+
(`ERR_COLLECTION_NOT_FOUND`) instead of silently producing an empty model
60+
and passing. This is the one behavior change here that can break an
61+
existing `mvn metaobjects:generate`/`:verify` — most likely to bite a
62+
multi-module reactor where a parent pom configures `<loader>` and one child
63+
module never adds its own `<sourceDir>`. To restore the old outcome, declare
64+
`<sourceDir>`/`<sources>` explicitly in that module's pom, or give it a real
65+
metadata source (a `metaobjects/` directory or a `.metaobjects/config.json`
66+
`sources` entry).
67+
68+
### Changed — a committed migration chain must replay from empty, and `meta migrate` stops writing chains that cannot ([#313](https://github.com/metaobjectsdev/metaobjects/issues/313))
69+
70+
**`meta migrate --from-db` now REFUSES a drop for a table or view the committed schema
71+
snapshot never contained**, exiting 2 and naming each object. This is the one change here
72+
that can fail an existing project's `meta migrate`, so it leads. Pass
73+
**`--allow drop-unmanaged`** when the drop is genuinely intended.
74+
75+
The refusal exists because the drop it blocks produces a migration nobody can replay. The
76+
live migrate path diffs metadata against introspection and never reads the snapshot, so a
77+
table another tool owns reads as "in the database, not in the model" and is proposed for a
78+
`DROP TABLE`. Every incremental migrate then keeps succeeding against the database that
79+
already has that table — the chain only fails the day someone provisions a fresh one, which
80+
for the reporter was **three months later**, by which point the only working database left
81+
was a leftover CI container. `drift/classify.ts` has always said objects present in the DB
82+
but not the snapshot "must never be treated as actionable drift or auto-dropped"; this is
83+
the first place that doctrine is enforced where it mattered.
84+
85+
It does not false-fire on brownfield projects, and the reason is structural rather than
86+
special-cased: **both mechanisms ADD to the snapshot.** A `baseline --from-db` snapshot
87+
contains the foreign table; a project declaring `migrate.scope` carries its out-of-scope
88+
entries forward. The guard fires precisely when nothing ever claimed the object. It fails
89+
OPEN with no snapshot on disk — refusing there would break the first `meta migrate` of every
90+
greenfield project — and it lives on the live path only, because the offline path diffs
91+
against the snapshot and so cannot propose a snapshot-absent drop at all.
92+
93+
**Emitted forward drops now carry `IF EXISTS`**`drop-table`, `drop-view` (plain and
94+
CASCADE), `drop-index` (both the plain form and #285's constraint-backed
95+
`ALTER TABLE … DROP CONSTRAINT`), `drop-fk` and `drop-check` — in both dialects, so an
96+
already-absent object cannot break a replay. **Down statements stay bare, deliberately:**
97+
`rollbackTo` runs `down.sql` and the ledger delete in ONE transaction, so a guarded down
98+
would no-op and still record the rollback as done. Rollback is the one place a loud failure
99+
is load-bearing. Also left bare on purpose: the sqlite recreate-and-copy rebuild's
100+
`DROP TABLE` and d1-cascade's, each of which drops a table the same recipe just
101+
`INSERT…SELECT`ed from, where `IF EXISTS` converts a caught corruption into a silent one.
102+
`drop-column` is excluded as the one genuine dialect limit — sqlite has no
103+
`DROP COLUMN IF EXISTS` — and the new refusal covers it instead. D1 inherits the sqlite
104+
change, since `emit/d1.ts` renders through `renderSqlite`.
105+
106+
**A chain creating a table or view in a non-default schema now emits
107+
`CREATE SCHEMA IF NOT EXISTS`** ahead of it. `CREATE SCHEMA` was emitted nowhere in either
108+
emitter — only by the ledger's own setup — so an `@schema` project's chain could never apply
109+
to a virgin database. Views count, not only tables: a first migration creating just a view in
110+
a non-default schema failed identically. The down does not drop the schema; it may hold
111+
objects this tool does not own and cannot restore.
112+
113+
### Added — `meta verify --replay` and `--replay-snapshot`
114+
115+
Two new verify subverbs that answer the question the toolchain was already promising an
116+
answer to. `docs/features/migrations-and-drift.md` and `meta migrate --help` both said
117+
`apply-pending` "is the way to provision a fresh or CI database"; that is true only of a
118+
chain that builds the schema, and nothing checked.
119+
120+
- **`--replay`** replays the committed chain into an empty throwaway database and asserts it
121+
**applies**. This is the #313 gate.
122+
- **`--replay-snapshot`** additionally asserts the replayed schema **equals the committed
123+
snapshot**, finally wiring `verifyReplay` — built, exported, and without a CLI caller since
124+
the 2026-05-31 design retained it as "the optional `verify --replay` integrity aid". It
125+
catches a different defect: hand-edited structural DDL that still applies but no longer
126+
builds the recorded schema.
127+
128+
They are two tiers rather than one gate because the populations differ. A project adopted via
129+
`migrate baseline --from-db` passes the first trivially and **cannot** pass the second by
130+
construction — its snapshot is the whole introspected database against an empty chain. The
131+
reporter's failure was an *apply* error, so the weaker assertion is the one that answers the
132+
bug and is immune to that class. The limitation is documented rather than auto-detected: the
133+
only candidate signal has no production caller and would live in the *target* database's
134+
ledger, while the gate runs against a fresh engine with no ledger at all.
135+
136+
Neither needs a `--db`. The engine is local and disposable — real Postgres in-process via
137+
**PGlite**, a throwaway temp file for sqlite — so there is nothing to provision, no
138+
credentials, and no scratch database to collide with or drop by mistake. **`@electric-sql/pglite`
139+
is a new OPTIONAL peer dependency of `@metaobjectsdev/migrate-ts`** (~22 MB of WASM, so it is
140+
not forced on every adopter): install it to replay a postgres chain. With no URL to infer from,
141+
the dialect precedence is `--dialect` > `migrate.dialect` > refuse naming `--dialect`.
142+
`--migration-format flyway` and `--dialect d1` are refused, mirroring `apply-pending`. An empty
143+
chain and a missing snapshot both pass and **say which**, because a gate that is silent when it
144+
checked nothing cannot be told apart from one that passed.
145+
146+
`verifyReplay` also gains an optional `governed` so a project declaring `migrate.scope` can use
147+
the second tier at all: such a project carries the other owner's tables into its snapshot on
148+
purpose and its chain never creates them, so without this they were reported as missing on
149+
every replay.
150+
10151
### Added — pre-release publishing to a private registry (no more real releases just to test a change)
11152

12153
Trying an unreleased change against a downstream project required cutting a real release on

bun.lock

Lines changed: 9 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

docs/CONFORMANCE.md

Lines changed: 30 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Conformance coverage
22

3-
The MetaObjects standard ships **20 shared conformance corpora** under
3+
The MetaObjects standard ships **21 shared conformance corpora** under
44
[`fixtures/`](../fixtures/). Every port runs every corpus that is *applicable to
55
it* and asserts the same expected behaviour against the same fixtures. **This page
66
is the inverse index**: fixture → feature doc + per-port pass status, and it is the
@@ -42,6 +42,7 @@ regenerate with `ls -d fixtures/<corpus>/*/ | wc -l`.
4242
| [`fixtures/template-output-render-conformance/`](../fixtures/template-output-render-conformance/) | 5 ||||||
4343
| [`fixtures/generator-registry-conformance/`](../fixtures/generator-registry-conformance/) | 1 canonical manifest ||||||
4444
| [`fixtures/provider-composition-conformance/`](../fixtures/provider-composition-conformance/) | 9 (5 error-shape + 4 compose-load) ||| — (JVM registry via Java) |||
45+
| [`fixtures/source-resolution-conformance/`](../fixtures/source-resolution-conformance/) | 25 cases | ✓ (reference implementation) || inherits via Java |||
4546
| [`fixtures/scope-conformance/`](../fixtures/scope-conformance/) | 10 cases | ✓ (reference implementation) |||||
4647
| [`fixtures/agent-context-conformance/`](../fixtures/agent-context-conformance/) | 4 | ✓ (the emitter is TS-owned) |||||
4748
| [`fixtures/metamodel-docs/`](../fixtures/metamodel-docs/) | 1 | ✓ (docs emit is TS-owned) |||||
@@ -182,6 +183,31 @@ inheritance), `m2m/` (3), `jsonb/` (2, typed value-object columns) and
182183
Kotlin, C#, Python — run it in BOTH lanes: a hand-rolled reference server and
183184
the port's own GENERATED API artifact booted over HTTP.
184185

186+
### `fixtures/source-resolution-conformance/` (25 cases)
187+
188+
All 25 cases → [features/metadata-sources.md](features/metadata-sources.md) (how a
189+
declared `sources` set resolves to a file list). Companion to
190+
`scope-conformance/` below — `sources` decides which files are read, `scope`
191+
filters what is emitted from them. The corpus is file-shaped: one committed
192+
`cases.json`, read directly by every port's runner, with no per-port fixture
193+
and no ledger.
194+
195+
It pins the resolved file **SET** for a declared `sources` list — the default
196+
directory, replacement-not-merge, the relative-path base (the directory
197+
HOLDING `.metaobjects/`, never the process cwd), recursive directory walking,
198+
case-insensitive extension matching, union-with-de-duplication, and every
199+
error condition (an unresolvable path, an unsupported `resource`/`package`
200+
kind, and a malformed config — `"expectError": true` pins only that
201+
resolution RAISES, since which error code it raises with is deliberately NOT
202+
a cross-port contract; see the corpus README). **All four CLI surfaces run
203+
it** — TypeScript (the reference implementation), C#, Python, and Java (Kotlin
204+
inherits it, since Kotlin has no CLI entry point of its own and runs through
205+
the same Maven plugin as Java):
206+
`server/typescript/packages/sdk/test/source-resolution-conformance.test.ts`,
207+
`server/csharp/MetaObjects.Conformance.Tests/SourceResolutionConformanceTests.cs`,
208+
`server/python/tests/conformance/test_source_resolution_conformance.py`, and
209+
`server/java/metadata/src/test/java/com/metaobjects/config/SourceResolutionConformanceTest.java`.
210+
185211
### `fixtures/scope-conformance/` (10 cases)
186212

187213
All 10 cases → [features/metadata-sources.md](features/metadata-sources.md) (the
@@ -207,9 +233,9 @@ grammar rather than four.
207233

208234
## Orphaned fixtures (tested but not yet documented)
209235

210-
The fixtures in the seven corpora mapped above (metamodel 255 + yaml 15 + verify 31
211-
+ render 15 + persistence 33 + api-contract 41 + scope 10) each map to a feature doc. None
212-
are orphaned today. The remaining corpora in the totals table gate tooling
236+
The fixtures in the eight corpora mapped above (metamodel 255 + yaml 15 + verify 31
237+
+ render 15 + persistence 33 + api-contract 41 + source-resolution 25 + scope 10) each
238+
map to a feature doc. None are orphaned today. The remaining corpora in the totals table gate tooling
213239
contracts (registry manifests, provider composition, agent context, docs emit)
214240
rather than user-facing metamodel behaviour, so they have no feature-doc row.
215241

0 commit comments

Comments
 (0)