0.1d — Julia round-trip: validate the emitted result.toml + run-dir against the shared schemas (closes #18) - #31
Conversation
jack-champagne
left a comment
There was a problem hiding this comment.
The schema craft here is solid and the parts that are present check out — Julia reads the same packages/schema/schemas/*.json the TS side does (real "no second copy" guard), and the schema-roundtrip CI job genuinely runs Julia and reds the build (not a skipped/false-green gate). I ran the suite locally: 32 / 47 / 63 / 22 Julia, all green.
Two changes before this closes #18 — both are "validate against the right thing," not "build more."
1. The round-trip validates fixtures, not the producer — which is the exact gap this slice exists to close. It validates committed test/fixtures/* + the committed demo/run/ dir. Nothing here runs a solve to emit a fresh result.toml and validates that. AC5's "or freshly-emitted ones" is unmet, and the "slow/nightly" lane the description points to doesn't exist (ci.yml is the only workflow). This is the β shape: an integration test that doesn't exercise the integration — it goes green while the real emitter is free to drift. Gate #18 on one real lane: a single canonical solve → run-dir → validate-with-validate.jl in CI. One prompt is enough. Until that exists, #18 stays open and the round-trip is validating its own fixtures.
(Architectural note: the orchestrator, not Julia, writes manifest.toml — so a Julia emit→validate can only cover result.toml. The title's "emitted … manifest.toml" overclaims; scope the producer round-trip to result.toml.)
2. AC6 "mutating the shared schema flips BOTH validators" is asserted, not exercised — and it isn't quite true. I tested it: added a required key to result.schema.json, then validated the clean fixture.
- Julia (
runtests.jl) flipped — 3 tests failed. It reads the schema at runtime. ✓ - The shipped TS bundle did not flip (still
OK). TS bakes schemas intodist/at build time (import … with { type: "json" }); it only flipped after a rebuild.
runtests.jl's AC6 testset checks file-path identity + the Julia flip; it never exercises the TS side. So in any "edit schema, don't rebuild" window — including a shipped VSIX whose schema is frozen at package time — Julia and TS can disagree and this gate won't catch it. Either add a step that rebuilds TS then asserts both flip on the same perturbed schema, or document the build-time-vs-runtime asymmetry in the contract.
On scope — don't overbuild ahead of this. The fix above is one real lane, not more fixtures or more schema surface. Same for the stack: hold the env-contract (requires/provides) and any further validation machinery until the producer seam is actually validated. Validating the right thing once beats validating the wrong thing thoroughly.
Stack coordination: this hardcodes manifest.toml (validate.jl, fixtures, demo) — it must follow #28's manifest.toml → run.toml rename in lockstep or the round-trip validates a dead filename. And per #15's note, land #28 to main and rebase these off main rather than running a 4-deep chain.
Net: the validator is good; it's just pointed at fixtures instead of the producer. Repoint it at one real emitted run and this becomes the integration gate β never had.
…all five, wrong-type fixture, rename lockstep Address Jack's #30 review (rebased on #28's run.toml rename): - bin now points at a COMMITTED launcher (packages/schema/launcher/amico-validate, mirrors amico-run) instead of the un-built dist/amico-validate.js — so the `amico-validate` bin links on a clean install (was "Failed to create bin … ENOENT"; dist is built by `pnpm build`, absent at install time). CI now exercises the LINKED bin (`pnpm --filter amicode-v2 exec amico-validate --help`) so a broken entry reds. - CI gate validates ALL FIVE: added solvespec + catalog-entry via --schema (AC5 / DoD). - Self-contained wrong-type fixture (invalid/result-wrongtype.toml) + a cli.test case — the class matrix no longer relies on an in-test mutation (AC7). - Rename lockstep: index.ts kindForFilename + cli.ts usage + ci.yml gate + cli.test KINDS + invalid/run.toml all follow #28's manifest.toml→run.toml / "manifest"→"run". schema 34 / amico-run 47 / extension 64 (+1 skip) green; linked bin + gate verified locally. (The bigger real-emit→validate lane Jack flagged lands in #31, the round-trip slice.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
f8daf3f to
c8f9863
Compare
…inst the shared schemas (closes #18) Closes the producer/consumer loop: a Julia validator that checks result.toml / manifest.toml against the SAME packages/schema/schemas/*.json the TS side uses — read DIRECTLY (no transcribed copy), so mutating a schema flips both languages. This is the cross-language anti-drift guarantee (#18 AC6). - packages/schema/julia/validate.jl: AmicoValidate module + CLI. Reads ../schemas via JSONSchema.jl (draft-07, the aju<->JSONSchema.jl common floor), parses TOML, validates, prints field-precise errors. A formatting shim turns JSONSchema.jl's coarse SingleIssue (path + keyword) into "/path/key: reason" comparable to the TS side — incl. reconstructing the missing key for `required` and the version-specific schema_version message. Exit 0 valid / 64 invalid. jsonify() coerces unquoted TOML datetimes (Dates → ISO string) so they validate like quoted (S2, Julia side). - packages/schema/julia/runtests.jl (19 tests): valid golden corpus conforms; invalid corpus field-precise (offending key/path); schema_version absent + unrecognized; unquoted-datetime; single-source asserts (SCHEMA_DIR == ../schemas, no .schema.json copy in the Julia tree) — the AC6 anti-drift proof. - ci.yml: `schema-roundtrip` job (setup-julia + instantiate + runtests.jl). FAST tier — JSONSchema only, NO Piccolo solve (per the design review split). The freshly-emitted round-trip over a real solve is the slow/nightly extension; solve_common.jl + solve_template.jl already stamp schema_version so emitted artifacts are round-trippable there. Committed Manifest.toml for reproducibility (precedent: extension/julia). All four slices green: TS schema 32 / amico-run 47 / extension 63 + Julia 19. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ise parity, demo-golden round-trip Address the adversarial review of 0.1d: - MUST-FIX: re-resolved packages/schema/julia/Manifest.toml so `Test` (a direct dep added after the manifest was generated) is present — the committed manifest lacked it, so `Pkg.instantiate()` hard-failed the new schema-roundtrip CI job before any test ran. Verified a fresh instantiate from the committed manifest now exits 0. - Field-precise PARITY with the TS validator (the shim now reads JSONSchema.jl's iss.val): additionalProperties NAMES the offending key (`unknown key "rogue_key"`), enum lists allowed values (`must be one of (completed, failed, aborted)`), and type/maximum/minimum/exclusiveMinimum/minLength render as prose (`must be number`, `must be <= 1.0001`). where-path normalized to `/a/b` | `(root)`. - Round-trip now also validates the REAL bundled demo run dir (manifest/result/ FINISHED) — emitted golden, not just hand-authored fixtures (AC5). Julia round-trip 22 tests green; TS schema 32 / amico-run 47 / extension 63. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nd-to-end (rebased on #28 rename) Address Jack's #31 review: - REAL producer lane (not just fixtures): scripts/producer_roundtrip.sh runs amico-run on a minimal non-Piccolo emitter (julia/emit_fixture.jl) so it produces a live run-dir (run.toml + FINISHED from the orchestrator, result.toml from the Julia emitter), then validates the FRESHLY-EMITTED artifacts with validate.jl. CI runs it (fast tier; the real Piccolo solve stays the slow/nightly extension). Scoped to what each side actually emits — the orchestrator writes run.toml/FINISHED, the Julia producer writes result.toml. - AC6 exercised, not asserted: scripts/ac6_drift_check.sh perturbs the SHARED schema, REBUILDS TS (it bakes schemas at build time — the gap Jack found), then asserts BOTH the TS bin AND validate.jl reject a previously-valid fixture, then reverts. Proven locally; wired into the schema-roundtrip CI job. runtests.jl's AC6 testset notes the build-vs- runtime asymmetry. - Rename lockstep: validate.jl + runtests.jl manifest.toml→run.toml, "manifest"→"run", run.schema.json; resolved the ci.yml conflict keeping #30's all-five gate + this job. schema 34 / amico-run 47 / extension 64 (+1 skip) / julia 22 green; producer + AC6 scripts pass locally. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
5dcdc82 to
2ef1d88
Compare
|
Pushed addressing both must-changes: 1. Real producer lane (not fixtures). 2. AC6 exercised, not asserted. Plus the |
jack-champagne
left a comment
There was a problem hiding this comment.
Re-reviewed 2ef1d88c. Both must-changes resolved — the validator now points at the producer, not its own fixtures:
- Real producer lane —
producer_roundtrip.shrunsamico-runon a minimal emitter → a live run-dir →validate.jlon the freshly-emittedresult.toml/run.toml/FINISHED.set -euo pipefail+ validate exit codes make it red on drift; wired as a gating step inschema-roundtrip. Title scoped toresult.tomlper the note. - AC6 exercised, not asserted —
ac6_drift_check.shperturbs the sharedrun.schema.json, rebuilds TS (the build-time-bake gap I flagged), asserts BOTH the TS bin andvalidate.jlflip on a previously-valid fixture, then reverts viatrap cleanup EXIT. Closes the runtime-vs-build-time asymmetry.
Approving. Retargeting to main + merging behind #30.
Stacked on #30 (0.1c). Do not merge. Completes the Phase 0′ SchemaPackage set.
Closes the producer/consumer loop: a Julia validator that checks
result.toml/manifest.tomlagainst the samepackages/schema/schemas/*.jsonthe TS side uses — read directly, no transcribed copy — so mutating a schema flips both languages. That's the cross-language anti-drift guarantee (#18 AC6), verified empirically in review (mutate a shared schema → both TS and Julia flip the same fixture).What's here
packages/schema/julia/validate.jl:AmicoValidatemodule + CLI over JSONSchema.jl (draft-07, the ajv↔JSONSchema.jl common floor). Reads../schemasdirectly; parses TOML; field-precise errors via a shim that reaches TS parity — names the missing/extra key (missing required key "run_id",unknown key "rogue_key"), lists enum values, renders type/range prose (must be number,must be <= 1.0001), and the version-specificschema_versionmessage.jsonify()coerces unquoted TOML datetimes (S2). Exit 0/64.packages/schema/julia/runtests.jl(22): valid golden + invalid corpus (field-precise) + schema_version policy + unquoted datetime + the real bundled demo run dir + the single-source anti-drift asserts (SCHEMA_DIR == ../schemas, no schema copy in the Julia tree).ci.ymlschema-roundtripjob:setup-julia+ instantiate +runtests.jl. Fast tier — JSONSchema only, no Piccolo solve (per the design-review tier split). CommittedManifest.tomlfor reproducibility.Tier split (S1)
Fast = golden + negative round-trip (this job). Freshly-emitted-via-solve = slow/nightly extension;
solve_template.jl+solve_common.jlalready stampschema_version, so emitted artifacts are round-trippable there.Review
Adversarial review empirically verified the AC6 keystone (schema mutation flips both validators) and caught a must-fix (committed Manifest lacked
Test→ would red the new CI job) — fixed (re-resolved; fresh instantiate exits 0). Field-precise parity (#2/#3) + demo-golden (#4) folded in.All four slices green: TS schema 32 / amico-run 47 / extension 63 + Julia 22.
🤖 Generated with Claude Code