Skip to content

0.1d — Julia round-trip: validate the emitted result.toml + run-dir against the shared schemas (closes #18) - #31

Merged
jack-champagne merged 3 commits into
mainfrom
rchari/0.1d-julia-roundtrip
Jun 29, 2026
Merged

0.1d — Julia round-trip: validate the emitted result.toml + run-dir against the shared schemas (closes #18)#31
jack-champagne merged 3 commits into
mainfrom
rchari/0.1d-julia-roundtrip

Conversation

@Rchari1

@Rchari1 Rchari1 commented Jun 27, 2026

Copy link
Copy Markdown
Member

Stacked on #30 (0.1c). Do not merge. Completes the Phase 0′ SchemaPackage set.

Closes the producer/consumer loop: a Julia validator that checks result.toml/manifest.toml against the same packages/schema/schemas/*.json the TS side uses — read directly, no transcribed copy — so mutating a schema flips both languages. That's the cross-language anti-drift guarantee (#18 AC6), verified empirically in review (mutate a shared schema → both TS and Julia flip the same fixture).

What's here

  • packages/schema/julia/validate.jl: AmicoValidate module + CLI over JSONSchema.jl (draft-07, the ajv↔JSONSchema.jl common floor). Reads ../schemas directly; parses TOML; field-precise errors via a shim that reaches TS parity — names the missing/extra key (missing required key "run_id", unknown key "rogue_key"), lists enum values, renders type/range prose (must be number, must be <= 1.0001), and the version-specific schema_version message. jsonify() coerces unquoted TOML datetimes (S2). Exit 0/64.
  • packages/schema/julia/runtests.jl (22): valid golden + invalid corpus (field-precise) + schema_version policy + unquoted datetime + the real bundled demo run dir + the single-source anti-drift asserts (SCHEMA_DIR == ../schemas, no schema copy in the Julia tree).
  • ci.yml schema-roundtrip job: setup-julia + instantiate + runtests.jl. Fast tier — JSONSchema only, no Piccolo solve (per the design-review tier split). Committed Manifest.toml for reproducibility.

Tier split (S1)

Fast = golden + negative round-trip (this job). Freshly-emitted-via-solve = slow/nightly extension; solve_template.jl + solve_common.jl already stamp schema_version, so emitted artifacts are round-trippable there.

Review

Adversarial review empirically verified the AC6 keystone (schema mutation flips both validators) and caught a must-fix (committed Manifest lacked Test → would red the new CI job) — fixed (re-resolved; fresh instantiate exits 0). Field-precise parity (#2/#3) + demo-golden (#4) folded in.

All four slices green: TS schema 32 / amico-run 47 / extension 63 + Julia 22.

🤖 Generated with Claude Code

@jack-champagne jack-champagne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The schema craft here is solid and the parts that are present check out — Julia reads the same packages/schema/schemas/*.json the TS side does (real "no second copy" guard), and the schema-roundtrip CI job genuinely runs Julia and reds the build (not a skipped/false-green gate). I ran the suite locally: 32 / 47 / 63 / 22 Julia, all green.

Two changes before this closes #18 — both are "validate against the right thing," not "build more."

1. The round-trip validates fixtures, not the producer — which is the exact gap this slice exists to close. It validates committed test/fixtures/* + the committed demo/run/ dir. Nothing here runs a solve to emit a fresh result.toml and validates that. AC5's "or freshly-emitted ones" is unmet, and the "slow/nightly" lane the description points to doesn't exist (ci.yml is the only workflow). This is the β shape: an integration test that doesn't exercise the integration — it goes green while the real emitter is free to drift. Gate #18 on one real lane: a single canonical solve → run-dir → validate-with-validate.jl in CI. One prompt is enough. Until that exists, #18 stays open and the round-trip is validating its own fixtures.

(Architectural note: the orchestrator, not Julia, writes manifest.toml — so a Julia emit→validate can only cover result.toml. The title's "emitted … manifest.toml" overclaims; scope the producer round-trip to result.toml.)

2. AC6 "mutating the shared schema flips BOTH validators" is asserted, not exercised — and it isn't quite true. I tested it: added a required key to result.schema.json, then validated the clean fixture.

  • Julia (runtests.jl) flipped — 3 tests failed. It reads the schema at runtime. ✓
  • The shipped TS bundle did not flip (still OK). TS bakes schemas into dist/ at build time (import … with { type: "json" }); it only flipped after a rebuild.

runtests.jl's AC6 testset checks file-path identity + the Julia flip; it never exercises the TS side. So in any "edit schema, don't rebuild" window — including a shipped VSIX whose schema is frozen at package time — Julia and TS can disagree and this gate won't catch it. Either add a step that rebuilds TS then asserts both flip on the same perturbed schema, or document the build-time-vs-runtime asymmetry in the contract.

On scope — don't overbuild ahead of this. The fix above is one real lane, not more fixtures or more schema surface. Same for the stack: hold the env-contract (requires/provides) and any further validation machinery until the producer seam is actually validated. Validating the right thing once beats validating the wrong thing thoroughly.

Stack coordination: this hardcodes manifest.toml (validate.jl, fixtures, demo) — it must follow #28's manifest.toml → run.toml rename in lockstep or the round-trip validates a dead filename. And per #15's note, land #28 to main and rebase these off main rather than running a 4-deep chain.

Net: the validator is good; it's just pointed at fixtures instead of the producer. Repoint it at one real emitted run and this becomes the integration gate β never had.

Rchari1 added a commit that referenced this pull request Jun 28, 2026
…all five, wrong-type fixture, rename lockstep

Address Jack's #30 review (rebased on #28's run.toml rename):
- bin now points at a COMMITTED launcher (packages/schema/launcher/amico-validate,
  mirrors amico-run) instead of the un-built dist/amico-validate.js — so the
  `amico-validate` bin links on a clean install (was "Failed to create bin … ENOENT";
  dist is built by `pnpm build`, absent at install time). CI now exercises the LINKED
  bin (`pnpm --filter amicode-v2 exec amico-validate --help`) so a broken entry reds.
- CI gate validates ALL FIVE: added solvespec + catalog-entry via --schema (AC5 / DoD).
- Self-contained wrong-type fixture (invalid/result-wrongtype.toml) + a cli.test case —
  the class matrix no longer relies on an in-test mutation (AC7).
- Rename lockstep: index.ts kindForFilename + cli.ts usage + ci.yml gate + cli.test
  KINDS + invalid/run.toml all follow #28's manifest.toml→run.toml / "manifest"→"run".

schema 34 / amico-run 47 / extension 64 (+1 skip) green; linked bin + gate verified locally.
(The bigger real-emit→validate lane Jack flagged lands in #31, the round-trip slice.)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Rchari1
Rchari1 force-pushed the rchari/0.1c-amico-validate-cli branch from f8daf3f to c8f9863 Compare June 28, 2026 20:35
Rchari1 and others added 3 commits June 28, 2026 16:37
…inst the shared schemas (closes #18)

Closes the producer/consumer loop: a Julia validator that checks result.toml /
manifest.toml against the SAME packages/schema/schemas/*.json the TS side uses —
read DIRECTLY (no transcribed copy), so mutating a schema flips both languages.
This is the cross-language anti-drift guarantee (#18 AC6).

- packages/schema/julia/validate.jl: AmicoValidate module + CLI. Reads ../schemas
  via JSONSchema.jl (draft-07, the aju<->JSONSchema.jl common floor), parses TOML,
  validates, prints field-precise errors. A formatting shim turns JSONSchema.jl's
  coarse SingleIssue (path + keyword) into "/path/key: reason" comparable to the TS
  side — incl. reconstructing the missing key for `required` and the version-specific
  schema_version message. Exit 0 valid / 64 invalid. jsonify() coerces unquoted TOML
  datetimes (Dates → ISO string) so they validate like quoted (S2, Julia side).
- packages/schema/julia/runtests.jl (19 tests): valid golden corpus conforms;
  invalid corpus field-precise (offending key/path); schema_version absent +
  unrecognized; unquoted-datetime; single-source asserts (SCHEMA_DIR == ../schemas,
  no .schema.json copy in the Julia tree) — the AC6 anti-drift proof.
- ci.yml: `schema-roundtrip` job (setup-julia + instantiate + runtests.jl). FAST
  tier — JSONSchema only, NO Piccolo solve (per the design review split). The
  freshly-emitted round-trip over a real solve is the slow/nightly extension;
  solve_common.jl + solve_template.jl already stamp schema_version so emitted
  artifacts are round-trippable there.

Committed Manifest.toml for reproducibility (precedent: extension/julia). All four
slices green: TS schema 32 / amico-run 47 / extension 63 + Julia 19.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ise parity, demo-golden round-trip

Address the adversarial review of 0.1d:
- MUST-FIX: re-resolved packages/schema/julia/Manifest.toml so `Test` (a direct
  dep added after the manifest was generated) is present — the committed manifest
  lacked it, so `Pkg.instantiate()` hard-failed the new schema-roundtrip CI job
  before any test ran. Verified a fresh instantiate from the committed manifest now
  exits 0.
- Field-precise PARITY with the TS validator (the shim now reads JSONSchema.jl's
  iss.val): additionalProperties NAMES the offending key (`unknown key "rogue_key"`),
  enum lists allowed values (`must be one of (completed, failed, aborted)`), and
  type/maximum/minimum/exclusiveMinimum/minLength render as prose (`must be number`,
  `must be <= 1.0001`). where-path normalized to `/a/b` | `(root)`.
- Round-trip now also validates the REAL bundled demo run dir (manifest/result/
  FINISHED) — emitted golden, not just hand-authored fixtures (AC5).

Julia round-trip 22 tests green; TS schema 32 / amico-run 47 / extension 63.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nd-to-end (rebased on #28 rename)

Address Jack's #31 review:
- REAL producer lane (not just fixtures): scripts/producer_roundtrip.sh runs amico-run
  on a minimal non-Piccolo emitter (julia/emit_fixture.jl) so it produces a live run-dir
  (run.toml + FINISHED from the orchestrator, result.toml from the Julia emitter), then
  validates the FRESHLY-EMITTED artifacts with validate.jl. CI runs it (fast tier; the
  real Piccolo solve stays the slow/nightly extension). Scoped to what each side actually
  emits — the orchestrator writes run.toml/FINISHED, the Julia producer writes result.toml.
- AC6 exercised, not asserted: scripts/ac6_drift_check.sh perturbs the SHARED schema,
  REBUILDS TS (it bakes schemas at build time — the gap Jack found), then asserts BOTH the
  TS bin AND validate.jl reject a previously-valid fixture, then reverts. Proven locally;
  wired into the schema-roundtrip CI job. runtests.jl's AC6 testset notes the build-vs-
  runtime asymmetry.
- Rename lockstep: validate.jl + runtests.jl manifest.toml→run.toml, "manifest"→"run",
  run.schema.json; resolved the ci.yml conflict keeping #30's all-five gate + this job.

schema 34 / amico-run 47 / extension 64 (+1 skip) / julia 22 green; producer + AC6
scripts pass locally.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Rchari1 Rchari1 changed the title 0.1d — Julia round-trip validates emitted artifacts against the shared schemas (closes #18) 0.1d — Julia round-trip: validate the emitted result.toml + run-dir against the shared schemas (closes #18) Jun 28, 2026
@Rchari1
Rchari1 force-pushed the rchari/0.1d-julia-roundtrip branch from 5dcdc82 to 2ef1d88 Compare June 28, 2026 20:42
@Rchari1

Rchari1 commented Jun 28, 2026

Copy link
Copy Markdown
Member Author

Pushed addressing both must-changes:

1. Real producer lane (not fixtures). scripts/producer_roundtrip.sh runs amico-run on a minimal non-Piccolo emitter → a live run-dir (orchestrator writes run.toml+FINISHED, the Julia emitter writes result.toml) → validates the freshly-emitted artifacts with validate.jl. Wired into the schema-roundtrip CI job (fast tier; the real Piccolo solve stays the slow/nightly extension). Title scoped to result.toml per your note — the orchestrator, not Julia, writes the header.

2. AC6 exercised, not asserted. scripts/ac6_drift_check.sh perturbs the shared schema, rebuilds TS (the build-time-bake gap you found), then asserts both the TS bin and validate.jl flip on a previously-valid fixture, then reverts. Verified locally (AC6 PASS — both flipped); runs in CI.

Plus the manifest.toml→run.toml rename lockstep (validate.jl/runtests.jl). schema 34 / amico-run 47 / extension 64(+1 skip) / julia 22 green. Re-requesting.

@Rchari1
Rchari1 requested a review from jack-champagne June 28, 2026 20:42

@jack-champagne jack-champagne left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 2ef1d88c. Both must-changes resolved — the validator now points at the producer, not its own fixtures:

  • Real producer laneproducer_roundtrip.sh runs amico-run on a minimal emitter → a live run-dir → validate.jl on the freshly-emitted result.toml/run.toml/FINISHED. set -euo pipefail + validate exit codes make it red on drift; wired as a gating step in schema-roundtrip. Title scoped to result.toml per the note.
  • AC6 exercised, not assertedac6_drift_check.sh perturbs the shared run.schema.json, rebuilds TS (the build-time-bake gap I flagged), asserts BOTH the TS bin and validate.jl flip on a previously-valid fixture, then reverts via trap cleanup EXIT. Closes the runtime-vs-build-time asymmetry.

Approving. Retargeting to main + merging behind #30.

@jack-champagne
jack-champagne changed the base branch from rchari/0.1c-amico-validate-cli to main June 29, 2026 03:49
@jack-champagne
jack-champagne merged commit 86a4545 into main Jun 29, 2026
4 checks passed
@kateebonner kateebonner self-assigned this Jul 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants