M1 PR-6: pipeline orchestrator, bounded repair loop, audit report v1#4
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
Adds Milestone 1’s full pipeline orchestrator (generation → S1–S3 → bounded repair loop → A2UI emission + A1–A3 validation) and introduces Audit Report v1 as a first-class, versioned artifact, with deterministic CI coverage via a scripted adapter.
Changes:
- Implement
runPipelineorchestrator with bounded repair loop and emission/validation through pinned@aestheticfunction/dspack-to-a2ui. - Add Audit Report v1 types + Markdown renderer and a v1 JSON schema contract (+ AUDIT stability documentation).
- Add deterministic end-to-end pipeline tests via
ScriptedAdapterand golden repair-message fixtures; extend CLI withrun.
Reviewed changes
Copilot reviewed 11 out of 13 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| src/run/pipeline.test.ts | Deterministic acceptance tests covering pass, repair, exhaustion, gate-failure, and adapter-failure trails. |
| src/run/orchestrator.ts | Core pipeline orchestration with bounded repair loop and A2UI emission + validation. |
| src/repair/render.ts | Deterministic repair-message rendering from lint findings (ADR-7). |
| src/index.ts | Public exports expanded to include adapters, orchestrator, repair rendering, and audit reporting. |
| src/cli.ts | Adds run command that writes audit report outputs and exits with orchestrator exit codes. |
| src/audit/report.ts | Audit Report v1 types, hashing helpers, and Markdown rendering of the JSON artifact. |
| src/adapters/fake.ts | Scripted deterministic adapter used for offline/CI and golden trails. |
| schemas/audit-report.v1.schema.json | JSON Schema contract for audit report v1 (consumer-facing stability surface). |
| package.json | Adds pinned GitHub dependency for @aestheticfunction/dspack-to-a2ui. |
| package-lock.json | Lockfile updates for the pinned emitter dependency. |
| fixtures/golden/repair/F1.repair.txt | Golden repair prompt text used to assert deterministic ADR-7 rendering. |
| docs/AUDIT.md | Documents audit report v1 guarantees (additive-only, stable enums, reproducibility/honesty fields). |
| .gitignore | Ignores out* directories produced by local runs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+139
to
+142
| if (command === "run") { | ||
| void commandRun(flags); | ||
| return; | ||
| } |
Comment on lines
+98
to
+109
| try { | ||
| result = await runPipeline({ | ||
| contract, | ||
| intent, | ||
| prompt, | ||
| adapter: adapterFor(modelRef), | ||
| maxRepairs: flags.has("max-repairs") ? Number(flags.get("max-repairs")) : undefined, | ||
| compile: { | ||
| depth: flags.has("depth") ? Number(flags.get("depth")) : undefined, | ||
| omitRuleSteering: flags.get("no-steering") === "true", | ||
| }, | ||
| }); |
Comment on lines
+123
to
+127
| writeFileSync(join(resolve(outDir), "audit-report.json"), JSON.stringify(result.report, null, 2) + "\n"); | ||
| writeFileSync(join(resolve(outDir), "audit-report.md"), renderMarkdown(result.report)); | ||
| if (result.surfaceMessages) { | ||
| writeFileSync(join(resolve(outDir), "generated.surface.json"), JSON.stringify(result.surfaceMessages, null, 2) + "\n"); | ||
| } |
Comment on lines
+65
to
+68
| const { contract, intent, prompt, adapter } = options; | ||
| const maxRepairs = options.maxRepairs ?? 2; | ||
| const now = options.now ?? (() => new Date()); | ||
| const startedAt = now(); |
Comment on lines
+146
to
+149
| if (index < maxRepairs) { | ||
| const repair = renderRepairMessage(lint.findings, contract); | ||
| repairMessages.push(repair); | ||
| conversation.push({ role: "assistant", content: generated.raw }, { role: "user", content: repair }); |
Comment on lines
+94
to
+100
| "emitted": { | ||
| "type": "object", | ||
| "required": ["target", "validations"], | ||
| "properties": { | ||
| "target": { "const": "a2ui" }, | ||
| "surfaceMessages": {}, | ||
| "warnings": { |
… PR-6) - src/run/orchestrator.ts: generate → surface gates S1-S3 → bounded repair (default 2; system prompt immutable across attempts — the only delta is the model's own output + rendered repair feedback, snapshot per attempt) → emit via the pinned @aestheticfunction/dspack-to-a2ui git dependency → emitter gates A1-A3 (both A2UI versions) → audit report v1. Every outcome is a first-class artifact: passed / failed-lint-exhausted (exit 2) / failed-gate (exit 3) / failed-adapter (exit 1; added to the plan's enum — the S0 spike showed runtimes can fail to constrain, and that must be a reported outcome, never a silent retry). - src/repair/render.ts (ADR-7): one findings object, two serializations — the repair message is rendered deterministically from the same findings embedded in the report, with linked examples verbatim as corrected references. Golden-file tested. - src/audit/: report v1 + schemas/audit-report.v1.schema.json + markdown rendering + docs/AUDIT.md (additive-only guarantee, stable enums, reproducibility fields: contract sha256, schema sha256, adapter id, per-attempt model + provider meta). - src/adapters/fake.ts: ScriptedAdapter — the deterministic instrument for CI, the demo's verification mode, and eval goldens. - CLI `run` writes audit-report.json/.md + generated.surface.json; exit codes per the README table. Live verification (recorded in out/ locally, not committed): qwen3:8b violates S3 on attempt 1 (5 findings), repairs to 1, exhausts honestly (failed-lint-exhausted); gpt-oss 20B passes S1-S3 + A1-A3 on attempt 1. Verify: npm test (52 tests, offline/deterministic); npx tsx src/cli.ts run --dspack fixtures/shadcn.v0_3.dspack.json --intent destructive-action --prompt "a screen to delete my account" --model ollama:<tag> --out out Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ryandmonk
added a commit
that referenced
this pull request
Jul 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
PR-6 of Milestone 1 (stacked on #3). The orchestrator: generate → S1–S3 → bounded repair (default 2; ADR-7) → emit via the pinned git dependency
@aestheticfunction/dspack-to-a2ui#e68a17e→ A1–A3 (both A2UI versions) → audit report v1.passed/failed-lint-exhausted(exit 2) /failed-gate(exit 3) /failed-adapter(exit 1 — added to the plan's enum because the S0 spike showed runtimes can silently fail to constrain; documented indocs/AUDIT.md).schemas/audit-report.v1.schema.json+docs/AUDIT.md): additive-only guarantee, stable enums, reproducibility fields (contract sha256, schema sha256, adapter id, per-attempt model + provider meta), honesty fields (ruleSteering, emitter warnings). Markdown rendering is a derived view.runwritesaudit-report.json/.md+ the emitted surface;ScriptedAdapter(src/adapters/fake.ts) is the deterministic instrument for CI/demo/evals.Live runs during development (local
out*/, not committed): one local 8B-class model violated S3 on attempt 1, repaired 5 findings → 1, exhausted honestly; a ~20B model passed everything first attempt. Model-quality claims stay out of docs pending the PR-10 eval matrix.Acceptance (in CI on this PR)
Hand-review focus
schemas/audit-report.v1.schema.json+docs/AUDIT.md(the enterprise-facing artifact contract — on your queue) and theRunOptions/RunResultsurface insrc/run/orchestrator.ts.ADRs: 4, 5, 7, 8, 9, 11.
🤖 Generated with Claude Code