Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ AI proposal (hosted | local | scripted) ← the only seam that varies
→ registry renderer (native + wireframe fallback)
```

The proposal provider differs; **the deterministic authority does not**. Hosted proposals come from managed Claude via the AI Gateway; local proposals go through the agent to your endpoint; scripted replays the intent's worked example — all through identical gates, in your browser for browser projects, or against files on disk for repository projects. Contract-time emission (catalogs from contracts) and runtime rendering (AG-UI events → A2UI processor → registry) are the same machinery the Studio exhibit demonstrates. Deeper docs: [docs/COMPOSER.md](docs/COMPOSER.md), [docs/renderer-abstraction.md](docs/renderer-abstraction.md), [docs/AUDIT.md](docs/AUDIT.md).
The proposal provider differs; **the deterministic authority does not**. Hosted proposals come from managed Claude via the AI Gateway; local proposals go through the agent to your endpoint; scripted replays the intent's worked example — all through identical gates, in your browser for browser projects, or against files on disk for repository projects. Contract-time emission (catalogs from contracts) and runtime rendering (AG-UI events → A2UI processor → registry) are the same machinery the Studio exhibit demonstrates. Deeper docs: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) (why the product is shaped this way — the thesis, the invariants, what Composer deliberately does not do), [docs/COMPOSER.md](docs/COMPOSER.md) (what runs where), [docs/renderer-abstraction.md](docs/renderer-abstraction.md), [docs/AUDIT.md](docs/AUDIT.md). The twelve-prompt enterprise acceptance corpus, its expected classification, and a reproducible harness live in [acceptance/gateway-corpus](acceptance/gateway-corpus/README.md).

## The Studio (second app)

Expand Down
1 change: 1 addition & 0 deletions acceptance/gateway-corpus/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
evidence/
33 changes: 33 additions & 0 deletions acceptance/gateway-corpus/EXPECTED.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Expected classification

The state of each prompt against the current product. **Solved** means the requested interface is achievable as a governed artifact. **Partial** means a real version builds but a named part of the request is not representable. **Gap** means the vocabulary to express it honestly does not exist, and Composer says so rather than faking it.

Runtime caveats are called out because they are the product's stated boundary: Composer composes and governs *representations*; it does not bind live data or execute workflows.

| # | Prompt | Status | Owner of what remains |
|---|---|---|---|
| 1 | Service Catalog Explorer | **Partial** | Design-system vocabulary — a dependent selector needs option lists that react to another control; `Select.options` is a static literal and the change handlers are dropped props. What builds is a static configuration form |
| 2 | Filing Directory Browser | **Partial** | Design-system vocabulary — no combobox / type-ahead / query vocabulary. The shadcn contract's own `select.whenNotToUse` names the missing component |
| 3 | Estimate Workspace | **Solved** | — |
| 4 | Create Estimate | **Solved** as a flow | Runtime — the post-create read-back is live data |
| 5 | Convert Estimate to Project | **Solved** as a flow | Runtime — same |
| 6 | Direct Project Creation | **Solved** as a flow | Runtime — same |
| 7 | Project Workspace | **Solved** | — (line-item tables render natively; the renderer defect that once emptied shadcn tables is fixed and regression-guarded) |
| 8 | Mutation Confirmation | **Solved** for one state | Product model — a reusable *multi-variant* composition (calm vs destructive from one definition) is parameterized reuse, deliberately deferred |
| 9 | Operation Progress | **Gap** | Design-system vocabulary + one governance rule — indeterminate progress is the request's core demand and the shadcn rule requires a determinate value; Astryx has no progress vocabulary and refuses honestly |
| 10 | File / Artifact Card | **Partial** | Runtime — metadata, states, and a download *action* are representable; downloading is not |
| 11 | Gateway Error / Recovery | **Gap** | Design-system vocabulary — field-level error binding does not exist, and the alert action slot is still dropped by the profile pending upstream work |
| 12 | Full demo composition | **Solved** as a flow | Runtime — read-backs and context carriage between steps |

**Tally: 6 solved, 4 partial, 2 gaps.** At the start of the investigation that produced this corpus, none of the twelve was achievable as intended.

## How to read a failure

- **A gate failure (S1/S2/S3)** is the model producing something the design system does not allow. Expected occasionally; the repair loop gets a bounded number of tries.
- **An emit refusal** is the surface being contract-legal but unrepresentable — a declared casualty, a dissolution rule, a missing join key. Since the pipeline-trust milestone these ride the repair loop with the refusal text as the instruction, so a terminal refusal means repair genuinely could not fix it.
- **A planning vocab-gap** is the honest one: the planner said the request needs a capability the design system does not have, and no generation was attempted.
- **An adapter error** is infrastructure, not product. Re-measure.

## Keeping this current

Update this table when a milestone changes a verdict, and say what changed it. Do not soften a classification to make a release look better — the corpus's only value is that it has never been graded generously.
41 changes: 41 additions & 0 deletions acceptance/gateway-corpus/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# The Gateway acceptance corpus

Twelve prompts describing a real enterprise workflow product — an IP-services Gateway: service configuration, entity directories, estimates, estimate→project conversion, project workspaces, confirmation boundaries, operation progress, artifact cards, error recovery, and one end-to-end lifecycle.

**This is the honest measuring stick for "can Composer build real enterprise software?"** It is not a demo script and not a unit-test fixture. It was written by the product owner as an acceptance suite before the product could satisfy any of it, and it has driven four milestones of measured improvement since.

## Rules

1. **The prompts are immutable.** Do not rewrite, simplify, shorten, or "fix" them. Their awkward parts are the point — real requests are not tidy. If a prompt cannot be satisfied, that is a finding about the product, never a reason to edit the corpus.
2. **The goal text is exactly the body of each prompt.** Titles, tool names, and the author's side-notes are metadata and are never pasted into the product.
3. **New prompts are additions, never replacements.** Preserve the original twelve and their numbering.
4. **Do not use the corpus to justify new vocabulary on its own.** A gap it reveals is evidence for a decision, not the decision.

## What is here

| File | What it is |
|---|---|
| `corpus.mjs` | The twelve prompts verbatim, with their metadata |
| `EXPECTED.md` | The current per-prompt classification, with the first blocker and which layer owns it |
| `harness.mjs` | A reproducible runner: drives the real pipeline per prompt × design system and writes one evidence file per cell |
| `paraphrases.mjs` | Reworded versions of four prompts, used to prove an improvement generalized beyond the exact strings |

## Running it

The harness makes **live model calls** through the hosted gateway (or a local model). It is therefore **not part of CI** and never should be: it costs real inference, it is nondeterministic, and a red result usually means "the model had a bad day," which is not a signal CI can act on.

```bash
node acceptance/gateway-corpus/harness.mjs all
```
Comment on lines +27 to +29

Evidence lands in `acceptance/gateway-corpus/evidence/` (git-ignored), one JSON file per prompt × design system, containing the plan, every generation attempt with its gate results, the emitted output, raw validator findings, and renderer coverage.

Practical notes learned the hard way:

- **Run it sequentially and paced.** After a few hundred calls in a day the shared provider begins refusing build-paced traffic while still answering slow probes; bursts make cells fail with adapter errors that say nothing about the product. The harness paces itself and retries, and it is resumable — existing evidence files are skipped.
- **Generation is stochastic.** One sample per cell is what a user experiences; two or three samples are what a *conclusion* needs. Composition-heavy prompts flip between runs.
- **An adapter failure is not a product result.** Discard and re-measure those cells rather than reporting them.

## Provenance

Delivered by the product owner on 2026-08-10 with the instruction: *"Treat these prompts as immutable acceptance tests. Do not rewrite them. Do not simplify them. Investigate why they succeed or fail."* The corpus has since been the acceptance basis for the pipeline-trust, generation-quality, and flow-composition milestones.
Loading
Loading