fix: require passing evidence for supported claims

.github/workflows/ci.yml

@@ -358,8 +358,17 @@ jobs:
         run: |
           mkdir -p runs
           pytest -q --ignore=tests/e2e --basetemp=runs/ci \
+            --junitxml=runs/unit-claims-junit.xml \
             --cov=openadapt_flow --cov-report=
 
+      # A supported claim is not a function of file existence. Bind every
+      # cited unit-test file to this job's real Pytest result. Missing and
+      # all-skipped files fail closed even when another test passed.
+      - name: Validate passing unit claim evidence
+        run: |
+          python scripts/validate_claims.py --check \
+            --ci-job test --junit runs/unit-claims-junit.xml
+
       # Enforced RATCHET floor on the safety-critical path (branch-inclusive).
       # Scoped via --include to exactly the safety modules (compiler, identity,
       # effects, policy, replayer, resolver). 85% sits just under the fast
@@ -384,6 +393,7 @@ jobs:
             runs/**/BENCH.md
             runs/**/report.json
             runs/**/*.png
+            runs/unit-claims-junit.xml
           if-no-files-found: warn
 
   # --- Browser record -> compile -> replay E2E (REQUIRED on PRs) -----------
@@ -447,8 +457,16 @@ jobs:
           mkdir -p runs
           pytest -q tests/e2e \
             --ignore=tests/e2e/test_free_path_e2e.py \
+            --junitxml=runs/e2e-claims-junit.xml \
             --basetemp=runs/ci
 
+      # The browser maturity claims cite the record/compile/replay E2E file.
+      # Require that file to appear and pass in this required job's own JUnit.
+      - name: Validate passing browser claim evidence
+        run: |
+          python scripts/validate_claims.py --check \
+            --ci-job e2e-browser --junit runs/e2e-claims-junit.xml
+
       - name: Upload run artifacts
         if: always()
         uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
@@ -459,6 +477,7 @@ jobs:
             runs/**/BENCH.md
             runs/**/report.json
             runs/**/*.png
+            runs/e2e-claims-junit.xml
           if-no-files-found: warn
 
   # --- Native Linux: real GTK3 + AT-SPI under isolated X11 (REQUIRED) ------

.github/workflows/validate-claims.yml

@@ -2,13 +2,15 @@ name: Validate claims
 
 # Make every public maturity claim a FUNCTION of automated evidence.
 #
-#   - pull_request: run the fast claim->evidence GATE
-#     (scripts/validate_claims.py --check) on every PR. This is a
+#   - pull_request: run the fast claim->evidence STRUCTURE GATE
+#     (scripts/validate_claims.py --check --structure-only) on every PR. This is a
 #     required-check CANDIDATE: wire it into branch protection as the context
 #     "gate" (the actual CheckRun job name from the "Validate claims" workflow)
 #     to block a PR that overclaims (a `supported` claim whose only backing is
 #     an opt-in/infra-gated test, a `field` result labeled `supported`, or a
-#     missing evidence path). It NEVER touches infra.
+#     missing evidence path). The required CI `test` and `e2e-browser` jobs
+#     separately bind every supported test file to their real JUnit results.
+#     It NEVER touches infra.
 #   - schedule + workflow_dispatch: the infra-gated leg that provisions a real
 #     Windows VM (via oa-vm) and runs the OPT-IN desktop + Citrix e2e that back
 #     the `validating` tiers, then regenerates the evidence report. Where the
@@ -55,8 +57,8 @@ jobs:
       - name: Install PyYAML
         run: pip install "pyyaml>=6"
 
-      - name: Claim -> evidence gate
-        run: python scripts/validate_claims.py --check
+      - name: Claim -> evidence structure gate
+        run: python scripts/validate_claims.py --check --structure-only
 
       # Prove the doc is regenerable and in sync with the registry: regenerate
       # docs/VERIFICATION.md + docs/verification.json into a temp dir would be
@@ -154,7 +156,7 @@ jobs:
         if: steps.infra.outputs.available == 'true'
         run: |
           python scripts/validate_claims.py --report \
-            --junit runs/validating-junit.xml
+            --ci-job validating --junit runs/validating-junit.xml
 
       - name: Upload refreshed evidence
         if: steps.infra.outputs.available == 'true'

claims.yaml

@@ -13,7 +13,8 @@
 # -----------------------------------------------------
 # Evidence strength is derived from the repo, not asserted here:
 #   * a test file with NO module-level env skipif, that exists      -> "supported"
-#     (it actually runs, and can be green, in CI)
+#     candidate evidence; its required `test` or `e2e-browser` job must bind the
+#     file to at least one passing case in that job's real JUnit result
 #   * a test file gated by a module-level `pytestmark` env skipif    -> "validating"
 #     (opt-in / infra-gated: grounded, but never runs on default CI)
 #   * a doc / benchmark artifact (.md or a benchmark dir)            -> "roadmap"
@@ -24,7 +25,7 @@
 # result that is not CI-reproducible is never presented as "supported").
 #
 # TIERS
-#   supported  — CI-proven today on the default (non-opt-in) suite.
+#   supported  — bound to a real passing result in required default CI.
 #   validating — grounded in a REAL opt-in/infra-gated proof or a field test;
 #                being validated (incl. with design partners). NOT "supported".
 #   roadmap    — designed/specified, not yet proven by a running test.

docs/VERIFICATION.md

@@ -3,20 +3,21 @@
 > GENERATED by `scripts/validate_claims.py --report` from `claims.yaml`. Do not edit by hand — edit the registry and regenerate.
 
 - Generated at: **committed registry state (regenerate: scripts/validate_claims.py --report)**
-- Green-check against a junit artifact: **not run** (no `--junit` artifact supplied)
-- Gate: `python scripts/validate_claims.py --check` (a claim whose tier outranks its strongest backing evidence fails CI).
+- Green-check against a junit artifact: **not embedded in this generated registry view** (required CI jobs enforce pass evidence)
+- Structure gate: `python scripts/validate_claims.py --check --structure-only` (a claim whose tier outranks its strongest backing evidence fails CI).
+- Pass gates: required `test` and `e2e-browser` jobs supply their own JUnit files; an absent, all-skipped, or failed supported evidence file fails that required job.
 
-**What this harness does and does not do.** It makes each public maturity claim a *function* of automated evidence: a `supported` claim must be backed by a test that actually runs on the default (non-opt-in) CI suite; a `validating` claim must be grounded in a REAL opt-in / infra-gated proof or a field test, and is never presented as supported. It does not replace workflow- and deployment-specific acceptance: application controls, identity rules, effect oracles, and live transport conditions remain bound to their counted evidence.
+**What this harness does and does not do.** It makes each public maturity claim a *function* of automated evidence: a `supported` claim must be backed by a test file that has a real passing case in its required default CI job; a `validating` claim must be grounded in a REAL opt-in / infra-gated proof or a field test, and is never presented as supported. It does not replace workflow- and deployment-specific acceptance: application controls, identity rules, effect oracles, and live transport conditions remain bound to their counted evidence.
 
-## What is CI-proven today vs. being validated
+## What is bound to required CI vs. being validated
 
-- **CI-proven today (11):** `web-supported`, `deterministic-zero-model-replay`, `effect-verification-silent-writes`, `identity-gate-halt-armed`, `identifier-crop-compile-emission`, `halt-teach-promote`, `typed-business-decisions-supported`, `reviewed-judgment-cases-supported`, `qualified-remote-decision-v2-supported`, `mockmed-benchmark-ci-reproducible`, `substrate-runtime-validation-supported`
+- **Bound to required CI pass evidence (11):** `web-supported`, `deterministic-zero-model-replay`, `effect-verification-silent-writes`, `identity-gate-halt-armed`, `identifier-crop-compile-emission`, `halt-teach-promote`, `typed-business-decisions-supported`, `reviewed-judgment-cases-supported`, `qualified-remote-decision-v2-supported`, `mockmed-benchmark-ci-reproducible`, `substrate-runtime-validation-supported`
 - **Being validated — opt-in / infra-gated or field (8):** `effect-verifier-kit`, `windows-desktop-validating`, `macos-native-validating`, `linux-native-validating`, `rdp-validating`, `desktop-recording-validating`, `citrix-pixel-validating`, `openemr-field-benchmark`
 - **Roadmap / research (1):** `win32-window-replay-roadmap`
 
 ## Claims
 
-### `web-supported` — supported — CI-proven today
+### `web-supported` — supported — bound to required CI pass evidence
 
 > Web (browser) workflows are supported today: record a GUI workflow once, then replay it deterministically and locally.
 
@@ -34,7 +35,7 @@
 - "Supported" is scoped to the reference headless-browser backend in this registry. Desktop and remote-display workflows use the separately scoped acceptance and code-qualified claims below.
 - The full record->compile->replay browser suite runs in the required e2e-browser PR gate and repeats in the weekly compatibility matrix.
 
-### `deterministic-zero-model-replay` — supported — CI-proven today
+### `deterministic-zero-model-replay` — supported — bound to required CI pass evidence
 
 > Replay is deterministic, local, and $0 — zero model calls on either the record or the replay side.
 
@@ -51,7 +52,7 @@
 
 - A VLM rung exists as the lowest fallback of the ladder; it is not invoked on a clean deterministic replay, and never on the record side.
 
-### `effect-verification-silent-writes` — supported — CI-proven today
+### `effect-verification-silent-writes` — supported — bound to required CI pass evidence
 
 > Effect verification against the system of record catches silent wrong writes the screen-only oracle misses (5 of 7 transactional fault classes).
 
@@ -90,7 +91,7 @@
 
 - CONTRACT-PROVEN, not live-proven: the SQL verifier is exercised against sqlite fixtures (no production MariaDB/Postgres), the SFTP arrival path against a fake transport (no real SFTP server), and the worked-example configs are templates — only the FHIR substrate has an additional opt-in live-OpenEMR test. Per-verifier claims are exactly as strong as these tests.
 
-### `identity-gate-halt-armed` — supported — CI-proven today
+### `identity-gate-halt-armed` — supported — bound to required CI pass evidence
 
 > On ambiguity the runtime halts instead of guessing, and an identity gate refuses a wrong-entity (wrong-patient) click — on ARMED steps only.
 
@@ -108,7 +109,7 @@
 
 - Identity verification covers ONLY armed steps. Real bundles arm a MINORITY of clicks (4 of 12 on a recent live OpenEMR bundle); an UNARMED click proceeds with no identity check whatsoever. Coverage is an auditable per-step metric, but disclosure does not close the gap.
 
-### `identifier-crop-compile-emission` — supported — CI-proven today
+### `identifier-crop-compile-emission` — supported — bound to required CI pass evidence
 
 > The compiler emits a pixel identifier crop (anchor.identifier_crop) for identity-armed steps without structured identity — and for any step whose identifying region is marked at record time (--identifier) — so the pixel-compare identity tier arms on remote-display/pixel replays; every crop-less identity-applicable step records an explicit degrade reason, and lint surfaces per-bundle pixel-identity coverage.
 
@@ -125,7 +126,7 @@
 - Mechanism proven on synthetic fixtures and the bundled demo app only — this does NOT re-qualify any desktop/remote-display substrate, and the pixel tier remains MISMATCH-or-ABSTAIN (it can add a safe halt on a wrong identifier, never authorize a match; PIXEL_VERIFY_ENABLED stays off pending a jitter-robust distance).
 - Automatic emission requires an OCR-readable identity band at compile time; icon-only or unreadable rows still compile crop-less (with the recorded reason) and fall back to the OCR band tier at replay.
 
-### `halt-teach-promote` — supported — CI-proven today
+### `halt-teach-promote` — supported — bound to required CI pass evidence
 
 > A halt is learnable: halt -> teach a correction -> promote a guarded branch, refusing underdetermined fixes.
 
@@ -141,7 +142,7 @@
 
 - Promotion is governed by a regression gate that blocks identity- weakening corrections; the loop refuses to guess when the fix is underdetermined.
 
-### `typed-business-decisions-supported` — supported — CI-proven today
+### `typed-business-decisions-supported` — supported — bound to required CI pass evidence
 
 > A qualification client can add or update a typed finite business-policy decision without editing an internal manifest. The compiled workflow can then pause at that decision, accept one authorized and attributed choice, retain a signed durable receipt, revalidate the live application, and continue only through the certified successor branch.
 
@@ -158,7 +159,7 @@
 - The engine validates a principal and its roles; it does not authenticate a self-asserted user. Desktop, Cloud, or a customer-local identity route must supply the authenticated principal.
 - A human choice is control authority only. It cannot satisfy entity identity, a postcondition, or effect verification, and it cannot turn a screen statement or a human assertion into VERIFIED.
 
-### `reviewed-judgment-cases-supported` — supported — CI-proven today
+### `reviewed-judgment-cases-supported` — supported — bound to required CI pass evidence
 
 > A qualification project can bind typed local facts, evidence hashes, reviewer provenance, and the exact decision contract to reviewed examples and counterfactuals. It can retain permanent human authority, record a reviewed automatic-rule candidate, or refuse certification until more evidence exists, without converting one historical choice into policy.
 
@@ -176,7 +177,7 @@
 - An automatic-rule case names only a reviewed rule identifier and finite option. Flow does not infer or generate executable policy from the case; the rule must be authored and qualified through the normal program path.
 - A retained human-node case preserves runtime human authority. A more-evidence-required case refuses certification instead of guessing.
 
-### `qualified-remote-decision-v2-supported` — supported — CI-proven today
+### `qualified-remote-decision-v2-supported` — supported — bound to required CI pass evidence
 
 > A negotiated V2 attended task binds optional reviewed entity wording to the exact qualification, bundle, step, policy, and pause authority. An unqualified or unrecognized entity class stays local and the remote task uses the signed neutral record or item fallback.
 
@@ -194,7 +195,7 @@
 - V2 requires openadapt-types 0.10.x and explicit peer negotiation. The dependency does not upgrade a V1 consumer, and the byte-compatible V1 path remains available.
 - The entity class is presentation metadata only. Actual record identity remains inside the customer-controlled runner and is revalidated before any resumed action.
 
-### `mockmed-benchmark-ci-reproducible` — supported — CI-proven today
+### `mockmed-benchmark-ci-reproducible` — supported — bound to required CI pass evidence
 
 > The MockMed benchmark is CI-reproducible: compiled replay finishes ~4.9s p50 with zero model calls versus ~37.5s p50 (~$0.27/run) for the agent. Measured 2026-07-08 on Flow 0.1.0, a pre-v0.2.0 source build; not re-measured on a later release.
 
@@ -289,7 +290,7 @@
 - Acceptance covers the named 1280x800 Windows Run/file task and exact VM environment, not arbitrary applications, identity policies, or Citrix.
 - The multi-window campaign uses deterministic synthetic applications. It exercises the real FreeRDP pixel and input path but does not qualify a named customer application or environment.
 
-### `substrate-runtime-validation-supported` — supported — CI-proven today
+### `substrate-runtime-validation-supported` — supported — bound to required CI pass evidence
 
 > Runtime-validation v3 binds Standard and Regulated hosted activation to the exact governed authorization template reproduced from the sealed qualification. It also binds the successful run's resolved target kind (`web|windows|macos|linux|rdp|citrix`), artifact, compiler, parameter schema, strict lint, policy, risk, report, and opaque environment. Web retains its exact HTTPS/host boundary. Native and remote execution emits no app, window, host, readiness, or backend-hint values to Cloud. Local verification retains signed v1 and v2 compatibility.
 
@@ -385,4 +386,3 @@
 
 - The historical result uses OCR evidence from a visible saved message row. It does not use an out-of-band OpenEMR system-of-record read.
 - FIELD TEST, NOT CI-reproducible: the head-to-head ran against a SHARED public demo that other users mutate and that resets daily; the sample is small (10 agent runs). Only the verifier + task-prompt units run in CI. `reproducibility: field` forbids ever labeling this `supported`.
-