Skip to content

fix: require passing evidence for supported claims - #368

Closed
abrichr wants to merge 1 commit into
codex/quickstart-lifecyclefrom
codex/claims-passing-evidence
Closed

fix: require passing evidence for supported claims#368
abrichr wants to merge 1 commit into
codex/quickstart-lifecyclefrom
codex/claims-passing-evidence

Conversation

@abrichr

@abrichr abrichr commented Aug 18, 2026

Copy link
Copy Markdown
Member

Summary

This is a stacked draft PR. It depends on #367 and targets its branch so the review contains only the claims-evidence change.

A supported claim no longer passes because a cited test file exists. The required CI job must supply real JUnit evidence that each cited file ran and passed.

Root cause

The former gate had four fail-open paths:

  • test-file existence was enough to classify evidence as supported;
  • --junit was optional;
  • a missing JUnit file produced a warning and passed;
  • the parser expected a file attribute that Pytest xUnit2 does not emit, so real Pytest output could produce an empty status map.

Change

  • The existing required test job emits unit JUnit and validates unit-backed claims.
  • The existing required e2e-browser job emits browser JUnit and validates browser-backed claims.
  • The separate fast claims workflow remains a structure-only gate.
  • Missing, malformed, empty, absent, all-skipped, failed, or errored evidence fails closed.
  • Pytest xUnit2 dotted class names map to canonical repository test paths.
  • Generated verification output names the required CI binding and does not claim a live pass from its static registry view.
  • The public artifact inventory contains the new workflow, registry, and generated JSON hashes.

This reuses the existing required test executions. It does not add a duplicate test suite or a new expensive CI job.

Validation

  • Focused validator and workflow contract suite: 29 passed.
  • Exact cited unit evidence: 681 passed, 2 skipped. The real unit JUnit claim gate passed.
  • Exact cited browser evidence: 12 passed. The real browser JUnit claim gate passed.
  • Claims structure gate: passed.
  • Release consistency and public artifact inventory: passed.
  • actionlint: passed.
  • Ruff check and format check: passed.
  • Generated verification files: reproducible with the fixed sentinel.
  • git diff --check: passed.

Stack

After #367 merges, retarget or rebase this PR onto main. Do not merge this PR first.

@abrichr
abrichr deleted the branch codex/quickstart-lifecycle August 18, 2026 18:20
@abrichr abrichr closed this Aug 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant