From 73dccc316017084235d719600a73aebda74e6789 Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Fri, 7 Aug 2026 00:13:41 -0500 Subject: [PATCH 1/2] feat(security): audit the latest release's pinned runtime, which the daily audits do not cover (BACKLOG #1081) The `pip-audit` gate audits the CHECKED-OUT tree, so on the daily cron it audits `main`. "Is what we would ship next current" is a different question from "does the version we already shipped carry a known advisory", and the two answers diverge from the moment a fix lands on main until a release carries it. Nothing watched that window. `released-line-audit` does, on the existing daily cron. WHAT THIS IS NOT. The framing that motivated this -- "advisories disclosed after a tag are never re-evaluated" -- was WRONG and is retracted in #1081. CVE-2026-69247 was caught the day it published, by an existing required gate: commit ac87246f records "pip-audit (a required gate) flagged cryptography 49.0.0 for CVE-2026-69247". Nor was main ahead of the tag; both read 49.0.0 at that moment. Detection was never missing. Only the release-lag window was unwatched. THE INSTRUMENT WAS PROVEN BEFORE THE GATE WAS WRITTEN, because a gate nobody proved can see its own defect class is the failure this repo keeps re-learning. Against v0.3.2's core lock `pip-audit` exits 1 naming PYSEC-2026-3552 on cryptography 49.0.0; against origin/main's it exits 0 -- so it distinguishes the two states rather than only ever reddening, which is the false-deny direction no red test surfaces. An empty lock reads 0 pinned requirements against a floor of 25 and takes the fail-closed path. The tag selector returns exactly v0.3.2 and excludes webconsole-v0.2.15. The positive control is durable: the vulnerable lock is in git history, so a dispatch with released_line_audit_tag=v0.3.2 re-arms it forever. NO VEX IS APPLIED TO THIS GATE, and that is the load-bearing design decision. security/vex/README.md's own worked example names the product with no version qualifier, so a `fixed` or `not_affected` statement written on main would suppress the finding against the ALREADY-SHIPPED release -- turning the gate green at the exact moment the assessment was written and before any release carried the fix. `--ignore-vuln ` is the escape hatch instead: explicit, per-advisory, greppable. Advisory by PLACEMENT, not by continue-on-error -- the dast.yml posture. It is schedule/dispatch-only so it can never report on a PR and must never be required (the required-but-absent trap), but it still goes red on a finding. tests/test_security_posture.py could not express that: it had two buckets and assumed anything not required carries continue-on-error. It now has three, and the new bucket asserts the opposite -- no continue-on-error, not required, and never reachable from a pull_request. nightly-notice.yml now watches `Security` as well as `CI`, because a red schedule-only job otherwise reports nowhere. Its issue title is DERIVED from the completed workflow's name rather than hardcoded, so a green nightly CI cannot close an issue a red Security run opened; for CI the title is byte-identical to today's, so any currently-open issue stays matched. Also corrects docs/CI.md, which said the only non-required security.yml jobs are `sbom` and `trivy` and that both declare continue-on-error. Three jobs now sit outside the required set, in two different ways, and the new one deliberately declares no continue-on-error. Deferred deliberately: the docs/SUPPLY-CHAIN.md half -- scoping "continuously audited by pip-audit" to main's lockfiles, and the releases/latest/download permanent fetch URLs. PR #264 edits that same file; stacking the two would risk a conflict. #1081 records it, along with at least five other things left unbuilt and why. Self-tests A-D above were run pre-merge. E-G (live dispatch, notifier, recovery) need this on main and are listed in #1081. --- .github/required-contexts.txt | 8 ++ .github/workflows/nightly-notice.yml | 22 ++-- .github/workflows/security.yml | 149 +++++++++++++++++++++++++++ docs/BACKLOG.md | 29 ++++++ docs/CI.md | 10 +- tests/test_nightly_notice.py | 11 ++ tests/test_security_posture.py | 63 ++++++++++- 7 files changed, 280 insertions(+), 12 deletions(-) diff --git a/.github/required-contexts.txt b/.github/required-contexts.txt index c158e301..c740b721 100644 --- a/.github/required-contexts.txt +++ b/.github/required-contexts.txt @@ -115,6 +115,14 @@ a PR that implements BACKLOG #N must update BACKLOG.md # never report on a PR (required-but-absent trap). Advisory by # placement, not by continue-on-error: the job goes red on a finding. # See ADR 0155. +# security.yml's -- `released-line-audit (latest release's pinned runtime)` is +# released-line-audit schedule/dispatch only, so it can never report on a PR +# (required-but-absent trap). Advisory by placement, not by +# continue-on-error: it goes red on a finding, and +# tests/test_security_posture.py holds that. It also could not be +# actioned by a PR author -- its remedy is cutting a release on the +# latest line (docs/SUPPORT-POLICY.md), not editing the branch under +# review. BACKLOG #1081. # ingress-rate-probe.yml -- workflow_dispatch only, so it never reports on a PR. It is also # structurally unfit to gate anything: it MEASURES a runner's ingress # service rate and exits 0 even when the reconcile fails, because a diff --git a/.github/workflows/nightly-notice.yml b/.github/workflows/nightly-notice.yml index 8841037f..9828a20f 100644 --- a/.github/workflows/nightly-notice.yml +++ b/.github/workflows/nightly-notice.yml @@ -19,9 +19,15 @@ name: Nightly failure notice # # SCOPE: schedule-only, deliberately. A push/PR failure is already visible on the PR itself; alerting # there would be pure noise, and `workflow_run` fires for every CI completion regardless of trigger. +# +# TWO WORKFLOWS, ONE ISSUE PER WORKFLOW. `Security` was added because its daily cron carries jobs that +# do not run on a PR -- `released-line-audit` most of all, whose whole subject (the LATEST RELEASE) no +# PR can change. The issue title is DERIVED from the completed workflow's name, so a green nightly CI +# cannot close an issue opened by a red Security run: a single shared title would let one signal close +# the other, which is the same silence this workflow exists to end. on: workflow_run: - workflows: ["CI"] + workflows: ["CI", "Security"] types: [completed] # Read-only by default; the one job that writes escalates to `issues: write` and nothing else. @@ -49,10 +55,13 @@ jobs: RUN_URL: ${{ github.event.workflow_run.html_url }} RUN_STARTED: ${{ github.event.workflow_run.run_started_at }} HEAD_SHA: ${{ github.event.workflow_run.head_sha }} + # Derived, not hardcoded: one issue per watched workflow. A shared title would let a green + # nightly CI close an issue a red Security run opened. + WF_NAME: ${{ github.event.workflow_run.name }} run: | set -euo pipefail - TITLE="Nightly CI is failing" + TITLE="Nightly $WF_NAME is failing" LABEL="bug" # Find an OPEN issue with this exact title. `--search` is deliberately anchored on the title @@ -68,7 +77,7 @@ jobs: # nag nobody reads -- an alert that is always on is the same as no alert. if [ -n "$existing" ]; then gh issue comment "$existing" --body \ - "Nightly CI is green again as of [this run]($RUN_URL) (\`$HEAD_SHA\`). Closing." + "Nightly $WF_NAME is green again as of [this run]($RUN_URL) (\`$HEAD_SHA\`). Closing." gh issue close "$existing" echo "closed #$existing — nightly recovered" else @@ -90,9 +99,10 @@ jobs: - commit: \`$HEAD_SHA\` - started: $RUN_STARTED - A scheduled run is not a PR context, so this failure appears nowhere else. The nightly-only - legs are the server-DB store, load/throughput and service-smoke suites — exactly the ones the - three required \`test\` legs skip — so a break here is invisible until someone looks. + A scheduled run is not a PR context, so this failure appears nowhere else. For CI that is the + server-DB store, load/throughput and service-smoke legs the three required \`test\` legs skip; + for Security it is the daily dependency audits and the released-line audit, whose subject is a + published release that no pull request can change. This issue is opened once and commented on each subsequent failure; it closes itself when a nightly goes green again." diff --git a/.github/workflows/security.yml b/.github/workflows/security.yml index 8d98086d..271f0ab3 100644 --- a/.github/workflows/security.yml +++ b/.github/workflows/security.yml @@ -38,6 +38,11 @@ on: # clock. pip-audit/npm-audit here are a second detector independent of GitHub's advisory-review lag. - cron: "0 6 * * *" workflow_dispatch: + inputs: + released_line_audit_tag: + description: "released-line-audit: audit THIS release tag instead of the selected latest. Leave blank for normal runs; used to re-run the positive control against an older tag." + required: false + type: string concurrency: # Per-ref for PR events so a rapid re-push supersedes its own in-flight scan (ci.yml's pattern — @@ -164,6 +169,150 @@ jobs: # fast and reproducible. Default level fails on ANY severity, matching pip-audit's strict posture. run: npm audit --package-lock-only + released-line-audit: + name: released-line-audit (latest release's pinned runtime) + runs-on: ubuntu-latest + # WHAT THIS ANSWERS, and why the `pip-audit` job above does not. That job audits the CHECKED-OUT + # tree, so on the daily cron it audits `main`. "Is what we would ship next current" is the right + # question and it is a DIFFERENT question from "does the version we already shipped carry a known + # advisory". The two answers diverge from the moment a fix lands on main until a release carries + # it, and nothing watched that window. + # + # MEASURED 2026-08-06, stated in the conditional because there are ZERO deployments (CLAUDE.md + # section 0). v0.3.2 was tagged 2026-07-28 pinning cryptography 49.0.0. GHSA-g6cj-pr64-35w5 / + # CVE-2026-69247 / PYSEC-2026-3552 (fixed in 50.0.0) published 2026-08-03. The required `pip-audit` + # gate above caught it on main THE SAME DAY and the pin was bumped -- that gate worked, and this job + # is NOT a second detector for it. What did not exist was anything noticing that days later no + # release carries the fix, so an operator scanning the published v0.3.2 SBOM WOULD see an advisory + # our shipped VEX says nothing about (security/vex/messagefoundry.openvex.json has no statements). + # + # SCOPE = the latest released line only. docs/SUPPORT-POLICY.md -- only the latest released version + # is supported pre-1.0 and there is no back-port. A finding against an older tag has no supported + # remedy, so scanning one would manufacture unactionable work. + # + # ADVISORY BY PLACEMENT, NOT BY continue-on-error -- the dast.yml posture recorded in + # .github/required-contexts.txt. It goes RED on a finding. It is schedule/dispatch-only, so it can + # never report on a PR and must never become a required context (the required-but-absent trap). + # tests/test_security_posture.py pins both halves of that. + # + # NO VEX IS APPLIED HERE, deliberately. Applying our maintained OpenVEX would let a `fixed` or + # `not_affected` statement written on main suppress the finding against the ALREADY-SHIPPED + # artifact -- security/vex/README.md's own worked example names the product with no version + # qualifier, so a statement applies to every release forever. That would turn this gate green at + # the exact moment the assessment was written and before any release carried the fix. To accept a + # triaged advisory, add `--ignore-vuln ` below with the reason in a comment: explicit, + # greppable and per-advisory, the same escape hatch the pip-audit job documents. + if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + # Nothing in this job pushes or uploads the tree, so the persisted git credential has no work + # to do. Fixed at the source rather than leaning on this file's artipacked entry in + # .github/zizmor.yml, which that config's own header asks for. + persist-credentials: false + - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: "3.14" + - name: Install pip-audit (hash-pinned, from the CI toolchain lock) + # Same install as the pip-audit job above: no new tool, no new pin, no second vulnerability + # database, nothing that can drift out of step with an operator-facing command. + run: python -m pip install --require-hashes -r ci/locks/ci-scanners.lock + - name: Audit the latest released line's pinned core runtime + env: + GH_TOKEN: ${{ github.token }} + GH_REPO: ${{ github.repository }} + # Hoisted, never interpolated into the script body (zizmor: template injection). A dispatch + # input is untrusted text and is shape-checked before it reaches `gh`. + TAG_OVERRIDE: ${{ inputs.released_line_audit_tag }} + run: | + set -euo pipefail + + # WHICH TAG. Deliberately NOT `/releases/latest`: that endpoint excludes pre-releases, and + # release.yml publishes pre-release `-*` tags to PRODUCTION PyPI -- so its default answers a + # narrower question than the one asked. The web console is separately versioned under a + # `webconsole-v*` namespace and ships no engine runtime lock, so it is excluded by shape. + # Enumerate, filter, and PRINT the rule applied plus every candidate considered. + if [ -n "${TAG_OVERRIDE:-}" ]; then + case "$TAG_OVERRIDE" in + v[0-9]*.[0-9]*.[0-9]*) targets="$TAG_OVERRIDE" ;; + *) echo "::error::refusing tag override '$TAG_OVERRIDE' -- engine tags are vX.Y.Z[-suffix]" >&2 ; exit 2 ;; + esac + echo "selection rule: OPERATOR OVERRIDE via workflow_dispatch input" + else + releases="$(gh release list --limit 100 --json tagName,isDraft,isPrerelease,publishedAt)" + echo "--- candidates considered ---" + printf '%s' "$releases" | jq -r '.[] | "\(.publishedAt) \(.tagName) draft=\(.isDraft) prerelease=\(.isPrerelease)"' + echo "selection rule: newest non-draft engine tag, PLUS the newest pre-release when it is newer than that" + targets="$(printf '%s' "$releases" | jq -r ' + [ .[] + | select(.isDraft | not) + | select(.tagName | test("^v[0-9]+\\.[0-9]+\\.[0-9]+(-.+)?$")) ] + | sort_by(.publishedAt) | reverse + | ( (map(select(.isPrerelease | not)) | .[0:1]) + (.[0:1] | map(select(.isPrerelease))) ) + | map(.tagName) | unique | .[]')" + fi + + if [ -z "$targets" ]; then + echo "::error::no published engine release matched the selection rule -- this gate examined NOTHING. Failing closed." >&2 + exit 2 + fi + + examined=0 + for tag in $targets; do + echo "=== $tag ===" + # Read the lock AT THE TAG. This is the exact file release.yml installs into a clean venv to + # generate the shipped CycloneDX SBOM, so it is the same component population the published + # messagefoundry-sbom.cdx.json inventories -- not a fresh resolve of main. + gh api "repos/$GH_REPO/contents/docker/locks/requirements-core.lock?ref=$tag" \ + -H "Accept: application/vnd.github.raw" > "core-$tag.lock" + pinned="$(awk '/^[^#[:space:]-]/ && /==/ {n++} END {print n+0}' "core-$tag.lock")" + echo "$tag: docker/locks/requirements-core.lock carries $pinned pinned requirements" + # A run that examined nothing must never read as clean. 41 at the time of writing; this + # floor is a collapse detector, not a pin. + if [ "$pinned" -lt 25 ]; then + echo "::error::$tag: read only $pinned pinned requirements -- the audit input is missing or truncated. Failing closed." >&2 + exit 2 + fi + pip-audit -r "core-$tag.lock" --desc + echo "$tag: CLEAN -- $pinned pinned requirements of the CORE RUNTIME closure, no known advisory" + examined=$((examined + 1)) + done + + echo "released-line-audit: examined $examined release(s):" + printf '%s\n' "$targets" + echo "SCOPE: the CORE RUNTIME closure only (docker/locks/requirements-core.lock), which is" + echo "what the shipped SBOM inventories. Extras (webauthn, dicom, sqlserver) and the CI" + echo "toolchain are covered against main by the pip-audit job, not here. A wheel adopter" + echo "resolves against pyproject.toml's FLOORS, so a finding here is a statement about the" + echo "published SBOM's inventory, not about every install." + - name: What to do about a finding + if: failure() + run: | + set -euo pipefail + echo "The latest released line pins a component with a known advisory. The order matters." + echo + echo "1. TRIAGE FIRST. .github/SECURITY.md -- the clock starts at UPSTREAM-FIX availability," + echo " not at this run. Priority is KEV first, then EPSS >= 0.7, with CVSS only as a" + echo " tiebreaker, weighing reachability. Pick the row from that file's SLA table. A HIGH" + echo " CVSS that is unreachable lands on the LAST row, not the High row." + echo + echo "2. THE REMEDY IS A RELEASE. docs/SUPPORT-POLICY.md -- there is no back-port, and" + echo " adopting a fix is a one-line pin bump, so a patch release on the latest line is the" + echo " only vendor action that changes an adopter's state." + echo + echo "3. IF IT IS UNREACHABLE, record the assessment as an OpenVEX statement in" + echo " security/vex/messagefoundry.openvex.json (security/vex/README.md) and bump the" + echo " document version. It ships with the NEXT release: release.yml copies that file from" + echo " the working tree at tag time, so the VEX commit must PRECEDE the tag." + echo + echo "4. NEVER edit the shipped artifact. The released SBOM and VEX are Sigstore-signed and" + echo " SLSA-attested at tag time. A post-tag assessment reaches consumers via the next" + echo " release, never by amending an old one." + echo + echo "5. NO UPSTREAM FIX YET? .github/SECURITY.md -- apply a documented compensating control" + echo " and track to the fix. To silence a triaged advisory here, add '--ignore-vuln '" + echo " to the pip-audit call in the previous step, with the reason." + sbom: name: SBOMs (CycloneDX, multi-ecosystem) runs-on: ubuntu-latest diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index 8c32375a..f3ce9e13 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -5740,3 +5740,32 @@ environments.py:79 `if not base_dir: return cwd` <-- and base_dir i **Related:** #1063 (the same script, anchoring rather than reporting), #1000 (a gate whose green does not say what it was green about). **Source:** observed 2026-08-06 while arming a fresh worktree during #1063's fix. Held unfiled as marginal, and filed on the owner's instruction. + +## 1081. released-line audit: detect an advisory against the latest release's pinned runtime + +> ✅ **Shipped 2026-08-07.** `released-line-audit` in `.github/workflows/security.yml` audits the **latest release tag's** `docker/locks/requirements-core.lock` on the existing daily cron, plus `workflow_dispatch` with a tag override. Advisory by placement (schedule/dispatch-only, so it can never report on a PR) but **not** `continue-on-error`: it goes red on a finding. `nightly-notice.yml` was extended to watch `Security` so a red scheduled run reports somewhere. + +**Cluster:** Supply chain / CI. **Priority:** P3. **Verdict:** built, reduced. **Severity:** low — there are zero deployments, so this closes a window before anyone is in it. + +**The gap, stated correctly — and it is NOT the one first claimed.** The original framing was *"nothing re-evaluates a published VEX against advisories disclosed after its tag"*, offered with CVE-2026-69247 as evidence. That framing is **wrong and was retracted**. The advisory was caught the day it published, by an existing required gate: commit `ac87246f` records *"pip-audit (a required gate) flagged cryptography 49.0.0 for CVE-2026-69247"*. Nor was `main` ahead of the tag — `git show ac87246f^:docker/locks/requirements-core.lock` and `git show v0.3.2:docker/locks/requirements-core.lock` both read `cryptography==49.0.0`. Detection was never missing. + +What was unwatched is the **release-lag window**: the interval between a fix landing on `main` and a release carrying it. `pip-audit` reads the checked-out tree, so on the daily cron it answers *"is what we would ship next current"*. That is a different question from *"does the version we already shipped carry a known advisory"*, and the two answers diverge for exactly the length of that window. + +**Residual scope, so nobody over-reads a red run.** This audits the **core runtime closure only** (`requirements-core.lock`), which is what the shipped CycloneDX SBOM inventories. A wheel adopter resolves against `pyproject.toml`'s floors (`cryptography>=48.0.1`), and no container image is published at release, so a finding here is a statement about **the published SBOM's inventory**, not about every install. Extras and the CI toolchain stay covered against `main` by the `pip-audit` job. + +**Pre-merge self-tests (all four passed; the instrument was proven able to see the class).** Against `v0.3.2`'s lock `pip-audit` exits 1 naming `PYSEC-2026-3552` on `cryptography 49.0.0`; against `origin/main`'s lock it exits 0 — so it distinguishes the two states rather than only ever reddening. An empty lock reads 0 pinned requirements against a floor of 25, hitting the fail-closed path. The tag selector returns exactly `v0.3.2` and excludes `webconsole-v0.2.15`. The positive control is durable: the vulnerable lock lives in git history, so `workflow_dispatch` with `released_line_audit_tag=v0.3.2` re-arms it forever. + +**Deliberately NOT built, with reasons — this is the part worth not re-litigating.** + +1. **Scanning the published `messagefoundry-sbom.cdx.json` asset with trivy.** The SBOM is generated by installing the core lock into a clean venv, so the lock **is** the population. Scanning the asset answers the same question plus *"did the generator inventory it correctly"* — a real but different defect — at the cost of a second pinned scanner, a second vulnerability database, and divergence risk against the operator-facing command in `docs/SUPPLY-CHAIN.md`. +2. **Applying any VEX to this gate.** Refuted during design and the reason is subtle: `security/vex/README.md`'s own worked example names the product with **no version qualifier**, so a `fixed` or `not_affected` statement written on `main` would suppress the finding against the already-shipped release. The gate would turn green the moment the assessment was written, before any release carried the fix. `--ignore-vuln ` is the escape hatch — explicit, per-advisory, greppable. +3. **A merge-blocking VEX linter.** As specified it would reject `security/vex/README.md`'s own example and mandate a non-OpenVEX field in a document shipped to hospital scanners. There are zero statements today, so there is nothing to lint. +4. **A release-time VEX version-bump gate.** Real (nothing enforces the documented bump), but with no statements at `version: 1` across every release so far there is no violation and no way to exercise the failing shape. +5. **An in-job issue filer.** Two notifiers for one failure. Extending `nightly-notice.yml` covers every scheduled `Security` job, not only this one. +6. **A `release: published` trigger.** `release.yml` creates the release and uploads assets in one call, so a `published`-triggered run can race the upload. The daily cron bounds detection at ~24h. + +**Also deferred:** the `docs/SUPPLY-CHAIN.md` half of this change — a sentence scoping *"continuously audited by pip-audit"* to `main`'s lockfiles, and the `releases/latest/download/...` permanent fetch URLs. Held back only because PR #264 edits the same file and stacking the two would risk a conflict; land it once #264 merges. + +**Related:** #1079 (the same workflow's header denying a trigger its `on:` block declares), ADR 0149 (the SBOM/VEX program this sits beside — unchanged, and it needs no amendment). + +**Source:** found 2026-08-06 while auditing the shipped v0.3.2 release assets. The original design was refuted 3 of 3 by adversarial review and rebuilt at roughly one tenth the size; the retained design record is in the vault. diff --git a/docs/CI.md b/docs/CI.md index 5fc95d04..f1584087 100644 --- a/docs/CI.md +++ b/docs/CI.md @@ -17,7 +17,7 @@ claims move with it. | Workflow | What it does | |---|---| | `ci.yml` | Lint (`ruff check` + `ruff format --check`), types (`mypy --strict`, plus a `--platform win32` pass on Linux so Windows type-branches are checked), and the `pytest` suite across **ubuntu-latest**, **windows-2022**, and **windows-2025** (Python 3.14). Also builds the VS Code extension (`ide/`). A `CI gate` job rolls the legs up. | -| `security.yml` | Static and supply-chain security: `bandit` (Python SAST), `semgrep`, `pip-audit` and `npm-audit` against the hash-locked tree, `gitleaks` (secret scan), `forbidden-content` (customer/PHI leak guard), a crypto-inventory check, an SBOM build, and a `trivy` scan. A **daily cron** re-runs the dependency audits so a CVE filed against an unchanged pin is caught within ~24h. | +| `security.yml` | Static and supply-chain security: `bandit` (Python SAST), `semgrep`, `pip-audit` and `npm-audit` against the hash-locked tree, `gitleaks` (secret scan), `forbidden-content` (customer/PHI leak guard), a crypto-inventory check, an SBOM build, and a `trivy` scan. A **daily cron** re-runs the dependency audits so a CVE filed against an unchanged pin is caught within ~24h. A separate `released-line-audit` job runs on the same cron and audits the **latest release tag's** pinned core runtime, which the daily audits do not cover — they read the checked-out tree, so between a fix landing on `main` and a release carrying it the two answers differ. Hard-failing but **not** a required check (schedule/dispatch only), the same posture as `dast.yml`. | | `codeql.yml` | GitHub CodeQL analysis (python / javascript-typescript). | | `scorecard.yml` | OpenSSF Scorecard analysis. | | `cla.yml` | CLA Assistant — records the Contributor License Agreement signature on each PR. | @@ -51,9 +51,11 @@ The stable contexts required on `main` are — mirroring That last string is the **job key** in `cla.yml`, whose job declares no `name:`. Branch protection matches the job name, never the workflow name — so the context is `cla`, not "CLA Assistant". Every -non-advisory job in `security.yml` is in the set; the two that are not (`sbom`, `trivy`) declare -`continue-on-error: true`, and `tests/test_security_posture.py` pins which side of that line each one -is on. +non-advisory job in `security.yml` is in the set. Three are not, in two different ways: `sbom` and +`trivy` declare `continue-on-error: true`, while `released-line-audit` deliberately does **not** — it +is advisory by *placement*, being schedule/dispatch-only so it can never report on a PR, and it still +goes red on a finding (the `dast.yml` posture). `tests/test_security_posture.py` pins which of the +three buckets each job is in. CodeQL is **advisory** (not in the required set) — its SARIF upload needs `security-events: write`, which fork-PR tokens do not have, so requiring it would block PRs from forks. Scorecard is advisory for diff --git a/tests/test_nightly_notice.py b/tests/test_nightly_notice.py index 210ecbe6..8c19fe15 100644 --- a/tests/test_nightly_notice.py +++ b/tests/test_nightly_notice.py @@ -59,6 +59,17 @@ def test_it_keys_on_the_ci_workflow_s_actual_name() -> None: ) +def test_it_also_watches_the_security_workflow() -> None: + """security.yml's daily cron carries jobs no PR can trigger, so its failures need this notice too.""" + watched = _on(_load(_NOTICE))["workflow_run"]["workflows"] + sec_name = _load(_WORKFLOWS / "security.yml").get("name") + assert sec_name, "security.yml has no `name:` — workflow_run has nothing to key on" + assert sec_name in watched, ( + f"nightly-notice.yml watches {watched} but security.yml is named {sec_name!r}. Its " + "schedule-only jobs — released-line-audit above all — would then fail into silence." + ) + + def test_it_only_reacts_to_scheduled_runs() -> None: """Without this gate every PR and push failure opens an issue. diff --git a/tests/test_security_posture.py b/tests/test_security_posture.py index 0dc4e8ee..8a983932 100644 --- a/tests/test_security_posture.py +++ b/tests/test_security_posture.py @@ -64,6 +64,17 @@ # without also removing its `if:` would wedge every PR (see security.yml's own notes on trivy). _ADVISORY_SECURITY_JOBS = frozenset({"sbom", "trivy"}) +# ADVISORY BY PLACEMENT: hard-failing, but NOT in branch protection. This is a third posture the file +# previously could not express, and it is not new to the repo -- `dast.yml` already ships it and +# .github/required-contexts.txt records it: advisory by placement, not by continue-on-error, so the job +# goes red on a finding. +# +# The distinguishing rule, and the reason this is a separate list rather than an entry in +# _ADVISORY_SECURITY_JOBS: these jobs MUST NOT carry continue-on-error. A job that can never report on +# a PR cannot be required, but that is a reason to keep it out of branch protection -- not a reason to +# discard its findings. +_ADVISORY_BY_PLACEMENT_SECURITY_JOBS = frozenset({"released-line-audit"}) + # Job-level `if:` expressions that CANNOT skip the job on a pull_request, with the reason each is safe. # Anything else on a required job is a way for the context to silently not report. _JOB_IF_ALLOWLIST = { @@ -133,11 +144,14 @@ def test_every_security_job_is_classified() -> None: quietly stop covering the file it is named for. """ actual = set(jobs_of(_SECURITY)) - classified = _BLOCKING_SECURITY_JOBS | _ADVISORY_SECURITY_JOBS + classified = ( + _BLOCKING_SECURITY_JOBS | _ADVISORY_SECURITY_JOBS | _ADVISORY_BY_PLACEMENT_SECURITY_JOBS + ) print(f"[security-posture] classified {len(classified)} of {len(actual)} jobs in {_SECURITY}") assert actual == classified, ( f"security.yml jobs are not all classified.\n" - f" unclassified (add to _BLOCKING_SECURITY_JOBS or _ADVISORY_SECURITY_JOBS): " + f" unclassified (add to _BLOCKING_SECURITY_JOBS, _ADVISORY_SECURITY_JOBS or " + f"_ADVISORY_BY_PLACEMENT_SECURITY_JOBS): " f"{sorted(actual - classified)}\n" f" named here but gone from the workflow: {sorted(classified - actual)}" ) @@ -174,6 +188,51 @@ def test_advisory_security_jobs_are_not_required() -> None: ) +def test_advisory_by_placement_jobs_are_not_required() -> None: + """Placement is the whole mechanism: these are kept off the merge path by not being required.""" + required = set(required_contexts()) + jobs = jobs_of(_SECURITY) + promoted = sorted( + context_of(k, jobs[k]) + for k in _ADVISORY_BY_PLACEMENT_SECURITY_JOBS + if context_of(k, jobs[k]) in required + ) + assert not promoted, ( + f"{promoted} is schedule/dispatch-gated but present in the required set. It can never report " + "on a PR, so requiring it blocks every merge forever (the required-but-absent trap). Remove " + "the job-level `if:` first if promotion is genuinely intended." + ) + + +def test_advisory_by_placement_jobs_carry_no_continue_on_error() -> None: + """The point of this bucket. Off the merge path is NOT the same as findings discarded.""" + jobs = jobs_of(_SECURITY) + for key in sorted(_ADVISORY_BY_PLACEMENT_SECURITY_JOBS): + job = jobs[key] + assert job.get("continue-on-error") in (None, False), ( + f"security.yml job {key!r} is advisory BY PLACEMENT and must still go red on a finding; " + "it now declares continue-on-error, which discards them. It is already outside branch " + "protection, so there is nothing continue-on-error can protect here." + ) + for step in job.get("steps") or []: + name = (step or {}).get("name") or (step or {}).get("uses") or "" + assert (step or {}).get("continue-on-error") in (None, False), ( + f"security.yml job {key!r}, step {name!r} declares continue-on-error" + ) + + +def test_advisory_by_placement_jobs_cannot_run_on_a_pull_request() -> None: + """If one of these could report on a PR it would be requirable, and this bucket would be a lie.""" + jobs = jobs_of(_SECURITY) + for key in sorted(_ADVISORY_BY_PLACEMENT_SECURITY_JOBS): + expr = str(jobs[key].get("if") or "") + assert "schedule" in expr and "workflow_dispatch" in expr and "pull_request" not in expr, ( + f"security.yml job {key!r} is classified advisory-by-placement, which asserts it never " + f"reports on a PR, but its `if:` is {expr!r}. Either restore the schedule/dispatch gate or " + "reclassify it and add its context to .github/required-contexts.txt and branch protection." + ) + + def test_advisory_security_jobs_keep_continue_on_error() -> None: """The mirror of the blocking assertion: an accidental promotion must also be a deliberate edit.""" jobs = jobs_of(_SECURITY) From f3a9f6679e1dd084f71bbb010deee61b916b84ec Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Fri, 7 Aug 2026 00:43:04 -0500 Subject: [PATCH 2/2] fix(tests): released-line-audit is a third ci-scanners.lock install site (BACKLOG #1081) `test_lock_installed_toolchain_install_is_hash_verified` pins the EXACT number of `pip install --require-hashes -r ci/locks/ci-scanners.lock` sites per workflow, not just "at least one". 73dccc31 added a third to security.yml (the released-line-audit toolchain install) and left the constant at 2, so the test failed -- correctly. Verified the count directly rather than trusting the message: sites are at security.yml lines 116, 219 and 483. The exactness is the point, and the comment above the constant says why: a non-vacuity check satisfied by one surviving line lets either half of a pair be deleted while every check stays green, and the coverage job's failure mode is silent (`|| true` swallows the missing plugin). So the fix is the number, not the assertion. Updated that comment too. It enumerated the sites as "twice (the pip-audit step and the bandit step)"; bumping the constant while leaving the prose saying twice would be the same code-versus-prose drift BACKLOG #1081 exists to close. Caught by the coordinator on PR #267, not locally: the pre-commit run for 73dccc31 used a deliberately scoped `-k` selection that did not include this file. A scoped selection cannot report the absence of a failure outside its scope, and its green reads identically to the suite's green. --- tests/test_ci_venv_pinning.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/test_ci_venv_pinning.py b/tests/test_ci_venv_pinning.py index 34675f86..2f081925 100644 --- a/tests/test_ci_venv_pinning.py +++ b/tests/test_ci_venv_pinning.py @@ -255,7 +255,8 @@ def test_scratch_venvs_do_not_hide_an_unpinned_pip_fetch(workflow: str) -> None: #: not being looked at. The tests below are the ones that look. #: #: THE COUNT IS LOAD-BEARING, not decoration. Five install sites collapse onto three (workflow, lock) -#: pairs — `security.yml` installs `ci-scanners.lock` twice (the pip-audit step and the bandit step) and +#: pairs — `security.yml` installs `ci-scanners.lock` three times (the pip-audit step, the bandit step +#: and released-line-audit, which audits the latest release tag's core lock rather than the tree) and #: `quality-advisory.yml` installs `ci-quality.lock` twice (the coverage job and the mutation job). An #: `assert lines` non-vacuity check is satisfied by ONE surviving line, so deleting either of a pair #: leaves its job installing nothing while every check here stays green. Measured: with only "≥1", four @@ -263,7 +264,7 @@ def test_scratch_venvs_do_not_hide_an_unpinned_pip_fetch(workflow: str) -> None: #: is silent (`pytest -q --cov` dies on `unrecognized arguments`, `|| true` swallows it, and the #: diff-coverage step reports "skipped" and exits 0). LOCK_INSTALLED_TOOLCHAINS = ( - ("security.yml", "ci/locks/ci-scanners.lock", 2), + ("security.yml", "ci/locks/ci-scanners.lock", 3), ("zizmor.yml", "ci/locks/ci-scanners.lock", 1), ("quality-advisory.yml", "ci/locks/ci-quality.lock", 2), )