ci(security): audit the latest release line's pinned runtime on the daily cron (BACKLOG #1081) - #267
Merged
Merged
Conversation
…daily audits do not cover (BACKLOG #1081) The `pip-audit` gate audits the CHECKED-OUT tree, so on the daily cron it audits `main`. "Is what we would ship next current" is a different question from "does the version we already shipped carry a known advisory", and the two answers diverge from the moment a fix lands on main until a release carries it. Nothing watched that window. `released-line-audit` does, on the existing daily cron. WHAT THIS IS NOT. The framing that motivated this -- "advisories disclosed after a tag are never re-evaluated" -- was WRONG and is retracted in #1081. CVE-2026-69247 was caught the day it published, by an existing required gate: commit ac87246 records "pip-audit (a required gate) flagged cryptography 49.0.0 for CVE-2026-69247". Nor was main ahead of the tag; both read 49.0.0 at that moment. Detection was never missing. Only the release-lag window was unwatched. THE INSTRUMENT WAS PROVEN BEFORE THE GATE WAS WRITTEN, because a gate nobody proved can see its own defect class is the failure this repo keeps re-learning. Against v0.3.2's core lock `pip-audit` exits 1 naming PYSEC-2026-3552 on cryptography 49.0.0; against origin/main's it exits 0 -- so it distinguishes the two states rather than only ever reddening, which is the false-deny direction no red test surfaces. An empty lock reads 0 pinned requirements against a floor of 25 and takes the fail-closed path. The tag selector returns exactly v0.3.2 and excludes webconsole-v0.2.15. The positive control is durable: the vulnerable lock is in git history, so a dispatch with released_line_audit_tag=v0.3.2 re-arms it forever. NO VEX IS APPLIED TO THIS GATE, and that is the load-bearing design decision. security/vex/README.md's own worked example names the product with no version qualifier, so a `fixed` or `not_affected` statement written on main would suppress the finding against the ALREADY-SHIPPED release -- turning the gate green at the exact moment the assessment was written and before any release carried the fix. `--ignore-vuln <ID>` is the escape hatch instead: explicit, per-advisory, greppable. Advisory by PLACEMENT, not by continue-on-error -- the dast.yml posture. It is schedule/dispatch-only so it can never report on a PR and must never be required (the required-but-absent trap), but it still goes red on a finding. tests/test_security_posture.py could not express that: it had two buckets and assumed anything not required carries continue-on-error. It now has three, and the new bucket asserts the opposite -- no continue-on-error, not required, and never reachable from a pull_request. nightly-notice.yml now watches `Security` as well as `CI`, because a red schedule-only job otherwise reports nowhere. Its issue title is DERIVED from the completed workflow's name rather than hardcoded, so a green nightly CI cannot close an issue a red Security run opened; for CI the title is byte-identical to today's, so any currently-open issue stays matched. Also corrects docs/CI.md, which said the only non-required security.yml jobs are `sbom` and `trivy` and that both declare continue-on-error. Three jobs now sit outside the required set, in two different ways, and the new one deliberately declares no continue-on-error. Deferred deliberately: the docs/SUPPLY-CHAIN.md half -- scoping "continuously audited by pip-audit" to main's lockfiles, and the releases/latest/download permanent fetch URLs. PR #264 edits that same file; stacking the two would risk a conflict. #1081 records it, along with at least five other things left unbuilt and why. Self-tests A-D above were run pre-merge. E-G (live dispatch, notifier, recovery) need this on main and are listed in #1081.
wshallwshall
enabled auto-merge (squash)
August 7, 2026 05:22
…ite (BACKLOG #1081) `test_lock_installed_toolchain_install_is_hash_verified` pins the EXACT number of `pip install --require-hashes -r ci/locks/ci-scanners.lock` sites per workflow, not just "at least one". 73dccc3 added a third to security.yml (the released-line-audit toolchain install) and left the constant at 2, so the test failed -- correctly. Verified the count directly rather than trusting the message: sites are at security.yml lines 116, 219 and 483. The exactness is the point, and the comment above the constant says why: a non-vacuity check satisfied by one surviving line lets either half of a pair be deleted while every check stays green, and the coverage job's failure mode is silent (`|| true` swallows the missing plugin). So the fix is the number, not the assertion. Updated that comment too. It enumerated the sites as "twice (the pip-audit step and the bandit step)"; bumping the constant while leaving the prose saying twice would be the same code-versus-prose drift BACKLOG #1081 exists to close. Caught by the coordinator on PR #267, not locally: the pre-commit run for 73dccc3 used a deliberately scoped `-k` selection that did not include this file. A scoped selection cannot report the absence of a failure outside its scope, and its green reads identically to the suite's green.
…audit # Conflicts: # docs/BACKLOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
released-line-auditjob to security.yml's existing daily cron. It audits the LATEST RELEASETAG's docker/locks/requirements-core.lock.
The required
pip-auditjob audits the checked-out tree, so on a schedule it auditsmain. Thiscovers the different window: the release lag between a fix landing on main and a release carrying it.
Retraction carried in the commit and in #1081
An earlier framing of mine -- "advisories after a tag are never re-evaluated" -- was WRONG.
CVE-2026-69247 was caught same-day by the required
pip-auditgate (ac87246f). This job is not asecond detector for that case. It covers the release-lag window only.
Deliberately NOT a required context
released-line-auditis schedule/dispatch only, so it can never report on a PR -- making it requiredwould be the required-but-absent trap. It is advisory by placement, not by continue-on-error: it
goes red on a finding, and
tests/test_security_posture.pyholds that property.It also could not be actioned by a PR author: the remedy is cutting a release on the latest line
(docs/SUPPORT-POLICY.md), not editing the branch under review. The only change to
.github/required-contexts.txtis the comment recording this reasoning -- no context is added, andthe live required set stays at 13.
Pre-merge self-tests, all four run and passed
pip-audit -ragainst the v0.3.2 core lockpip-audit -ragainst the origin/main lockB is the one that matters: a gate that can only go red is not a gate.
Local verification
ruff check + format clean; 179 passed / 8 skipped across the workflow/backlog/ledger/posture
selection;
ledger_check --ciclean;backlog_status_checkOK; full pre-commit set passed includingactionlint. No
--no-verify. mypy ontests/shows only pre-existing dict type-arg errors alsopresent on origin/main (the gate's scope is
mypy messagefoundry).Not done, needs main
Self-tests E-G: dispatch security.yml on main and confirm
released-line-auditgoes RED on v0.3.2printing the candidate list, selection rule, 41 pinned requirements and the advisory; confirm the
06:00 UTC cron opens an issue titled "Nightly Security is failing"; and after a release carrying
cryptography 50.0.0, confirm it goes GREEN and the issue closes.
If the notice never appears, the gate is reporting into the void. That is the one thing worth
chasing post-merge.
A follow-up to docs/SUPPLY-CHAIN.md is deferred because #264 edits that file; #1081 records the
deferred half.