SOUP: dependency discovery, SBOM per build, and daily monitoring of the deployed version

[grafele]

Third-party components — SBOM, vulnerability monitoring and the Work Instruction that governs it.
Covers DEV-190, DEV-191, DEV-192, DEV-195, DEV-196, DEV-197.

Counterpart PR: QuickBirdEng/workflows#54 adds the three reusable workflows that call
these actions. That one should merge first — a product caller referencing
QuickBirdEng/workflows/.github/workflows/soup-sbom.yml@main fails until the file is on main.

What this does

Four points in time, each with its own record:

	Trigger	Question	Record
#0–#2	Pull request	Is every vulnerability of this version fixed or dispositioned?	SOUP record, blocks the change if not
#3	Tagged build	What is in this build, and is it assessed?	The inventory, attached to the release
#4	Daily	Is the version in production affected today?	A dated record, written even when nothing is found
#7	Quarterly or annually	What did not happen?	Reconciliation report

Stage #4 assesses what is actually deployed, resolved from the GitHub deployment record — not the
newest tag and not the newest release, because neither states what is running.

Three things it deliberately refuses to do

• Scan an artefact nobody classified. Discovery finds candidates; every one must appear in
  .soup-scope.yml as included or excluded, each with a reason. An unclassified candidate stops the
  run, because "not listed" is otherwise indistinguishable from "deliberately excluded".
• Report success on an empty result. A run that could not complete says so. This was a real
  defect twice: a select() inside a jq object constructor made the whole construction empty, and
  the action wrote a 0-byte record while reporting all-clear.
• Guess a required parameter. tier, cra_scope and maintenance_interval come from the
  customer SLA and have no safe default. cra_scope defaults to unknown rather than false,
  because an unnecessary report costs less than an omitted one.

Where the rules live

Not in this repository. The Work Instruction and its two annexes live in the QMS, which is where a
controlled document has a review workflow and a single version. policy-defaults.yml here is its
machine-readable counterpart: the document is what a person reads and what an auditor is shown, this is
what the tooling applies. A change to one is a change to the other and belongs in the same review.
Section references in comments (WI §5, Annex B B.1.1) point at the document.

Two places where BSI TR-03161 is stricter than the process default, enforced rather than documented:
O.TrdP_2 requires the newest version or the one preceding it, so a product in TR-03161 scope must
state a patch limit; O.TrdP_8 is a prohibition, so accepting obsolescence with a recorded reason is
not available to it. A configuration that violates either is refused by validate-policy.sh.

Nothing here is product-specific

No ticket number and no product name in any comment, README or example — tests excepted, where a
regression test names what it reproduces. Measured facts stay, because they are the reason the values
are what they are; only the attribution goes.

Per-product files are not in this repository either: the six scope and policy drafts and the one
deploy-workflow patch go to the product repositories, which is where the WI says a product's
configuration lives. examples/ holds three generic templates instead — a policy, a policy showing a
justified relaxation, and a scope declaration whose entries are the cases worth showing: an image in
scope in one artefact and out of scope in another, our own release-versioned image, a builder stage,
infrastructure.

Measured against real repositories

Everything below is from actual runs, not from examples. On one backend product:

• 522 findings resolve through 3 actions. 492 of them are RPM packages inside a single image; none
  is in code written here. Attaching the deadline to the finding produced 522 escalations for 3
  decisions, which is why timeframes attach to the remediation unit instead.
• 0 of 23 Critical findings were listed in KEV. A shared 72-hour deadline was applying the urgency
  of observed exploitation to findings that carried none. KEV is now its own track; the other bands
  got timeframes that can be met.
• Excluding OS packages changed "currency unknown" from 343 of 386 to 43. The image is the SOUP;
  its packages move when it does.
• Maintenance windows: two products hold, one has missed 1, one has missed 3, one cannot be
  determined because its deployment workflow declares no environment.

Seven defects in my own implementation were found by running it rather than by reading it. They are
listed in Annex A §A.10 and each has a regression test.

Tests

soup-discovery/tests/run-tests.sh — 139 cases, 5 of which hit live feeds and are skipped without
TEST_NETWORK=1. Most are regressions for defects found while building this.

Suggested reading order

1. WI-SOUP-Vulnerability-Management.html §1–§4 — what the process is and what each project decides.
2. soup-discovery/policy-defaults.yml — the same decisions in machine-readable form, with the reason
   for each value in comments.
3. patches/README.md — which file goes into which repository, and the order to wire a product.
4. soup-discovery/scripts/classify-findings.py and group-remediation.py — the two places where a
   wrong rule would be hardest to notice.

Not in this PR

• No product is wired. The callers are staged under patches/product-repo/ as templates. Nothing
  runs on a schedule yet, which also means no timeframe is running: both clocks start on the date a
  run first reports a finding.
• The scope declarations are drafts derived from reading the repositories, and they are not in this
  PR — they belong in the product repositories. Each needs confirmation by its project before the
  product is released under this document.
• Object store credentials are not distributed. Without them a run keeps its records as 90-day
  workflow artefacts and the backstop cannot reconcile a longer period.
• The retention period is a QMS determination and is not yet stated. It decides whether the object
  store alone is sufficient, since it has no object lock.

One thing worth a second opinion

The maintenance window model replaced a version that derived the deadline from each product's
observed release rhythm. That did not work: three of four products had a rhythm that had already
lapsed, so a finding was born overdue — one product's would have been 109 days late on the day it was
found. A product now declares a commitment instead, and the deadline is the next window on that grid.
The commitment is capped by the SLA tier and the cap cannot be waived. WI §6.2 and Annex A §A.3.

[grafele]

A full code review of this branch ran on 2026-08-05; everything it found is fixed in 7e03df7, each finding with a regression test. Summary for review focus:

Three critical defects, all in the layer that had no tests (the seam to deployment records and release assets):

1. resolve-deployed.sh passed --arg through gh api --jq, which takes one expression and forwards nothing — gh refused the call, || echo "" swallowed it, and every deployed tag reported "no SBOM asset". The daily monitor could never have resolved a real inventory. Now piped to jq, and a fake gh in the test suite makes this path testable at all.
2. monitor-kev.sh filtered targets on /prod/ while the backstop counts prod|study. A Study-only product fell out of both the targets and the unscannable list — the record said all_clear with nothing scanned. Filter aligned; all_clear now also requires that at least one target was scanned.
3. The reusable soup-sbom.yml never passed release-tag to the action, so the entire workflow_run path (the normal trigger) produced branch-tier bundles that were never attached to a release. Fixed in SOUP: three reusable workflows for third-party component monitoring workflows#54 (85227d3).

High: the TR-03161 patch limit was validated but never measured (check-currency.py compared major/minor only); scan-vulns.sh exited 0 on incomplete OSV results and dropped findings whose advisory fetch failed (now: retries, querybatch pagination for >1000-vuln packages, unfetchable advisories carried unscored, incompleteness is exit 1); the monitor kept one state file for all targets, restarting the second target's clocks daily (state/lifecycle/escalation now per target); a VEX not_affected recorded for package A suppressed the same CVE on package B (analysis now applies only when every affected component is covered by its own record), and justification codes are validated against the CycloneDX vocabulary — an invalid code was a working mute button.

Also fixed: units never listed their KEV members (string compared with is True — the old test fixture had encoded the bug); CVSS-4-only advisories fell to "unscored" regardless of severity; the onboarding baseline missed the whole backlog when the first scan ran on the onboarding day; stale hardcoded deadlines in the Slack alert; discovery id collisions across equal basenames (ids are path-based now — the six per-product scope drafts were re-verified against their live repos; apellis surfaced one genuinely new candidate, its digest-pinned GPU inference image); unknown policy keys are refused; action inputs reach scripts via env instead of ${{ }} interpolation; a failed Slack post fails the step.

Suite: 135 → 154 tests, 5 against live feeds, shellcheck and pyflakes clean.

[grafele]

End-to-end verified on a real runner — six iterations on a temporary branch (mindnet@soup/e2e-test, workflow runs only there, touches no releases or stores; delete after merge). Both composite actions pinned to this PR branch. Final state: both jobs green, and the monitor record on the runner is byte-for-byte what the design says it should be:

"verdict": "incomplete",
"not_scanned": [
  {"name": "Production", "version": "v1.0.15", "why": "release exists but carries no sbom-v1.0.15.cdx.json — released before the SBOM pipeline, …"},
  {"name": "mobile",     "version": "v1.0.15", "why": "production release carries no SBOM asset"}
]

That is the honest pre-adoption answer; it becomes an all-clear the moment a release carries its SBOM asset. The SBOM job assessed the full repository on the runner: 34 candidates, scope gate 20/14/0, 5442 components, 1725 findings → 118 remediation units, 0 in KEV, three grq-4 contradictions surfaced against real SOUP records.

The iterations found five defects no local test could reach, all fixed on this branch with regression tests (159 total now):

1. 35bb128 — the action never passed a policy to the pipeline, so no CI bundle had ever been assessed. Local runs always passed it by hand.
2. The runner lacked gh; the prerequisite check caught it loudly (runner provisioning is the durable fix; the e2e bootstraps a pinned binary meanwhile).
3. 3fa73bd — with contents: read only, the per-environment deployments fetch 403s; the old || continue silently dropped Production. Environments the token cannot read are now named gaps, and the callers grant deployments: read.
4. 670bb47 — gh api prints the HTTP error body to stdout; captured unchecked, {"message":"Resource not accessible…"} became a phantom environment name and every real environment vanished errorlessly. Error bodies are never accepted as data now, and the env names fall back to the deployment records themselves.
5. 8e44649 — Linux caps a single process argument at 128KB (MAX_ARG_STRLEN); folding deployment pages via --argjson blew it, jq died, and the empty substitution emptied every downstream list. macOS has a larger limit, which is why 150+ local tests never saw it. Pages accumulate in files now, and a ~400KB fixture reproduces the failure on any Linux machine the suite runs on.

Pattern across all five: the wrong answer looked exactly like a right one. Every fix converts silence into a named reason.