Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/linters/kernel-findings.rego
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
package trivy

import rego.v1

default ignore := false

# Kernel packages never execute inside a container - only the host's kernel runs -
# so vulnerabilities against them can't be exploited here, regardless of CVE ID.
ignore if {
startswith(input.PkgName, "linux-")
}
23 changes: 21 additions & 2 deletions .github/workflows/vulnerability-scan.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,18 +14,37 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
flavor: [cpp, docs, embedded-cpp, embedded-rust, rust]
include:
- flavor: cpp
dockerfile: .devcontainer/cpp/Dockerfile
- flavor: docs
dockerfile: .devcontainer/docs/Dockerfile
- flavor: embedded-cpp
dockerfile: .devcontainer/cpp/Dockerfile
- flavor: embedded-rust
dockerfile: .devcontainer/rust/Dockerfile
- flavor: rust
dockerfile: .devcontainer/rust/Dockerfile
permissions:
contents: read # is needed by actions/checkout
security-events: write # is needed by github/codeql-action/upload-sarif to upload sarif files
steps:
- uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
with:
egress-policy: audit
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: crazy-max/ghaction-container-scan@ffcba8deb5cb3531954cbedbbcd0179d79edf510 # v4.1.0
id: scan
env:
# No direct input for this; trivy binds every CLI flag to a TRIVY_* env var
TRIVY_IGNORE_POLICY: .github/linters/kernel-findings.rego
with:
# Unfixed findings (no upstream patch yet) aren't actionable and would just re-appear next scan
ignore_unfixed: true
image: ghcr.io/${{ github.repository }}-${{ matrix.flavor }}:latest
dockerfile: .devcontainer/Dockerfile
dockerfile: ${{ matrix.dockerfile }}
- uses: github/codeql-action/upload-sarif@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4.37.8
if: steps.scan.outputs.sarif != ''
with:
Expand Down
2 changes: 1 addition & 1 deletion .mega-linter.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,4 @@ FILTER_REGEX_EXCLUDE: (CHANGELOG.md|package-lock.json)
# dynamically based upon context (e.g. installed extensions)
# hence the exclusion
JSON_V8R_FILTER_REGEX_EXCLUDE: (\.vscode)
REPOSITORY_TRIVY_ARGUMENTS: --ignorefile .github/linters/.trivyignore.yml
REPOSITORY_TRIVY_ARGUMENTS: --ignorefile .github/linters/.trivyignore.yml --ignore-unfixed --ignore-policy .github/linters/kernel-findings.rego
77 changes: 77 additions & 0 deletions docs/adr/0001-vulnerability-scanning-noise-reduction.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Reduce Vulnerability Scanning Noise Instead of Requiring It to Block Merges

* Status: accepted
* Deciders: Development Team
* Date: 2026-09-05

## Context and Problem Statement

Container images are scanned for vulnerabilities by two independent Trivy invocations: [vulnerability-scan.yml](../../.github/workflows/vulnerability-scan.yml) (`crazy-max/ghaction-container-scan`, scanning each built flavor image daily) and MegaLinter's `REPOSITORY_TRIVY` linter (scanning the repository filesystem on every pull request), alongside Grype, Checkov, Hadolint, secretlint, and betterleaks, all uploading SARIF to GitHub code scanning.

An audit of the repository's code-scanning alerts found 9,590 open Trivy alerts (169 unique CVEs after de-duplicating across flavors), the oldest dating back to 2024-05-15 — over two years unaddressed. Two structural causes were identified:

* The overwhelming majority of findings have no upstream fix available yet (unfixed OS-package CVEs), so they are not currently actionable, yet they were reported identically to fixable ones.
* A visible share of the remaining findings were kernel CVEs (e.g. `CVE-2026-74394`, `CVE-2026-72287` against `linux-libc-dev`) that cannot be exploited inside a container, since containers share the host kernel and never execute their own.

Neither of these two categories can be resolved by changing our own package pins, and manually triaging or dismissing thousands of individual alerts is not a sustainable use of maintainer time. At the same time, supply-chain security is a top priority for this project, so any noise-reduction approach must not risk hiding a real, fixable vulnerability.

## Decision Drivers

* **Actionability:** findings should represent something a maintainer can actually act on.
* **Low ongoing effort:** the approach must not require manually re-triaging a large, ever-growing backlog.
* **No loss of real signal:** suppression must be based on documented, auditable criteria — never a blanket "ignore everything" or an undocumented manual dismissal.
* **Reversibility:** anyone should be able to see exactly what is excluded and why, and reconsider it later.

## Considered Options

* **Leave scanning output as-is:** keep reporting every finding, unfiltered.
* **Disable or stop running one or more scanners:** remove the source of the noise entirely.
* **Filter to actionable, in-scope findings:** use Trivy's built-in filtering (`--ignore-unfixed`, a Rego `--ignore-policy`) to suppress specific, documented categories of non-actionable findings, while leaving everything else visible.

## Decision Outcome

Chosen option: **filter to actionable, in-scope findings**, applied identically to both Trivy invocations ([.mega-linter.yml](../../.mega-linter.yml)'s `REPOSITORY_TRIVY_ARGUMENTS`, and `vulnerability-scan.yml`'s `crazy-max/ghaction-container-scan` step):

* `--ignore-unfixed` (`ignore_unfixed: true` on the container-scan action) — suppress vulnerabilities with no upstream fix available. Because this is a status-based filter, not an ID-based one, it requires no maintenance list and automatically starts reporting a vulnerability again the moment a fix becomes available.
* A package-class Rego ignore-policy, [.github/linters/kernel-findings.rego](../../.github/linters/kernel-findings.rego), suppressing findings whose `PkgName` starts with `linux-` — kernel/kernel-header packages that cannot execute inside a container regardless of which CVE ID is involved. This was implemented as a Rego policy rather than `.trivyignore`/`.trivyignore.yaml` because those formats only suppress specific CVE IDs one at a time; excluding an entire package class regardless of ID requires Trivy's `--ignore-policy` mechanism.

This is a deliberate **noise-reduction** decision, not a merge-gating one: the branch-protection ruleset's required `code_scanning` check still only names `CodeQL`, `SonarCloud`, and `zizmor` — Trivy/Grype/Checkov findings remain advisory-only for now. Whether to promote (a filtered, realistic) Trivy signal to a required check is an open follow-up, deferred until the filtered alert volume has been observed for a period.

### Positive Consequences

* The bulk of the 9,590-alert backlog is expected to self-clear: GitHub automatically dismisses code-scanning alerts a tool no longer detects on a subsequent analysis, so no manual bulk-dismissal is needed.
* Remaining open alerts should be overwhelmingly ones with an available fix and plausible in-container exploitability — a list small enough to actually triage.
* Both suppression criteria are version-controlled, reviewed via pull request, and self-documenting (unlike ad hoc manual alert dismissals in the GitHub UI, which carry no rationale).

### Negative Consequences

* `--ignore-unfixed` means genuinely-real but currently-unfixable vulnerabilities are no longer visible in the default scan output (the full picture remains available on demand via `trivy image --ignore-status ""` / `--show-suppressed`, just not surfaced by default).
* The kernel-package exclusion is a name-prefix heuristic (`PkgName` starting with `linux-`). It is possible, though considered unlikely, that a package matching this prefix does execute meaningful code in some flavor's container — this trade-off is accepted and can be revisited if such a case is found.
* Vulnerability-scan findings still do not block merges even after filtering; a genuinely fixable high-severity CVE introduced by a dependency bump could still be merged. This is an explicitly deferred decision, not a resolved one.

## Pros and Cons of the Options

### Leave scanning output as-is

* ✅ Good, because it requires no change and shows every possible finding.
* ❌ Bad, because it produces the exact alert-fatigue backlog this decision responds to.
* ❌ Bad, because it does not distinguish actionable from non-actionable findings.

### Disable or stop running one or more scanners

* ✅ Good, because it removes the noise immediately with minimal configuration.
* ❌ Bad, because it also removes visibility into genuinely fixable, exploitable vulnerabilities.
* ❌ Bad, because it conflicts with supply-chain security being a top priority for this project.

### Filter to actionable, in-scope findings

* ✅ Good, because every exclusion is scoped, documented, and reviewable in version control.
* ✅ Good, because it self-corrects as fixes become available (`--ignore-unfixed` is status-based, not a static list).
* ✅ Good, because it does not touch severity or the set of scanners run — only removes findings that are provably not actionable or not exploitable in context.
* ❌ Bad, because it still requires judgment calls (e.g. the kernel-package prefix) that could, in principle, be too broad.

## Links

* [Trivy: Filtering](https://trivy.dev/latest/docs/configuration/filtering/)
* [Trivy: Filtering by Rego](https://trivy.dev/latest/docs/configuration/filtering/#by-rego)
* [Open Policy Agent / Rego](https://www.openpolicyagent.org/docs/latest/policy-language/)