Skip to content

Add OSV-Scanner-based security workflow#798

Open
vikrantpuppala wants to merge 4 commits into
mainfrom
vp/security-scan
Open

Add OSV-Scanner-based security workflow#798
vikrantpuppala wants to merge 4 commits into
mainfrom
vp/security-scan

Conversation

@vikrantpuppala

@vikrantpuppala vikrantpuppala commented May 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds .github/workflows/securityScan.yml — single workflow, single job, three triggers (PR / weekly cron / manual). PR runs fail on CVSS ≥ 7 only; weekly runs report all findings and email the team.
  • Adds osv-scanner.toml — empty suppressions file (populate iteratively as real false positives surface).
  • Reuses the existing ./.github/actions/setup-jfrog composite action — no duplicate OIDC-token logic.

Mirrors the JDBC driver's workflow (databricks-jdbc#1460), adapted for Python: reads poetry.lock natively via OSV-Scanner (no separate SBOM tool needed).

Day-one results

The workflow is not yet wired into branch protection, so its first PR-time runs are advisory. A dry-run against current main surfaces:

  • 14 HIGH (CVSS ≥ 7) — concentrated in cryptography (45.0.6, 43.0.3), urllib3@2.2.3, black@22.12.0, pyjwt (2.10.1, 2.9.0), pyarrow (17.0.0, 22.0.0)
  • 10 MED, 1 LOW

Note: cryptography/pyjwt/pyarrow each appear at two versions in poetry.lock because the project uses Python-version-gated dependency markers (python = "^3.8.0" with a CI matrix of [3.9, 3.10, 3.11, 3.12, 3.13, 3.14]). Bumping these will require coordinating constraints across Python versions.

A follow-up dep-bump PR will address the findings. Once that's green, branch protection can be flipped to require this check.

Test plan

  • Dry-run OSV-Scanner v2.3.8 locally against poetry.lock — produces expected findings
  • YAML validates
  • First CI run on this PR exercises the PR path (will fail by design — the 14 HIGHs above)
  • Manual workflow_dispatch after merge exercises the weekly path
  • Secrets (SMTP_USERNAME, SMTP_PASSWORD, EMAIL_RECIPIENTS) wired in repo settings before the first scheduled run

This pull request was AI-assisted by Isaac.

@vikrantpuppala vikrantpuppala mentioned this pull request Jun 1, 2026
Open
5 tasks
@vikrantpuppala
Add OSV-Scanner-based security workflow
ec9f1ef 
Single workflow, single job, three triggers:
  - pull_request to main: fails on CVSS >= 7 findings only
    (HIGH/CRITICAL block merges; MED/LOW visible but non-blocking)
  - cron weekly (Sunday 00:00 UTC): reports ALL findings via email
  - workflow_dispatch: behaves like cron

Mirrors the JDBC driver's security workflow (databricks-jdbc#1460)
adapted for Python:
  - Reads poetry.lock natively via OSV-Scanner --lockfile (no
    separate SBOM tool needed)
  - Reuses the existing ./.github/actions/setup-jfrog composite action
    for parity with other workflows (the workflow functionally doesn't
    need JFrog since OSV reads the lockfile directly, but keeping the
    composite action preserves the established pattern)
  - Suppressions in osv-scanner.toml ([[IgnoredVulns]] schema)

The workflow is not yet wired into branch protection. Day-one scan
against current main surfaces 14 HIGH / 10 MED / 1 LOW (25 total) --
concentrated in cryptography, urllib3, pyjwt, pyarrow, requests,
black, pytest, python-dotenv, idna. These will be addressed by a
follow-up dep-bump PR.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
@vikrantpuppala
Harden OSV gate (fail-closed) + refresh lockfile to clear all CVEs
02f2f6b 
Port the fail-closed hardening from the Go (#362) and Node (#388) OSV
workflows, and refresh poetry.lock so the gate passes with zero
suppressions.

securityScan.yml hardening (was fail-open in three places):
- Capture osv-scanner's exit code; tolerate only 0/1 and fail closed on
  any other code (network error, corrupt binary) instead of masking it
  with `|| true`.
- Validate the output is well-formed JSON with a .results array before
  parsing, so a truncated/partial scan fails closed rather than parsing
  to zero findings.
- Resolve empty group max_severity via a cvss_num fallback to an
  UNKNOWN sentinel (using `try (x|tonumber) catch null`, not
  `tonumber?`), so a scoreless finding can never sort to 0 and sail past
  the CVSS>=7 gate. UNKNOWN always blocks (PyPA advisories carry CVSS; a
  scoreless finding is a GHSA-only/malware advisory).
- Integer-count guards fail closed on parse failure.
- Drop per-repo SMTP email in favor of artifact upload for the planned
  cross-repo collator (parity with Go/Node).

CVE clearing WITHOUT forcing dependency floors:
- Bump the Python floor to ^3.10. The CVE-fixed cryptography (>=46) and
  pyjwt (>=2.12) require Python >=3.10 upstream, so a single CVE-clean
  lockfile cannot span 3.8/3.9. This is the only breaking change.
- All runtime dependency pins are UNCHANGED (thrift ~=0.22.0,
  urllib3 >=1.26, requests ^2.18.1, pyjwt ^2.0.0, pyarrow floors). The
  existing constraints already ALLOW the CVE-free versions; the refreshed
  lock simply resolves to them (urllib3 2.7.0, cryptography 49.0.0,
  pyarrow 23.0.1, requests 2.34.2, pyjwt 2.13.0, idna 3.18,
  python-dotenv 1.2.2). Customers do not need us to relax or raise any
  pin to become CVE-free.
- thrift stays ~=0.22.0 (no known advisory; the <0.23 cap avoids the
  ES-1960554 DBR-LTS install break).
- Bump dev-only black ^22 -> ^26 and pytest ^7 -> ^9 to clear their
  advisories (never shipped in the wheel); reformat src with black 26.

Result: OSV-Scanner v2.3.8 reports 0 findings on the refreshed lock;
osv-scanner.toml needs no suppressions.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
@vikrantpuppala
ci: drop Python 3.9 from CI matrices to match the ^3.10 floor
334331a 
The pyproject floor is now ^3.10, so 3.9 legs can no longer `poetry
install` (^3.10 is unsatisfiable on a 3.9 interpreter) and would fail.
Remove "3.9" from every unit-test / lint / type-check / pyarrow / kernel
matrix in code-quality-checks.yml and warm-deps-cache.yml, and drop the
now-moot 3.9-kernel exclude in the warm-deps cache.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
@vikrantpuppala
Document thrift CPE-gap accurately + add weekly NVD-CPE thrift watch
30aa6e3 
Corrects the earlier "thrift ~=0.22.0 (no known advisory)" wording, which
was wrong: thrift 0.22.0 IS affected by open Apache Thrift advisories
(CVE-2025-48431 + the CVE-2026-41602..41636 set, all fixed in 0.23.0).

Why we still hold at ~=0.22.0 and accept them:
- Apache Thrift is one monorepo shipping ~20 language libraries; the PyPI
  `thrift` package is built only from lib/py. Each of these CVEs is in a
  NON-Python binding -- verified against the upstream oss-security
  advisories: Node.js (41636), Go (41602), c_glib/C (48431), Java (41603),
  Swift (41604, 41605). None touches the Python code paths we ship.
- The only fix (0.23.0) is the version that caused SEV0 ES-1960554 on
  DBR-LTS old setuptools, so we cannot take it until a build-safe thrift
  ships (THRIFT-6067).

Why the OSV gate doesn't flag it (and why that is NOT proof Python is safe):
- These CVEs are in OSV with `affected[].package = null` -- only a GIT/CPE
  coordinate (cpe:2.3:a:apache:thrift), no PyPI/npm/Go package entry. OSV
  and Dependabot both match by package purl, so they return nothing for
  PyPI thrift. This is a coordinate blind spot, independent of whether
  Python is affected -- a FUTURE Python-affecting thrift CVE filed the same
  way would also be missed.

Mitigation: a supplementary NVD-CPE thrift watch in securityScan.yml,
scheduled/manual only (never PR; NVD rate limits). It lists all
apache:thrift CVEs affecting the locked version in the weekly summary and
hard-fails if any description names Python. Scoped to thrift alone because
an audit of all three drivers' full dependency sets found thrift is the
only dep with this purl-vs-CPE gap (Go/Node already ship the fixed 0.23.0).
The Python-detection is a heuristic (description must say python/lib/py);
the full list is always surfaced for human review.

Co-authored-by: Isaac
Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
@vikrantpuppala vikrantpuppala requested a deployment to azure-prod July 16, 2026 17:29 — with GitHub Actions Pending
@vikrantpuppala vikrantpuppala requested a deployment to azure-prod July 16, 2026 17:29 — with GitHub Actions Pending
@vikrantpuppala vikrantpuppala requested a deployment to azure-prod July 16, 2026 17:29 — with GitHub Actions Pending
@vikrantpuppala vikrantpuppala requested a deployment to azure-prod July 16, 2026 17:29 — with GitHub Actions Pending
@vikrantpuppala vikrantpuppala requested a deployment to azure-prod July 16, 2026 17:29 — with GitHub Actions Pending
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@vikrantpuppala