Skip to content

fix: safely suppress verified Hugging Face model-card image examples - #1791

Open
mldangelo-oai wants to merge 9 commits into
mainfrom
mdangelo/codex/fix-hf-readme-doc-false-positives-20260728
Open

fix: safely suppress verified Hugging Face model-card image examples#1791
mldangelo-oai wants to merge 9 commits into
mainfrom
mdangelo/codex/fix-hf-readme-doc-false-positives-20260728

Conversation

@mldangelo-oai

Copy link
Copy Markdown
Contributor

Summary

  • Downgrade Hugging Face image-example network findings only for three immutable, revision-pinned timm model-card READMEs with matching complete-file sizes and SHA-256 digests, documented Apache-2.0 or MIT provenance, and the expected official image URL.
  • Preserve --no-whitelist and --strict policy, and fail closed for modified or truncated cards, detector failures, reporting limits, and actionable network behavior.
  • Add LF/CRLF, malicious near-match, whitelist-policy, CLI, detector-failure, truncation, exit-code, and uncached-result regression coverage while preserving current main network and cache protections.

Validation

  • pytest tests/scanners/test_text_scanner.py tests/detectors/test_network_comm_detector.py -q -n 4 --disable-warnings: 1,742 passed.
  • Repository-wide Ruff lint and formatting: 424 files clean.
  • Changed scanner/test mypy passed. The complete 479-file mypy run passes with --python-version 3.13 --disable-error-code unreachable; the unsuppressed diagnostic concerns an unchanged baseline test.
  • Three independent exact-head Codex security reviews and fresh verification completed without findings.

Copilot AI review requested due to automatic review settings July 30, 2026 18:16
@mldangelo-oai

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: f088adcba8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the TextScanner to downgrade network communication findings to informational only for a small set of revision-pinned Hugging Face model-card README payloads that exactly match expected size + SHA-256 digests (including LF/CRLF variants), reducing false positives while keeping existing “fail closed” behavior for incomplete analysis and policy overrides.

Changes:

  • Add a digest-verified allowlist for three specific Hugging Face timm model-card READMEs and downgrade only the corresponding urllib/urlopen findings to INFO when whitelisting is enabled and analysis is complete.
  • Thread an explicit policy flag through network-finding classification so suppression can be disabled when scans are incomplete/limited or whitelisting is turned off.
  • Add extensive regression coverage (whitelist modes, CLI flags, tampering/near-match, truncation, detector failure, finding limits, LF/CRLF) and add the pinned model-card fixtures.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
modelaudit/scanners/text_scanner.py Introduces digest-verified Hugging Face README recognition and conditional network-finding downgrade logic.
tests/scanners/test_text_scanner.py Adds regression tests to ensure the downgrade is narrowly-scoped and fails closed under truncation/limits/failures and policy overrides.
tests/assets/huggingface_model_cards/timm_mobilenetv3_small_100.lamb_in1k_README.txt Adds pinned model-card fixture used to validate the digest allowlist behavior.
tests/assets/huggingface_model_cards/timm_convnext_femto.d1_in1k_README.txt Adds pinned model-card fixture used to validate the digest allowlist behavior.
tests/assets/huggingface_model_cards/timm_repvgg_a0.rvgg_in1k_README.txt Adds pinned model-card fixture used to validate the digest allowlist behavior.
CHANGELOG.md Documents the new false-positive suppression behavior for the verified model cards.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 2816 to 2819
finding: dict[str, Any],
*,
allow_verified_huggingface_documentation: bool = True,
) -> bool:
@github-actions

Copy link
Copy Markdown
Contributor

Workflow run and artifacts

Performance Benchmarks

Compared 13 shared benchmarks with a regression threshold of 15%.
Status: 1 regressions, 0 improved, 12 stable, 0 new, 0 missing.
Aggregate shared-benchmark median: 4.225s -> 4.431s (+4.9%).

Top regressions:

  • tests/benchmarks/test_scan_benchmarks.py::test_scan_single_checkpoint_before_load +22.0% (105.74ms -> 129.00ms, single-checkpoint-preflight, single_checkpoint.pkl, size=183.0 KiB, files=1)
Workload Benchmark Target Size Files Baseline Current Change Status
single-checkpoint-preflight tests/benchmarks/test_scan_benchmarks.py::test_scan_single_checkpoint_before_load single_checkpoint.pkl 183.0 KiB 1 105.74ms 129.00ms +22.0% regression
suspicious-pickle-intake tests/benchmarks/test_scan_benchmarks.py::test_scan_suspicious_pickle_intake suspicious-intake 183.8 KiB 4 147.10ms 163.44ms +11.1% stable
duplicate-heavy-registry tests/benchmarks/test_scan_benchmarks.py::test_scan_duplicate_registry_snapshot registry-snapshot 915.2 KiB 13 599.51ms 655.60ms +9.4% stable
mixed-model-repository tests/benchmarks/test_scan_benchmarks.py::test_scan_release_candidate_repository release-candidate 547.3 KiB 32 624.25ms 673.76ms +7.9% stable
padded-multi-stream-upload tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_padded_multi_stream_upload multi_stream_padded 4.1 KiB 1 291.2us 302.9us +4.0% stable
direct-malicious-upload tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_direct_malicious_upload malicious_reduce 52 B 1 188.3us 194.5us +3.3% stable
rejected-basic-auth-candidates tests/benchmarks/test_scan_benchmarks.py::test_rejected_basic_auth_candidates_scan_linearly - 371.1 KiB 1 2.379s 2.437s +2.4% stable
warm-cache-rescan tests/benchmarks/test_scan_benchmarks.py::test_scan_warm_cached_repository_rescan release-candidate 547.3 KiB 32 137.09ms 140.31ms +2.3% stable
nested-payload-review tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_nested_payload_review[nested_hex] nested_hex 130 B 1 261.3us 266.6us +2.0% stable
nested-payload-review tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_nested_payload_review[nested_base64] nested_base64 98 B 1 251.0us 255.6us +1.8% stable
nested-payload-review tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_nested_payload_review[nested_raw] nested_raw 78 B 1 235.3us 238.8us +1.5% stable
clean-training-checkpoint tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_clean_training_checkpoint safe_large 278.2 KiB 1 114.00ms 113.80ms -0.2% stable
chunked-upload-stream tests/benchmarks/test_picklescan_benchmarks.py::test_picklescan_chunked_upload_stream chunked_stream 278.2 KiB 1 116.76ms 116.72ms -0.0% stable

@chatgpt-codex-connector

Copy link
Copy Markdown

Security review completed. No security issues were found in this pull request.

Reviewed commit: f088adcba8

View security finding report

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants