Skip to content

chore(repo): replace Cisco skill scanner with NVIDIA SkillSpector#51

Open
GangGreenTemperTatum wants to merge 1 commit into
mainfrom
ads/eng-7098-replace-skill-scanner-with-nvidia-skillspector
Open

chore(repo): replace Cisco skill scanner with NVIDIA SkillSpector#51
GangGreenTemperTatum wants to merge 1 commit into
mainfrom
ads/eng-7098-replace-skill-scanner-with-nvidia-skillspector

Conversation

@GangGreenTemperTatum

@GangGreenTemperTatum GangGreenTemperTatum commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Replaces the Cisco AI skill scanner with NVIDIA SkillSpector for capability security scanning.

What's included

  • scripts/security-scan.sh — rewritten to use uvx --from git+https://github.com/NVIDIA/SkillSpector skillspector instead of cisco-ai-skill-scanner. Supports --format, --sarif, --json, and --llm flags. Defaults to --no-llm for deterministic, API-key-free scans.
  • .github/workflows/security-scan.yml — updated to install SkillSpector from git, run static analysis (--no-llm), generate SARIF, and upload to GitHub Code Scanning. The scan step uses continue-on-error: true because security-focused capabilities often score high by design.
  • scripts/test_security_scan.sh — rewritten for SkillSpector: verifies install from git, JSON/SARIF output, and that the malicious fixture scores higher than the clean fixture.
  • scan-policy.yaml — renamed to scan-policy.yaml.cisco-legacy because SkillSpector does not consume the Cisco policy format.
  • README.md and Justfile — updated references and notes.

Key differences from Cisco scanner

Aspect Cisco scanner SkillSpector
Install source PyPI cisco-ai-skill-scanner GitHub git+https://github.com/NVIDIA/SkillSpector (not on PyPI yet)
Scan command skill-scanner scan-all <dir> skillspector scan <path>
Recursive --recursive implicit for directories
Policy file scan-policy.yaml none (no per-rule suppression config yet)
LLM analysis --use-behavioral --no-llm to disable (default enabled)
Failure threshold --fail-on-severity high exits 1 when risk score > 50
Output formats summary, json, markdown, table, sarif, html terminal, json, markdown, sarif

Validation

  • bash -n scripts/security-scan.sh scripts/test_security_scan.sh
  • ./scripts/security-scan.sh ai-red-teaming produces terminal output and writes SARIF/JSON ✅
  • ./scripts/test_security_scan.sh -v16 passed, 0 failed
  • python3 -m pytest capabilities/web-security/tests/ --ignore=capabilities/web-security/tests/test_bbscope.py -q → 153 passed ✅

Notes and open questions

  • SkillSpector is not on PyPI, so CI installs from git on each run. uv caches builds aggressively, but this is slower than a PyPI install.
  • There is no policy/config mechanism in SkillSpector (yet), so the Cisco scan-policy.yaml tuning is removed. This means offensive-security capabilities will naturally produce HIGH/CRITICAL scores. The workflow currently reports these without blocking merges.
  • The exit-code behavior is different: Cisco failed on HIGH severity; SkillSpector fails when risk score > 50. The wrapper and workflow capture/report this without failing the build.

- Swap scripts/security-scan.sh from cisco-ai-skill-scanner to
  skillspector installed from git+https://github.com/NVIDIA/SkillSpector.
- Update .github/workflows/security-scan.yml to run SkillSpector in
  static mode (--no-llm) and upload SARIF to GitHub Code Scanning.
  The workflow reports findings but does not block merges while risk
  thresholds are being tuned for security-focused capabilities.
- Rewrite scripts/test_security_scan.sh for SkillSpector output format,
  SARIF validation, and malicious-vs-clean risk-score assertions.
- Rename scan-policy.yaml to scan-policy.yaml.cisco-legacy; SkillSpector
  does not consume the Cisco policy format.
- Update README.md and Justfile references.
@GangGreenTemperTatum GangGreenTemperTatum marked this pull request as ready for review June 12, 2026 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant