Skip to content

feat(analyzer): detect external model selection - #426

Open
deepujain wants to merge 2 commits into
NVIDIA:mainfrom
deepujain:feat/370-model-pin-cost-exposure
Open

feat(analyzer): detect external model selection#426
deepujain wants to merge 2 commits into
NVIDIA:mainfrom
deepujain:feat/370-model-pin-cost-exposure

Conversation

@deepujain

Copy link
Copy Markdown
Contributor

Summary

  • add EA5 detection for external model or provider selection
  • report top-level SKILL.md frontmatter pins as MEDIUM cost exposure
  • report actionable coding-CLI shell-outs as HIGH silent account-switch exposure
  • avoid nested/frontmatter defaults, prose-only mentions, language-runtime -m flags, and unknown local model names

Validation

  • pytest tests/unit/test_patterns_new.py tests/nodes/analyzers/test_static_patterns.py -q (538 passed)
  • pytest -m 'not integration and not provider' tests/ -q (2818 passed, 13 skipped, 38 deselected, 4 xfailed)
  • ruff check src tests
  • ruff format --check src tests
  • real CLI JSON smoke: one HIGH command finding and one MEDIUM frontmatter finding

Risk

  • deterministic static analysis only; no provider calls or runtime model changes
  • generic model flags require a recognized model family and exclude Python/Node/Perl/Ruby module invocation
  • findings are advisory and describe billing/cost exposure rather than malicious intent

Fixes #370

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SkillSpector Review]

Requesting changes for a HIGH-severity prose false positive. Every inline-code command is classified as actionable regardless of the surrounding sentence, so documentation that merely names a supported CLI command is reported as silent model/account switching. Please require directive or shell-example context for inline code and add descriptive inline-code and fenced-example negative regressions.

@deepujain
deepujain force-pushed the feat/370-model-pin-cost-exposure branch 2 times, most recently from faf5bda to 69e8f78 Compare August 24, 2026 18:17
@rng1995

rng1995 commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

[SkillSpector Maintainer Ping]

@deepujain, thank you for the follow-up commit and thread reply. The new directive-context gate plus descriptive inline-code and fenced-example regressions appear to address the EA5 review feedback. Please resolve the review thread and reply here when you want the updated head re-reviewed.

The branch is currently behind main. GitHub reports it as mergeable with no conflicts, but please update it from main, resolve any conflicts that surface during that update, and keep the required checks green.

Signed-off-by: Deepak Jain <deepujain@gmail.com>
Signed-off-by: Deepak Jain <deepujain@gmail.com>
@deepujain
deepujain force-pushed the feat/370-model-pin-cost-exposure branch from fa0dbe6 to 052cd9f Compare August 24, 2026 18:50
@deepujain

Copy link
Copy Markdown
Contributor Author

Updated the branch onto current main at 052cd9f, resolved the addressed review thread, and reran the focused EA5/static-pattern suite (542 passed) plus repository-wide Ruff checks. The required GitHub checks are rerunning on the new head. Ready for re-review when convenient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: detect model/provider pins in skill frontmatter

2 participants