Skip to content

test: add signature near-miss fixtures - #82

Merged
jpbelmo merged 2 commits into
Redential:mainfrom
BharadwajKanneveti:fix/tier2-false-positives
Aug 12, 2026
Merged

test: add signature near-miss fixtures#82
jpbelmo merged 2 commits into
Redential:mainfrom
BharadwajKanneveti:fix/tier2-false-positives

Conversation

@BharadwajKanneveti

@BharadwajKanneveti BharadwajKanneveti commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

What does this PR do, and why does it fit the north star?

This PR strengthens Tier 2 signature test coverage by adding genuine near-miss
negative fixtures for existing Firebase Auth, OAuth/OIDC, ActiveRecord,
GitHub Actions, and Jest signatures.

These fixtures represent code that looks similar to the corresponding
technology but should not match the signature. This helps verify that existing
signature detection remains precise and reduces the risk of false-positive
skill detection.

This fits the north star by making detected credentials more trustworthy and
verifiable through stronger fixture coverage.

Part of #27

Checklist

  • Tests pass locally (npm test)
  • npm run typecheck passes

If this adds or changes a detection signature

  • Includes at least one positive fixture and one negative fixture
    (the negative fixture is a genuine near-miss, not unrelated text)
  • The slug used already exists in taxonomy.json — if it's new, that's
    a separate PR with a rationale, linked here: #

If this touches WHAT data leaves the machine, or WHERE it's sent

  • Links the prior "Data boundary change" discussion issue (required
    before this PR was opened): #
  • Schema version bumped, with docs/schema.md updated
  • Adds/updates a test in test/privacy/

If this adds a new runtime dependency

  • Written justification included below (what it does, why the
    existing stack — commander/vitest — can't do it, what it adds to
    the supply-chain surface)

Docs and changelog

  • CHANGELOG.md entry added under [Unreleased] (if user-facing)
  • Relevant doc in docs/ added or updated, in English

Additional context

The changes are limited to existing Tier 2 signature fixtures and do not
modify the detection engine, add new taxonomy slugs, change runtime
dependencies, or change the data boundary.

The negative fixtures were chosen as genuine near-misses for each signature,
rather than unrelated text, so they exercise cases that could plausibly be
mistaken for a match.

@BharadwajKanneveti

Copy link
Copy Markdown
Contributor Author

Hi @jpbelmo , I know this isn’t related to the code here, but could I DM you about something? If so, what’s the best place to reach you?

@jpbelmo jpbelmo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three of these five are exactly what #27 wants pinned: the oauth-oidc string-literal miss, the ApplicationService inheritance shape, and the vitest mock surface. Keep those as-is. Four things before merge:

  1. The new github-actions negative is a byte-identical copy of the one already there (same path, same text). Swap it for a real near-miss, e.g. uses: actions/checkout@v4 inside a # comment, or workflow-shaped YAML living outside .github/workflows/.
  2. The firebase-auth fixture is a trap for future us: signInWithRedirect is a real firebase/auth API that just happens to be missing from the patterns today. As written, the fixture would block anyone who legitimately adds it to detection later. Make it a // comment line (so it tests the sanitizer, which is the interesting behavior) or use an API name that isn't a real detectable call.
  3. Changelog: main released 0.11.0 yesterday, so your entry currently merges INSIDE the released 0.11.0 section and rewrites shipped notes: GitHub won't flag it because the merge is textually clean. Rebase, add a fresh ## [Unreleased] heading, and put the line under ### Added (nothing is fixed here, and Added-before-Fixed is the file's convention).
  4. Closes #27Part of #27. That issue is the standing hunt, and its definition of done is fixtures that fail current detection as false positives plus the tightening that makes them pass honestly. This PR is the "pin current behavior" half, welcome but not the close.

Small list, and the three good fixtures make the suite genuinely stronger. On the next push this merges through the usual gate.

@jpbelmo

jpbelmo commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

@BharadwajKanneveti of course — DMs are open at x.com/jpbelmo, write me there anytime.

@BharadwajKanneveti
BharadwajKanneveti force-pushed the fix/tier2-false-positives branch from 420880e to 31df0cc Compare August 12, 2026 14:22
@jpbelmo
jpbelmo merged commit c4c5342 into Redential:main Aug 12, 2026
7 checks passed
@jpbelmo

jpbelmo commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Merged. The reworked fixtures came back better than asked: the github-actions and firebase-auth negatives now pin the comment-stripping sanitizer itself, which is the behavior that actually protects against false positives in prose and docs. The hunt in #27 stays open whenever you want another pass at it. Eleven merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants