Skip to content

scan.md

Latest commit

 

History

History
550 lines (487 loc) · 29.8 KB

scan.md

File metadata and controls

550 lines (487 loc) · 29.8 KB
Raw

redential scan

Reads git history from a local repository and prints the exact proof bundle that submit would upload later — nothing is sent anywhere by scan itself, unless a TTY run continues into the post-scan hand-off into submit's own flow (see below), which reuses this scan's own confirmations but keeps submit's own separate, unweakened consent surface (including its own single upload confirmation) fully intact.

redential scan --repo <path>              # interactive author + confirmation
redential scan --author you@example.com --yes   # non-interactive
redential scan --repo <path> --json       # force JSON-only, even in a terminal
redential scan --repo <path> --details    # TTY summary + hour/weekday histograms
redential scan --since 2years             # limit analysis to the last 2 years
redential scan --debug --repo <path>      # verbose diagnostics on stderr

Output at a glance (phase 2 of the console-UX redesign):

stdout is... --json output
piped/redirected any the exact bundle JSON only — byte-identical to every prior release
a real terminal (TTY) no the human-readable summary only (--details adds two extra sections) — no JSON dump
a real terminal (TTY) yes the exact bundle JSON only, nothing else — suitable for piping even from an interactive shell

--json is treated as "this run is scripted," full stop, even when stdout happens to be a real terminal: besides forcing JSON-only stdout, it also skips the huge-repo progress line and the post-scan hand-off into submit's own flow (below), exactly as if stdout were piped.

How it works

  1. Expectation-setting first line (TTY only). On a real TTY, before anything else, scan prints one line to stderr, in the terminal's own default (neutral) color, setting expectations for the whole credentialing flow, not just this one command: 
    Scanning takes seconds. Completing your credential afterwards takes a ~15-minute spoken defense in your browser.
    Piped/non-TTY output never shows this line — a script has no use for it.
  2. Local-only notice. Right after the line above (or first, on piped/non-TTY output) — before any prompt — scan prints one more line to stderr: 
    Local scan. Nothing leaves your machine.
    Non-blocking, printed in every mode (TTY and piped/non-TTY alike); stdout is unaffected, so the piped bundle JSON stays byte-identical. submit reaches the same lines too, since it shares this exact flow. The longer explanation of what this means lives in redential scan --help. Owner follow-up (2026-08): a calm, reassuring privacy statement should never read like an alert, so this line — and the end-of-scan connectable-repo notice below — are rendered dim/gray when stderr is a real terminal (src/dim.ts), plain text otherwise. Red/orange/warning colors are reserved for real errors only, never for either of these two lines.
  3. Enumerate authors. git log is read locally (git show/git diff never leave the machine) to list distinct author emails and their commit counts.
  4. Select identity. A dim Reading git history... line (stderr, TTY-only) prints right before the git log walk that enumerates authors — bug fix, owner follow-up 2026-08: on a repo with a few thousand commits this walk can take a few real seconds with otherwise zero feedback between the startup lines and the first question, easily read as a hang. With a single candidate, it's taken directly — no prompt at all here; the unified confirmation below (step 4) shows that same email and asks about it directly. With 2+ candidates, and the repo's own git config user.email matching one of them, that one is offered FIRST as a fast default, sharing step 4's exact wording AND its exact no-default/re-ask behavior (consistency fix, owner directive, 2026-08 — see below): 
    Scan 12 commits by you@example.com? This confirms you're authorized to analyze this repository. (y/n)
    Bug fix, owner follow-up 2026-08: this used to be a separate, simpler "Found 12 commits authored by you@example.com. Use this identity? (Y/n)" line, ALWAYS followed by a second, redundant confirmation — accepting it now already fully answers identity AND authorization in one shot, so step 4 is never asked again for that same pick. A follow-up consistency fix then also removed this offer's OWN Y-default: because its "yes" grants authorization (the exact same sentence step 4 asks), a bare Enter here used to silently grant it on the most common repeat-scan path — exactly what the owner's rule (below) forbids, so it now delegates straight to step 4's own prompt function and shares its no-default/re-ask behavior byte for byte. Declining (an explicit n) falls through to a numbered list instead, showing the FULL list including the declined entry — "no" often means "that one plus others" for a multi-identity repo, not "not that one at all"; the list itself is never a confirmation of its own, so whatever it yields still gets exactly one confirmation (step 4, in full). Non-interactively, pass --author <email> (repeatable) for every email that's yours — this skips identity selection entirely, unaffected by any of the above.
  5. Confirm (identity + authorization, unified). One question covers both at once: 
    Scan 1,378 commits by you@example.com? This confirms you're authorized to analyze this repository. (y/n)
    — interactively via this prompt, or non-interactively via --yes. Owner directive (2026-08): ANY question whose "yes" grants authorization — this one, and step 3's fast-path offer above — uses no implied default at all: lowercase (y/n), not (Y/n)/(y/N). Every OTHER y/n prompt in this CLI (the post-scan upload confirmation, the private-label prompt's own retry loop) keeps an ordinary Y-default, since granting upload consent (or entering a label) is a different, lower-stakes decision than granting authorization to analyze a repository. Pressing Enter (or typing anything that isn't an explicit y/yes or n/no, case-insensitive) neither proceeds nor cancels: it RE-ASKS the exact same question, looping until you give one of those two explicit answers. Nothing is ever inferred here. This question (or step 3's fast-path offer, which shares its exact wording and behavior) is the ONLY confirmation asked anywhere in this flow — it replaces three separate ones this CLI used to ask (a pre-scan connectable-repo guardrail question, a per-candidate identity confirmation, and a standalone authorization attestation), and it is never asked twice in a row for the same selection.
  6. Compute the bundle. Every field in schema/bundle.v1.json is derived from git log --numstat filtered to your selected commits: volume, span, hourly/weekday cadence, signed-commit ratio, churn share by file extension and by technical category (heuristic path/extension matching), ownership share against the repo's total commits, and detected skills (deterministic local matching of your commits' diffs against signatures/*.json — see docs/signatures.md; zero network, closed vocabulary, detected_skills may be empty if nothing matched).
  7. Print it. Piped/redirected stdout, or --json (even on a real terminal), prints the JSON — byte for byte what submit would send later, and nothing else. A real terminal with no --json prints the human-readable summary instead (see below) — not the JSON — so run redential scan --json whenever you specifically want the exact payload on screen or piped into something else (jq, a file redirect, etc.).

submit's own consent summary

scan itself no longer prints a "what would get uploaded" consent box — phase 2 of the console-UX redesign replaced it with the richer summary below, whose own footer already restates the same guarantees in plain language and points at redential scan --json for the literal payload. The boxed, itemized consent summary (formatConsentSummary, src/summary.ts) still exists and is still shown, unchanged, by redential submit right before its own upload confirmation — see login-submit.md for that command's own output order and copy ("gets" vs. this doc's historical "would", since submit actually uploads).

The summary (default TTY output)

When stdout is an interactive terminal and --json isn't passed, scan prints ONLY this human-readable summary — no JSON dump. It's a short, shareable overview: span/commits/ownership, detected capabilities (structural findings first, then grouped by category), top languages and categories, ownership and signed-commit ratios, and — owner follow-up, 2026-08 — a single closing line, dim/gray on a real terminal (never warning-colored; see src/dim.ts and the "Startup and closing line colors" note below), pointing at the repo for verification. It's rendered with ANSI colors and Unicode block/box-drawing characters only (no new dependency), and is derived entirely from the bundle scan already computed: no new data collection, no network, nothing beyond what's already in the JSON --json would print.

What used to be spelled out in the terminal, now documented here instead (owner follow-up, 2026-08 — the summary's closing block shrank from a multi-line paragraph plus footer hints to one line, so this is where that detail now lives):

  • Exactly what travels, if you submit. Only the bounded bundle: aggregates (span, commit counts, language/category shares, hour/weekday cadence), salted fingerprints (repo.repo_fingerprint, identity.author_identity_hashes — one-way hashes, not reversible to the original repo path or email), and closed-vocabulary capability slugs (detected_skills[].slug, always a taxonomy.json member). Never source code, file names, commit messages, the repo's name, or other contributors' identities. redential scan --json (below) is the literal, byte-for-byte proof of this — read it directly whenever you want the exact payload rather than trusting this description.
  • Inspecting the exact payload. redential scan --json prints exactly what a submit of this same repo/history would upload — nothing more, nothing rendered or summarized.
  • More detail. redential scan --details adds the COMMITS BY HOUR/WEEKDAY histogram sections (below) to this same summary.

Plain-terminal fallback. Plain Windows conhost (cmd.exe / classic PowerShell without Windows Terminal) doesn't reliably render either ANSI escapes or the Unicode block/box-drawing characters above on a legacy console codepage. shouldUsePlainOutput (src/summary.ts) detects this — win32 with none of WT_SESSION/TERM_PROGRAM/ConEmuANSI=ON set (all three mark a modern, UTF-8-and-ANSI-capable wrapper: Windows Terminal, VS Code's integrated terminal, ConEmu) — and swaps in a pure-ASCII, no-color theme instead: #/- bars, +/=/| box corners/edges, . through # sparkline levels. Every other platform, and every recognized Windows terminal, gets the rich theme. Same data either way — this only changes how it's drawn.

  PRIVATE WORK, LOCALLY DERIVED
  1 year · 1,378 authored commits · 100% ownership

  CAPABILITIES DETECTED

  Payment webhook flow    30 commits   STRUCTURAL · DIRECT

  Frontend
    Next.js              139 commits
    React                113 commits
    Tailwind CSS          80 commits
    Zustand               40 commits
    +1 more

  Backend
    Express               90 commits

  Databases
    PostgreSQL            60 commits

  AI
    Anthropic API         20 commits

  TOP LANGUAGES
  .ts   ████████████████████   62%
  .tsx  ██████░░░░░░░░░░░░░░   20%
  .md   ███░░░░░░░░░░░░░░░░░    8%

  TOP CATEGORIES
  Frontend  ████████████████████   59%
  Backend   ████████░░░░░░░░░░░░   25%
  Testing   ███░░░░░░░░░░░░░░░░░   10%

  Ownership       100% of this repo's commits are yours
  Signed commits  0% of your commits are cryptographically signed

  Nothing was uploaded. You choose what gets sent. github.com/Redential/redential-cli

(Generated from a real fixture bundle via formatSummary.) The closing line is the ONLY thing after the results (owner-mandated ordering, see "Post-scan hand-off to submit" below) — dim/gray on a real terminal, plain text otherwise, never warning-colored. Everything that used to follow it (the repeated privacy paragraph, the --json/--details pointers, the signed-commit tip, the textual next-step CTA) is either documented above/below instead, or — for the signed-commit tip specifically — relocated to submit's own post-upload output (see login-submit.md), and for the next-step CTA, replaced by the live post-scan hand-off described next.

Capabilities are grouped, not a flat list. Structural findings (evidence: "structural" — see proof-graph-spike.md) are pulled out and always listed FIRST, each tagged STRUCTURAL · DIRECT/INFERRED; if a scan has none, nothing is printed about their absence. Every remaining (ordinary import-tier) skill is grouped by its taxonomy slug prefix (frontend, auth, payments, db, ai, backend, queues, observability, testing, email, infra, storage, realtime, data, humanized to a display name — e.g. queues → "Background jobs & queues" — falling back to a capitalized prefix for anything not in that fixed list), groups ordered by their own total commit count descending, entries within a group ordered by commit count and capped at 4 with an honest +N more beyond that. Every label shown — capability names, group headers, category names — comes from taxonomy.json's own label field (never a raw lowercase slug); a slug with no taxonomy label (should not normally happen — skill detection already enforces closed vocabulary) falls back to the bare slug rather than inventing one. TOP CATEGORIES hides the catch-all other bucket entirely and any category under 2% churn share, using this same humanization map.

Signed-commit tip. No longer shown in scan's own summary at all (owner follow-up, 2026-08) — relocated to submit's post-upload output, a single neutral-toned line printed only when the ratio is 0%, right after "Uploaded. Bundle id: ...". See login-submit.md.

--details

Adds two sections right after the header line — the same COMMITS BY HOUR/WEEKDAY histograms this summary always showed before phase 2 moved them out of the default view:

  COMMITS BY HOUR (UTC)
  0     6     12    18
  ▁····▁▁▃▅█▇▄▃▂▂▁▁▁▁▁····

  COMMITS BY WEEKDAY
  Sun  ██░░░░░░░░░░░░░░░░░░  5
  Mon  ███████████████████░  40
  ...

No effect on --json or piped output — neither ever rendered histograms, JSON or otherwise.

Closing next-step hint and the post-scan hand-off to submit

Owner follow-up (2026-08): the old three-state text ("Add this private work to your public Redential profile: → ...") is GONE from the summary itself — formatSummary now ends in the single closing line shown above, full stop. What used to be three flavors of static text is now three different runtime behaviors, decided by the same plain local-state check as before (src/scan-command.ts's nextStepsState, never a network call), and — per an explicit owner-mandated ordering rule — whichever of them fires is always the true LAST thing scan prints:

  1. No stored session (never logged in, or the stored session is for a different SITE_URL): a single dim/gray stderr reminder, no live prompt (there's nothing to hand off to without a session): 
    Log in and run `redential submit` to add this to your Redential profile.
  2. Stored session, but this exact bundle hasn't been uploaded yet (nothing recorded locally yet, or the recorded hash doesn't match this scan's content — e.g. new commits since the last submit): scan continues UNCONDITIONALLY, straight into submit's own flow IN-PROCESS (executeSubmitCommand) — no question of its own first anymore (owner directive, 2026-08: the old separate "Add this to your Redential profile?" prompt is GONE entirely; it used to ask the same decision submit's own upload prompt asked again moments later, two questions for one). It reuses the exact author selection and authorization confirmation you already gave moments earlier in this same scan, so neither is asked again. Every other part of submit's own consent surface fires exactly as it does on a standalone redential submit run: the short summary, the boxed "WHAT GETS UPLOADED" consent box, the private-label prompt, the byte-for-byte exact JSON printed immediately before the single upload confirmation, and that confirmation itself — 
    Upload this to your Redential profile? (Y/n)
    Y is the default (Enter accepts) — this is the ONE decision the whole hand-off asks. Declining (an explicit n) prints "Aborted — nothing was uploaded." and scan exits normally (exit 0) — the scan itself already fully succeeded either way. If the stored session turns out to be invalid (wrong SITE_URL, or deleted since this check ran), this is reported as a plain stderr note rather than turning an otherwise-successful scan into a failed exit code.
  3. Stored session AND this exact bundle content was already uploaded: nothing at all — re-submitting would send nothing new, so the summary's own closing line really is the last thing printed.

"This exact bundle" is decided by bundleContentHash (src/submission-record.ts): a local, unsalted sha256 over the bundle with the fields derived purely from wall-clock time stripped first (created_at, attestation.confirmed_at, repo.age_days) — otherwise a re-scan a moment (or a day) later would never match an otherwise-unchanged repo. Everything else participates, including tool_version and detected_skills: a CLI upgrade can genuinely change what the next submit would upload, so it's deliberately never treated as still identical. redential submit records this hash locally (<config dir>/last-submission.json, alongside credentials.json — see login-submit.md) right after a successful upload; it's not a secret (just a hash of content you already reviewed and already chose to upload), so unlike credentials.json it isn't written with restricted file permissions.

The connectable-repo notice below (when applicable) prints right after the summary and BEFORE whichever of the three states above fires — never after — so state 1's reminder, or state 2's continuation into submit ending in its one upload confirmation, is always the genuinely last thing on screen, per the owner's ordering rule.

This only happens on a real TTY with no --json. scan | jq (or any redirected/piped stdout) prints only the raw JSON, byte-identical to before this summary existed — --json forces that same JSON-only behavior even on a terminal, for scripts that run interactively but still want machine output; neither the reminder, the hand-off into submit, nor the connectable-repo notice ever appear on that path. Exit codes for piped and --json runs are documented in exit-codes.md.

Connectable-repo notice

Console-UX overhaul (2026-08): if the repo's remote looks like it's hosted on a known public host (github.com/gitlab.com/bitbucket.org — isKnownPublicHost, src/public-remote.ts), scan prints ONE line to stderr — non-blocking, no question asked, dim/gray on a real terminal (never warning-colored; src/dim.ts) — right after the summary and before the closing next-step behavior above:

This repo's remote is on a public git host — the GitHub App can add server-side attestation on top of this scan.

This is a heuristic, never proof the repo is actually public — "known host" isn't "publicly accessible", and scan has zero network access to tell the difference (the CLI's primary use case is a private employer repo hosted on github.com), so it never blocks and never asks anything (the old pre-scan "Continue locally? (Y/n)" question is gone entirely). Printed in every mode, TTY and piped/non-TTY alike. submit does NOT print this line at all — its own real, definitive answer to a public remote is the network visibility gate, described in login-submit.md.

Startup and closing line colors

Owner follow-up (2026-08, reproduced via a real pty): calm, informational, non-alarming lines are never rendered in a warning color — red/orange/ warning colors are reserved for real errors only.

Root cause of the bug this fixed. Node.js automatically wraps console.error/console.warn output in red/yellow ANSI codes (\x1b[31m...\x1b[39m) whenever the destination stream is a color-capable TTY — a Node built-in, not something this codebase ever asked for. Every informational stderr line in this CLI defaulted to console.error as its warn implementation, so EVERY one of them (the local-only notice, the expectation line, the connectable-repo notice, the shallow-repo/thin- history notices, ...) rendered in red regardless of any dim() wrapping — wrapping a line in dim() only nested those codes INSIDE Node's own red wrapper, never removed it. The fix (src/stderr.ts's writeStderrLine, a plain process.stderr.write, never console.error) is now the default warn/log implementation everywhere EXCEPT src/program.ts's actual Error: ... path, which keeps using console.error on purpose — that red IS correct there, it's a real error.

This applies to:

  • The local-only notice ("Local scan. Nothing leaves your machine.", step 1 above) and the connectable-repo notice just above — both rendered dim/gray via src/dim.ts when stderr is a real terminal, plain text otherwise (a piped/redirected stderr never receives ANSI codes).
  • The summary's own closing line ("Nothing was uploaded. You choose what gets sent. ..."), rendered dim/gray via src/summary.ts's own plain/rich theme system (the same mechanism gating every other color in the summary), rather than src/dim.ts — it's part of the same themed string this whole function already builds, not a separate stderr call.
  • The expectation-setting first line ("Scanning takes seconds. ..."), deliberately printed in the terminal's own default (neutral) color, not dimmed — it's the one line actually meant to be read at a glance, not ambient reassurance text.
  • The "Reading git history..." interstitial (step 3 above) and the no-session post-scan reminder ("Log in and run redential submit...", scan-command.ts) — both dim/gray via src/dim.ts, same rule. There is no separate decline note anymore: when there's a session and new content, scan continues straight into submit's own flow, and any decline happens there, inside submit's own output (see "Closing next-step hint and the post-scan hand-off to submit" below).

Huge repositories and --since

scan walks git history once, streaming the whole way through — it never buffers a huge repo's full git log output in memory, and it never holds more than one batch's worth of diff content at a time (skill detection fetches added-line diffs in bounded batches of ~200 commits via a single git show process per batch, not one process per commit). A programmatically generated 20,000-commit fixture scans in a few seconds (asserted under 60s in test/slow/huge-repo.test.ts — a separate suite excluded from the default npm test, since building and scanning 20,000 commits, while still fast, shouldn't gate every quick local test run; run it directly with npm run test:slow. CI runs it as its own job on ubuntu-latest, see .github/workflows/ci.yml).

Progress. On a real TTY, scan prints a running line to stderr (never stdout) while it walks history:

scanning commits... 12,400/80,000

throttled to roughly every 200 commits so a huge walk doesn't scroll thousands of lines, and always finishing at the exact total. Piped or redirected stdout (scan | jq, --json) gets no progress output at all — scan's stdout contract (JSON only, byte-identical either way) is unaffected regardless of how large the repo is; this is covered by a test (test/scan-command.test.ts's "huge-repo progress" block) that asserts piped output is identical whether or not a progress reporter would have fired.

--since <spec> limits the WALK, not the truth. Pass a relative window (2years, 18months, 30days — singular or plural) or an absolute date (2024-01-01, or anything else JavaScript's Date parses):

redential scan --since 2years
redential scan --since 2024-01-01

This does not add any new field to the bundle. It changes which commits the existing fields are computed over: commits.user_total, first_at/last_at/span_days, the hour/weekday histograms, identity.other_contributors_count, ownership.user_commit_ratio, and integrity.date_forensics all simply reflect the analyzed window instead of full history — see docs/schema.md for the exact field-by-field breakdown. This is strictly narrower disclosure than a full scan, never a way to fabricate or hide history: a windowed scan can only ever show less of the repo's real activity, never claim activity that didn't happen. Two fields are deliberately exempt from the window and always reflect the whole repo: repo.age_days and repo.repo_fingerprint (both derived from the repo's true root commit) — otherwise a windowed scan could misleadingly make an old repo look freshly created. On a TTY, the summary states the active window next to the header line (e.g. "2 years · 1,847 authored commits · 78% ownership (last 2 years)"), so it's never ambiguous whether a window was applied.

If --since excludes every commit in the repo (but the repo isn't actually empty), scan fails with a message naming the window rather than the generic "no commits yet" error, so it's clear the fix is to widen or drop --since, not that the repo has no history at all.

Shallow clones

git rev-parse --is-shallow-repository is checked once per scan (src/git.ts's isShallowRepository). A shallow clone (git clone --depth N, or the default checkout depth of most CI actions) is missing history before its shallow boundary ENTIRELY — not filtered out like --since, genuinely absent locally — so commits.user_total, span, and repo.age_days would all silently understate real activity with no indication why. scan/submit print a warning (same "warn, never block" stance as the connectable-repo notice above) naming the remedy (git fetch --unshallow) and continue with whatever history IS available; on a TTY, the summary repeats a short note next to the header line too, so it's visible even if the stderr warning scrolled past.

--debug

redential scan --debug --repo <path>
redential --debug scan --repo <path>   # either position works

A global flag (works on every command). Writes verbose diagnostics to stderr only — git commands run (argv only: shas, dates, flags — never the repo path, which would reveal an employer/project name if pasted into a public issue; never diff content or an author's email), phase timings (commit walk, skill detection), and counts (commits walked, commits matching the selected author, diff-fetch batches). Piped/redirected stdout is completely unaffected — scan --debug | jq prints byte-identical JSON to scan | jq, covered by a test (test/privacy/debug-output.test.ts), which also asserts the stored session token and bundle field values (fingerprints, hashes) can never appear in --debug output, even if a future debugLog call got careless about what it logs.

Implementation note: src/debug.ts is module-level mutable state — the one deliberate exception to this codebase's everywhere-dependency- injection style (every other cross-cutting concern is threaded explicitly through an options object). Full DI would mean a debugLog parameter on essentially every function in git.ts/scan.ts/skill-detect.ts; a settable verbose-flag toggle is the standard CLI idiom instead (cf. Node's own util.debuglog).

Design notes

  • Device salt. repo_fingerprint and author_identity_hashes are salted with a random value generated once and persisted at salt inside the same per-platform config directory as credentials.json — see login-submit.md for the exact path on each OS. The salt is device-local, not account-anchored — it survives redential logout and its only job is preventing rainbow-table lookups, independent of any session.
  • Empty / unmatched repos fail loudly. A repository with zero commits, or a --author that matches no commits, raises an error and exits non-zero rather than fabricating a bundle with meaningless dates.
  • No JSON-Schema library at runtime. The CLI builds the bundle from a strongly-typed Bundle interface (src/types.ts) that mirrors the schema; actual conformance against schema/bundle.v1.json is verified by the test suite (test/support/schema-validate.ts), not by shipping a schema validator in the published package.