Problem
On 2026-07-21/22, metagraphed's registry-surface reviewer closed at least 4 legitimate PRs (#7469, #7589, #7591, #7594) citing "appears to include secret, wallet, PAT, or private-key material," because containsSecretLikeText (src/review/content-lane/registry-logic.ts) matches the bare words hotkey/coldkey anywhere in a subnet document — including pre-existing, unrelated text, and even explicit denials ("No wallet/hotkey data"). "hotkey" is Bittensor's standard public miner identifier, not secret material.
Root-causing this surfaced a deeper gap: the self-correction system that exists specifically to catch a reviewer repeatedly making the same mistake (the pr_outcome/reversal recording + auto-tune.ts's precision circuit-breakers) never engaged, for three independent, verified reasons:
recordReversalSignals explicitly excludes owner-initiated reopens ("administrative, not a contributor dispute") — every one of today's maintainer-driven rescues is invisible to the metric meant to measure mistakes.- A PR that stays closed with no further human action (3 of the 4 above) generates zero signal under the current "reversal = a specific webhook transition" model.
surface_lane_reject/secret-or-credential is hardcoded into CONCRETE_EVIDENCE_BLOCKER_CODES, which makes it categorically immune to the close-precision breaker (downgradeCloseToHold) — even if the breaker fully engages, this exact finding type can never be held back, because the architecture assumes "deterministic = trustworthy," which is false when the deterministic rule itself is buggy.
The public "Decision accuracy" homepage number reflects neither problem: the own-ledger figure is frozen (stopped receiving metagraphed data once it moved to self-hosted), and the fleet-wide figure reads the same reversal events, so it inherits blind spot #1.
Area
Review/gate accuracy infrastructure — spans ORB's content-lane + generic gate (src/review/, src/settings/agent-actions.ts) and is deliberately scoped to also serve AMS's own deterministic policy checks (src/ams/, packages/loopover-miner/), which face the identical risk shape: a hardcoded rule that's wrong for a class of legitimate input, with the system architecturally unable to notice.
Proposal
Six pieces, ordered so later ones build on earlier foundations. Filed as separate sub-issues so each lands as its own reviewable, tested PR:
- Fix the immediate regex, plus a companion redaction bug that degrades the bot's own error messages.
- A shared, deployment-mode-agnostic calibration/signal-tracking module (home for everything below), usable identically by ORB's cloud Worker, ORB self-hosted, and AMS (both deployment modes) — not an ORB-only bolt-on.
- A same-rule-code repeat alarm: N distinct rejections from the identical deterministic rule within a short window, in one repo/cohort, is a strong, fast, ground-truth-free signal something is broken — far faster than waiting for precision-over-time to accumulate.
- Per-rule (not just per-project) precision tracking, so one bad deterministic rule can't hide inside an otherwise-healthy aggregate.
- Narrow the reversal-signal exclusion: an owner reopen immediately followed by an approve/merge is a genuine correction signal, not administrative noise.
- Replace the blanket "concrete evidence = breaker-immune" code-list with a per-rule track record, so a deterministic rule only keeps its immunity while it's actually earning it.
Deliverables
- 6 linked sub-issues (tracked below), each independently scoped, tested, and deployable.
- This epic tracks sequencing/dependencies only; no code lands directly against it.
Resources
Boundaries
No change to what counts as a genuine security/secret blocker beyond the specific hotkey/coldkey fix in sub-issue 1 — this is about catching future rule-level mistakes faster, not loosening review standards. No autonomous loosening of any gate (mirrors auto-tune.ts's existing tightening-only design). No reward/scoring changes.
maintainer-only — cross-cutting review-infrastructure work, not a build task for contributors.
Problem
On 2026-07-21/22, metagraphed's registry-surface reviewer closed at least 4 legitimate PRs (#7469, #7589, #7591, #7594) citing "appears to include secret, wallet, PAT, or private-key material," because
containsSecretLikeText(src/review/content-lane/registry-logic.ts) matches the bare wordshotkey/coldkeyanywhere in a subnet document — including pre-existing, unrelated text, and even explicit denials ("No wallet/hotkey data"). "hotkey" is Bittensor's standard public miner identifier, not secret material.Root-causing this surfaced a deeper gap: the self-correction system that exists specifically to catch a reviewer repeatedly making the same mistake (the
pr_outcome/reversal recording +auto-tune.ts's precision circuit-breakers) never engaged, for three independent, verified reasons:recordReversalSignalsexplicitly excludes owner-initiated reopens ("administrative, not a contributor dispute") — every one of today's maintainer-driven rescues is invisible to the metric meant to measure mistakes.surface_lane_reject/secret-or-credentialis hardcoded intoCONCRETE_EVIDENCE_BLOCKER_CODES, which makes it categorically immune to the close-precision breaker (downgradeCloseToHold) — even if the breaker fully engages, this exact finding type can never be held back, because the architecture assumes "deterministic = trustworthy," which is false when the deterministic rule itself is buggy.The public "Decision accuracy" homepage number reflects neither problem: the own-ledger figure is frozen (stopped receiving metagraphed data once it moved to self-hosted), and the fleet-wide figure reads the same reversal events, so it inherits blind spot #1.
Area
Review/gate accuracy infrastructure — spans ORB's content-lane + generic gate (
src/review/,src/settings/agent-actions.ts) and is deliberately scoped to also serve AMS's own deterministic policy checks (src/ams/,packages/loopover-miner/), which face the identical risk shape: a hardcoded rule that's wrong for a class of legitimate input, with the system architecturally unable to notice.Proposal
Six pieces, ordered so later ones build on earlier foundations. Filed as separate sub-issues so each lands as its own reviewable, tested PR:
Deliverables
Resources
src/review/content-lane/registry-logic.ts(containsSecretLikeText)src/review/outcomes-wire.ts(recordReversalSignals, the GAP-4 feedback loop)src/review/auto-tune.ts(precision circuit-breakers)src/settings/agent-actions.ts(CONCRETE_EVIDENCE_BLOCKER_CODES,downgradeCloseToHold)src/review/public-stats.ts,src/services/public-accuracy-trend.ts,src/orb/analytics.ts(public + fleet accuracy metrics)src/ams/,packages/loopover-miner/(the AMS side this should also serve)Boundaries
No change to what counts as a genuine security/secret blocker beyond the specific hotkey/coldkey fix in sub-issue 1 — this is about catching future rule-level mistakes faster, not loosening review standards. No autonomous loosening of any gate (mirrors
auto-tune.ts's existing tightening-only design). No reward/scoring changes.maintainer-only — cross-cutting review-infrastructure work, not a build task for contributors.