Skip to content

Same-rule repeat alarm: fast-path detection when one deterministic rule rejects N distinct targets in a short window #7983

Description

@JSONbored

Implements #7980. Depends on #7982 (shared calibration module).

Problem

The existing self-correction system only detects a systematically-wrong rule via precision-over-time (auto-tune.ts), which needs a real, decided sample (≥10) accumulated over however long that takes — too slow for a bug that can mis-close 4 PRs within hours, as happened 2026-07-21/22. A much cheaper, ground-truth-free signal already exists in the data: the SAME deterministic rule/blocker code rejecting several DIFFERENT targets in a short window in the same repo/cohort is itself a strong "something's broken" signal, independent of whether any of those rejections ever gets corrected by a human.

Area

Built on the shared module (#7982). ORB-side: src/rules/advisory.ts finding codes / src/review/content-lane's deterministic codes. AMS-side: whatever policy-check codes AMS's own deterministic layer produces.

Proposal

  • A rolling counter, keyed by (deployment/repo-or-cohort, rule/finding code), over a short window (e.g. 1-24h, tunable).
  • When the counter crosses a threshold (e.g. ≥3 within 24h), emit an operator-visible alert (existing Discord/Slack notification path, or a new dedicated channel — reuse what's there) naming the rule code and the affected PR/attempt numbers, WITHOUT taking any autonomous action (detection + alert only, mirrors the existing gamingPatternFlags precedent in src/orb/analytics.ts: "Detection only — never an automatic action").
  • Applies uniformly to both ORB gate blockers and AMS policy rejections via the shared module's interface.

Deliverables

Resources

Boundaries

Detection + alert only — no autonomous hold/disable action (that's sub-issues 4-6 of #7980's job, and even those stay tightening-only per auto-tune.ts's existing design principle).

maintainer-only.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions