Nothing distinguishes a bot that structurally never reviews from one that is merely pending

[topij]

Found by comparing two real repos during the first adopter upgrade.

The observation

Same bot, same config key, two very different realities:

repo	CodeRabbit plan	posts walkthrough	posts a verdict	reviews
agentic-dev-kit (#43)	Pro Plus	yes	yes — "No actionable comments were generated"	[]
OpenKitchen (#256)	Free	yes	no	[]

Across two heads on OpenKitchen #256, CodeRabbit's check went SUCCESS, it posted its walkthrough with a "Files selected for processing (20)" list, and it produced no verdict of any kind. That is consistent with the Free plan being walkthrough-only.

If that reading is right, review.bots: [coderabbit] on OpenKitchen names a reviewer that structurally never delivers a verdict. The fallback panel is not that repo's fallback — it is its primary reviewer, on every PR, forever.

Why it needs surfacing

The engine models three bot states — ok, pending, unavailable — and this is none of them:

• not unavailable: no outage marker, the check succeeds
• not pending: the check is finished
• reported as ok: which is true of the bot and false of the review

So the operator sees signal: "ok", coverage: [], and a merge gate asking for review evidence, with nothing explaining that no evidence will ever arrive from this bot. The first time, you conclude the bot had a bad day. The second time you conclude it is flaky. It takes comparing two repos to see it is structural.

This matters most for the autonomous path, where nobody is comparing repos: an unattended loop on such a repo will run a full fallback panel on every PR while its config asserts it has a review bot, and nothing ever says the configured reviewer is decorative.

Suggested shape

Something that notices "this configured bot has completed N consecutive checks without ever producing a verdict" and says so — once, loudly — rather than reporting ok forever. Even a one-line note at merge time ("coderabbit: check succeeded, no verdict — this bot has produced no review on the last N heads") would turn an invisible structural fact into a config decision.

The narrower, cheaper version: report coverage: [] as a stated gap rather than an empty list, so "the bot reviewed nothing" is legible without the operator having to notice an absence.

Caveat

I have not confirmed with CodeRabbit that the Free plan omits reviews by design — this is inferred from two repos, two heads each, and the plan lines in their own walkthroughs. The kit-side gap (nothing distinguishes this state) holds regardless of the cause.

Related

• docs: the panel's isolation must be of the repo under review #43-adjacent bug (filed alongside this): bot_review_coverage cannot see a clean CodeRabbit review because it arrives as a comment. That bug and this one look identical from the report — coverage: [] — while having opposite causes: there the review exists and is unread, here it does not exist at all. Being unable to tell those apart is itself the argument for both fixes.

[topij]

Four more occurrences from the third session of 2026-07-27: CodeRabbit's status check reported pass with description "Review rate limited" on #85 (once) and #87 (three heads). pr_watch detected every one on the check surface (the #23 fix) and correctly withheld mergeable; the hazard remains that the GitHub checks UI reads as reviewed to a human scanning it.