bot_review_coverage is blind to a clean CodeRabbit review — it arrives as a comment, not a review

[topij]

Found while watching PR #43 through the loop.

The bug

bot_review_coverage(reviews, head, *, bots) reads the reviews list. CodeRabbit delivers a clean verdict as an issue comment, not as a review submission:

No actionable comments were generated in the recent review. 🎉

So on a PR the bot reviewed and passed:

statusCheckRollup  CodeRabbit: SUCCESS
comments           1 (coderabbitai — walkthrough + the verdict line above)
reviews            []
--------------------------------------------------
review_bots.signal    "ok"
review_bots.coverage  []
merge_blockers        ["independent review evidence is missing for current head"]

A PR that was independently reviewed and found clean is indistinguishable from one that was never reviewed.

Observed on this repo's PR #43 (CodeRabbit plan: Pro Plus, profile CHILL, Configuration used: defaults — nothing exotic).

Why it matters

This is the mirror of #19. #19 stopped a receipt binding to a review that had not happened. This makes a review that did happen invisible — and the consequence is worse than cosmetic: the merge gate reports missing review evidence, which pushes the operator toward recording a fallback receipt they did not need, for a review that already exists.

That is a receipt claiming coverage from a pass that never ran, arrived at by following the gate's own instruction. fallback-review-panel.md says "A receipt should never claim more coverage than the review it stands for" — the gate is currently steering people into violating it.

It also silently inflates the cost of every clean PR: a full two-lens panel gets run to satisfy a gate that a real reviewer had already satisfied.

Suggested fix

bot_review_coverage should treat a configured bot's comment-borne verdict as review evidence, the same way review_unavailable_reason already treats a comment-borne outage as an outage. The precedent is exact and already in the codebase — #23's whole lesson was that the same bot says the same thing on different surfaces, and matching only one surface makes detection depend on which one it happened to use.

Concretely: a configured-bot comment matching a verdict marker (no actionable comments were generated, actionable comments posted: N) is a completed review for the commit it references, and should populate coverage with that commit.

Note noise_markers already carries "actionable comments posted: 0" — the kit is currently filtering out the very signal it needs, on one of the two wordings. The other wording ("No actionable comments were generated in the recent review") matches no marker at all.

Related

• pr_watch accepts a review receipt while the primary bot's review is still queued #19 / pr_watch cannot see reviewer unavailability reported as a status-check description #23 — same family, opposite direction; pr_watch cannot see reviewer unavailability reported as a status-check description #23's "match both surfaces" lesson applies verbatim.
• The false-settle guard is one poll wide — a self-merge can fire on a partially-registered check rollup #39 — where I noted signal: "ok" alongside coverage: []. This is the explanation for that pairing: the bot is fine, the reader is looking in the wrong place.
• Verify review-lens coverage instead of self-reporting it (per-lens receipts) #32 — verifying coverage rather than self-reporting it. This bug is a reason the fallback path gets exercised more than it should.

[topij]

Two more occurrences plus a new detail from the third session of 2026-07-27. Clean reviews on #83 and #85 arrived exactly as this issue describes — CodeRabbit edited its outage comment in place into a walkthrough / "no actionable comments" summary, submitted no review object, review_bots.coverage stayed empty, and the receipts had to be recorded by hand. But #87's review — obtained via an explicit @coderabbitai review re-trigger — did submit a real review object with a commit SHA, and coverage populated (covers_head: true, later correctly flipping to false after a push). Review objects with commit SHAs also exist on earlier PRs (#63, #65, #69, #78), so the comment shape is common rather than universal — it appears to depend on how the review was triggered (automatic/recovered vs explicit command). A fix for this issue should handle both shapes rather than assuming the comment shape is the only one.

[coderabbitai]

@topij, thank you—this is a valuable distinction. The fix for #44 should treat comment-only recovered/automatic output and explicit-command GitHub review objects as separate valid receipt shapes, then normalize both into durable coverage/receipt state. It should also preserve the commit association so covers_head correctly changes after subsequent pushes.

🐇🔎

---

✏️ Learnings added

Learnt from: topij
URL: https://github.com/topij/agentic-dev-kit/issues/44

Timestamp: 2026-07-27T10:39:44.580Z
Learning: For GitHub reviews in `topij/agentic-dev-kit`, CodeRabbit review receipt shape can depend on the trigger: automatic or recovered clean reviews may edit an existing outage comment into a walkthrough or “no actionable comments” summary without creating a GitHub review object or populating `review_bots.coverage`; an explicit `coderabbitai review` can create a review object associated with a commit SHA and populate coverage. Any receipt/coverage implementation for Issue `#44` must normalize and support both shapes, with `covers_head` evaluated against the current PR head after pushes.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.

[coderabbitai]

@topij, confirmed. The implementation for #44 should model review receipts as a normalized union of two valid GitHub delivery shapes:

1. Comment-only receipt — automatic/recovered clean reviews that edit the outage comment into a walkthrough or “no actionable comments” result, without a GitHub review object.
2. Commit-bound review receipt — explicit-command reviews (and the earlier examples) that create a GitHub review object associated with a reviewed commit SHA.

For either shape, it should persist a durable receipt and associate it with the best available reviewed head SHA. review_bots.coverage must be populated from that normalized receipt, while covers_head remains derived by comparing the recorded SHA to the current PR head—so it naturally becomes false after a subsequent push.

🐇🔎

---

✏️ Learnings added

Learnt from: topij
URL: https://github.com/topij/agentic-dev-kit/issues/44

Timestamp: 2026-07-27T10:57:57.960Z
Learning: For GitHub review receipt handling in `topij/agentic-dev-kit` Issue `#44`, clean review outcomes have two valid shapes that must be normalized: (1) automatic or recovered reviews may edit an existing outage comment into a walkthrough or “no actionable comments” result without submitting a GitHub review object, as observed on PRs `#83` and `#85`; (2) explicit `coderabbitai review` triggers may submit a GitHub review object with a commit SHA and populate `review_bots.coverage`, as observed on PR `#87` and earlier PRs `#63`, `#65`, `#69`, and `#78`. Persist a durable receipt for both shapes, and derive `covers_head` by comparing any stored reviewed SHA with the current PR head so it becomes false after later pushes.

Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.