From ae691a9e9769a54d5563214eb11c9cf9914aeb6c Mon Sep 17 00:00:00 2001 From: Topi Jarvinen Date: Sun, 26 Jul 2026 02:45:03 +0300 Subject: [PATCH 1/2] Wrap-up: the fallback review is a panel, and its verification was deleted MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #31 merged, closing #26. `review.fallback_panel` is the primary substitute when a review bot can't review — one isolated, fresh-context reviewer per lens — with `fallback_commands` kept as the explicitly degraded one-lens mode. The lens contract lives in the new kit-owned `docs/agentic-dev-kit/fallback-review-panel.md`. Two issues filed from the session: **#32** (verify lens coverage instead of self-reporting it) and **#33** (the drift self-check makes mutation testing report false kills). Measured, not asserted: suite 300 at `1ef5fd6` → 320 at `7485512`. **The session's result is a retraction.** Four successive mechanisms were built to make a receipt's coverage claim verifiable by the engine, and each was defeated by the next review round — the last by one decorated character in a field the caller writes themselves, after which the render affirmed the forgery. Rule 1 already says to treat a tightened matcher as a stopgap; four tightenings is well past taking that seriously. What ships records the claim and labels it a claim. Three findings in the friction log matter beyond this PR: - **#33 is retroactive.** Mutation evidence cited across #25, #28 and #29 may be worthless wherever the reviewer did not exclude the drift test — and "N mutants died" was used as a reason to stop reviewing. A false-negative testing tool is worse than none, because it justifies confidence. - **Concurrent review lenses in one working tree destroy each other's work.** Fixed as contract item 7, but recorded for the pattern: any doctrine that says "run N reviewers concurrently" owes them isolation, and mine did not until it bit. - **Deleting a check reintroduced the bug it was masking.** When removing a mechanism as unfit, enumerate what it was rejecting and confirm each case is still handled — the deletion commit did neither. Adopter upgrade instructions are written and now unblocked (they were held until #31 landed): `~/Documents/openkitchen-devkit-upgrade.md` and `~/Documents/cs-toolkit-devkit-adoption.md`. cs-toolkit turned out **not** to be an adopter at all — no config, no init.sh, its own divergent engines — so that one is an adoption plus a fixer change from `done` to `converged`, not an upgrade. Claude-Session: https://claude.ai/code/session_011KVJzDj7Z2nYkUjqbGgSVp --- docs/kit-friction-log.md | 43 ++++++++++++++++++++++++ docs/kit-handoff.md | 72 ++++++++++++++++++++++++++++++++++++++-- 2 files changed, 112 insertions(+), 3 deletions(-) diff --git a/docs/kit-friction-log.md b/docs/kit-friction-log.md index b4c96be..2374b05 100644 --- a/docs/kit-friction-log.md +++ b/docs/kit-friction-log.md @@ -98,3 +98,46 @@ failure — closed by PR #28). that made the fix obvious. Worth keeping in mind when weighing whether a guard is worth its friction — this one cost ten seconds and prevented a narrative-file conflict with the wrap-up PR. No change proposed. + +## 2026-07-26 — inbox + +- **Mutation testing this repo reports false kills — filed as [#33](https://github.com/topij/agentic-dev-kit/issues/33) (severity: H).** + `kit_doctor`'s self-check rehashes every kit-owned file, so any byte change to an + engine fails it and every mutant looks killed. Measured on #31: 17/17 reported + killed; with that test excluded, **7 had survived**. Filed at H because it is + **retroactive** — mutation evidence cited across #25, #28 and #29 this session may be + worthless wherever the reviewer did not exclude it, and "N mutants died" was used as + a reason to stop reviewing. A false-negative testing tool is worse than none, because + it is used to justify confidence. + +- **Concurrent review lenses in one working tree destroy each other's work (severity: H).** + On #31 the adversarial lens mutated `pr_watch.py` to test it; the correctness lens, + running at the same time, saw those mutations as an external process corrupting the + repo and ran `git checkout --` to "restore" it. They fought for ~10 minutes. One + lens's results were unreliable; when I stopped the other it left a live mutant behind + (`if False and _PANEL_LENS_NAMES:`) that **silently disabled a guard in my working + tree**, and I caught it only because a test failed citing an error string that should + no longer have existed. Already fixed as contract item 7 (isolated worktrees) and + `.gitignore`/`init.sh` entries, so this is recorded for the pattern rather than as an + open item: **any doctrine that says "run N reviewers concurrently" owes them + isolation**, and mine did not until it bit. + +- **Deleting a check reintroduced the bug it was masking (severity: M).** + The roster check removed from #31 was the only thing catching `,` as punctuation in + `--lenses` — so `"adversarial, focused on the merge gate"` (an honest way to record + ONE lens) rendered as two, suppressing the one-lens warning that was the field's + entire remaining value. The commit that deleted it quoted that exact input as an + example of what it still blocked. **Lesson worth generalising:** when removing a + mechanism as unfit, enumerate what it was rejecting and confirm each case is either + still rejected elsewhere or deliberately allowed — the deletion commit did neither. + +- **Four rounds of tightening a matcher is the signal to delete it, and the rule + already says so (severity: M, pattern).** + `safety-critical-changes.md` rule 1: *"Treat 'we tightened the matcher' as a stopgap, + not a fix."* On #31 I tightened it four times before accepting that. The adversarial + lens told me in round 3 that the artifact was unverifiable from the engine — same + actor, same invocation, nothing bound to what ran — and I built two more epicycles + before acting on it. **Proposed fix:** rule 1 could name a threshold ("a second + tightening of the same matcher is a design signal, not a bug fix") so the decision + point is written down rather than requiring the author to notice it. + diff --git a/docs/kit-handoff.md b/docs/kit-handoff.md index 1df6381..c17edf6 100644 --- a/docs/kit-handoff.md +++ b/docs/kit-handoff.md @@ -14,10 +14,76 @@ > Older session blocks graduate to [`kit-handoff-history.md`](kit-handoff-history.md) once > this file crosses its line budget (`scripts/check_doc_budget.py`). -Last updated: 2026-07-25 — Phase 3b shipped: the queued-vs-unavailable ambiguity is -closed on both surfaces, and the receipt now says what it does *not* cover. +Last updated: 2026-07-26 — the fallback review is a panel, and the attempt to make +its claim machine-verifiable was built four times and then deleted. + +## Latest session — 2026-07-26 (#26, overnight) + +**Theme —** Built `review.fallback_panel`, then spent four rounds trying to make the +receipt's coverage claim verifiable *by the engine*, then deleted all of it. The +deletion is the result, not the failure. + +- **#31 merged, closing #26.** `review.fallback_panel` is the primary substitute when + a bot can't review: one isolated, fresh-context reviewer per lens. + `fallback_commands` stays as the explicitly degraded one-lens mode. The lens + *contract* — fresh context, raw diff, no author framing, execute rather than only + read, mutation-test, report-don't-fix — is the part worth having written down, and + lives in the new kit-owned `docs/agentic-dev-kit/fallback-review-panel.md`. +- **`pr_followup_hook` now names the panel.** It fires on every `gh pr create`/`ready`, + which made it the most-read statement of fallback policy in the kit — and it was + advertising the degraded mode. +- **#32 filed** — the design that would actually verify coverage (each lens recording + its own receipt from its own context), with all four defeats written up. -## Latest session — 2026-07-25 (Phase 3b) +**Decided** + +- **Four tightenings of a matcher is the signal to delete it.** `safety-critical- + changes.md` rule 1 says treat "we tightened the matcher" as a stopgap. I tightened + it four times — source equality, lens names, a required roster, a counted roster — + and each was defeated by the next round, the last by one decorated character in a + field the caller writes themselves. What ships records the claim and labels it a + claim. +- **Report, never gate — now for a third field.** `--lenses` joins `signal`, + `bot_signal` and `coverage`. All four make an omission legible; none blocks. + +**Learned** + +- **Mutation testing this repo reports FALSE KILLS.** `kit_doctor`'s self-check + rehashes every kit-owned file, so any byte change fails it — a run can report 100% + killed while nothing behavioural caught anything. One lens's first pass reported + 17/17; excluded, 7 had survived. Verified directly. **This invalidates mutation + results from earlier rounds this session**, including some I cited as evidence. + Contract item 5 now warns about it; anything relying on "N mutants died" from + 2026-07-25 should be re-checked. +- **Two review lenses in one working tree corrupt each other.** One mutates files to + test them; the other reads that as external corruption and `git checkout --`s it. + Stopping one left a live mutant behind that silently disabled a guard. Contract + item 7: isolated worktrees. +- **Deleting a check can reintroduce a bug it was masking.** The roster check was the + only thing catching comma-as-punctuation in `--lenses`; removing it brought back the + exact forgery the commit before it claimed to block, plus an honest input + misrendering. +- **A silent `str.replace()` no-op let me assert a fix that never landed** — twice. + Every substitution in this session's later rounds reports MISS rather than passing + quietly. + +**Open, and owned by nothing yet** + +- **#32** — verifying lens coverage rather than self-reporting it. The honest version + of what #31 tried. +- **Adopter upgrades are written but not run.** `~/Documents/openkitchen-devkit-upgrade.md` + and `~/Documents/cs-toolkit-devkit-adoption.md`. Held at the operator's request until + #31 landed — it has now landed, so both are unblocked. + +▶ Next: **run the OpenKitchen upgrade** (`~/Documents/openkitchen-devkit-upgrade.md`). +It is a real adopter on a pre-#8 install that cannot migrate its own config until +`init.sh` and `kitconfig.py` are copied in, and its pre-push hook is not installed. +cs-toolkit is the larger job — an *adoption*, not an upgrade, plus a fixer change from +`done` to `converged` that will otherwise wedge an unattended nightly loop. + +______________________________________________________________________ + +## Earlier session — 2026-07-25 (Phase 3b) **Theme —** Fixed #19 + #23 together, and then spent most of the session discovering that **the fix rounds were more dangerous than the original diff**. Seven review rounds From 2da81791afbe8e02b3306c39f380a6a0e46e1d3e Mon Sep 17 00:00:00 2001 From: Topi Jarvinen Date: Sun, 26 Jul 2026 03:07:15 +0300 Subject: [PATCH 2/2] Correct ten errors the wrap-up fact-check found MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit I had the wrap-up fact-checked against `git log` / `gh` / the code rather than self-certifying it. Ten problems, two HIGH. **The handoff never mentioned #33.** That file is what `/session-start` reads, so a next session would not have learned that an open, H-severity, self-described *retroactive* issue exists — while the friction log called it the session's most consequential finding. Now in both the session block and "Open, and owned by nothing yet". **"Verified directly" was attached to the wrong claim.** I verified the drift check's *mechanism* — trivially reproducible. The *figure* (17/17 reported, 7 survivors when excluded) is a reviewing lens's report; the 17 mutants are enumerated nowhere. Both files presented it as first-party measurement, and the friction log's "Measured on #31" was the stronger overclaim. Now labelled attested. Three MEDIUM omissions, each of which would have misled: - the 7 survivors were **closed inside #31** (7/7 caught once real coverage was added) — without that, a reader concludes 7 mutants are live on `main` - the comma-as-punctuation regression is described in past tense with no resolution; it was **fixed inside #31**, so a next session could re-fix something already shipped - the **autonomous self-merge path never displays review coverage** — `dev_session.sh merge` gates on `mergeable` alone. That is the caveat that qualifies the session's headline decision, and it appeared in neither file Also: "a third field" where the same sentence names three predecessors; the unverifiable-artifact conclusion attributed to the adversarial lens when the commit records it in the author's voice; the invalidation scope now naming #31 itself rather than a date; and the friction log flagged as at budget, so a triage sweep is required before the next entry. Claude-Session: https://claude.ai/code/session_011KVJzDj7Z2nYkUjqbGgSVp --- docs/kit-friction-log.md | 23 +++++++++++++++-------- docs/kit-handoff.md | 32 +++++++++++++++++++++++++------- 2 files changed, 40 insertions(+), 15 deletions(-) diff --git a/docs/kit-friction-log.md b/docs/kit-friction-log.md index 2374b05..39c29d3 100644 --- a/docs/kit-friction-log.md +++ b/docs/kit-friction-log.md @@ -101,14 +101,21 @@ failure — closed by PR #28). ## 2026-07-26 — inbox +> **At the 150-line budget.** A `triage-friction-log` sweep is required before the +> next entry — the two H entries below are issue-shaped and #33 is already filed. + - **Mutation testing this repo reports false kills — filed as [#33](https://github.com/topij/agentic-dev-kit/issues/33) (severity: H).** `kit_doctor`'s self-check rehashes every kit-owned file, so any byte change to an - engine fails it and every mutant looks killed. Measured on #31: 17/17 reported - killed; with that test excluded, **7 had survived**. Filed at H because it is - **retroactive** — mutation evidence cited across #25, #28 and #29 this session may be - worthless wherever the reviewer did not exclude it, and "N mutants died" was used as - a reason to stop reviewing. A false-negative testing tool is worse than none, because - it is used to justify confidence. + engine fails it and every mutant looks killed. The **mechanism** is trivially + reproducible (change one comment in an engine; only the manifest test fails). The + **figure** — a lens reporting 17/17 killed on #31, and 7 survivors once that test + was excluded — is attested rather than independently measured; the 17 are enumerated + nowhere. Those 7 were closed inside #31. Filed at H because it is **retroactive**: + mutation evidence cited across #25, #28, #29 and #31 may be worthless wherever the + reviewer did not exclude that test, and "N mutants died" was used as a reason to stop + reviewing. A false-negative testing tool is worse than none, because it is used to + justify confidence. #33's mechanical fix (a `driftcheck` marker) is not built; only + the panel doc's prose warning ships. - **Concurrent review lenses in one working tree destroy each other's work (severity: H).** On #31 the adversarial lens mutated `pr_watch.py` to test it; the correctness lens, @@ -127,7 +134,7 @@ failure — closed by PR #28). `--lenses` — so `"adversarial, focused on the merge gate"` (an honest way to record ONE lens) rendered as two, suppressing the one-lens warning that was the field's entire remaining value. The commit that deleted it quoted that exact input as an - example of what it still blocked. **Lesson worth generalising:** when removing a + example of what it still blocked. **Fixed inside #31** — recorded for the pattern. **Lesson worth generalising:** when removing a mechanism as unfit, enumerate what it was rejecting and confirm each case is either still rejected elsewhere or deliberately allowed — the deletion commit did neither. @@ -135,7 +142,7 @@ failure — closed by PR #28). already says so (severity: M, pattern).** `safety-critical-changes.md` rule 1: *"Treat 'we tightened the matcher' as a stopgap, not a fix."* On #31 I tightened it four times before accepting that. The adversarial - lens told me in round 3 that the artifact was unverifiable from the engine — same + panel told me by round 3 that the artifact was unverifiable from the engine — same actor, same invocation, nothing bound to what ran — and I built two more epicycles before acting on it. **Proposed fix:** rule 1 could name a threshold ("a second tightening of the same matcher is a design signal, not a bug fix") so the decision diff --git a/docs/kit-handoff.md b/docs/kit-handoff.md index c17edf6..eed34d1 100644 --- a/docs/kit-handoff.md +++ b/docs/kit-handoff.md @@ -34,6 +34,11 @@ deletion is the result, not the failure. advertising the degraded mode. - **#32 filed** — the design that would actually verify coverage (each lens recording its own receipt from its own context), with all four defeats written up. +- **#33 filed, and it is the one to read first.** `kit_doctor`'s drift self-check + rehashes every kit-owned file, so *any* byte change to an engine fails it — which + makes a mutation-testing run report every mutant as killed while nothing behavioural + caught anything. Its proposed fix (a `driftcheck` pytest marker) is **not built**; + only a prose warning in the panel doc ships. **Decided** @@ -43,18 +48,22 @@ deletion is the result, not the failure. and each was defeated by the next round, the last by one decorated character in a field the caller writes themselves. What ships records the claim and labels it a claim. -- **Report, never gate — now for a third field.** `--lenses` joins `signal`, +- **Report, never gate — now for a fourth field.** `--lenses` joins `signal`, `bot_signal` and `coverage`. All four make an omission legible; none blocks. **Learned** - **Mutation testing this repo reports FALSE KILLS.** `kit_doctor`'s self-check rehashes every kit-owned file, so any byte change fails it — a run can report 100% - killed while nothing behavioural caught anything. One lens's first pass reported - 17/17; excluded, 7 had survived. Verified directly. **This invalidates mutation - results from earlier rounds this session**, including some I cited as evidence. - Contract item 5 now warns about it; anything relying on "N mutants died" from - 2026-07-25 should be re-checked. + killed while nothing behavioural caught anything. **I verified the mechanism + directly** — disabling a behaviour outright fails only the manifest test. The + *figure* "17/17 reported, 7 survived when excluded" is a reviewing lens's report, + restated: the 17 mutants are enumerated nowhere, so treat it as attested, not + measured. **This invalidates mutation evidence cited in #25, #28, #29 and #31 + itself** wherever the reviewer did not exclude that test. Contract item 5 now warns + about it; #33 tracks the mechanical fix. (Those 7 survivors were themselves closed + inside #31 — 7/7 caught by named tests once real coverage was added — so nothing is + known to be live on `main` because of this.) - **Two review lenses in one working tree corrupt each other.** One mutates files to test them; the other reads that as external corruption and `git checkout --`s it. Stopping one left a live mutant behind that silently disabled a guard. Contract @@ -62,7 +71,8 @@ deletion is the result, not the failure. - **Deleting a check can reintroduce a bug it was masking.** The roster check was the only thing catching comma-as-punctuation in `--lenses`; removing it brought back the exact forgery the commit before it claimed to block, plus an honest input - misrendering. + misrendering. **Fixed inside #31** (`_countable_lenses` counts entries that look like + lens names, not prose) — recorded for the pattern, not as a live defect. - **A silent `str.replace()` no-op let me assert a fix that never landed** — twice. Every substitution in this session's later rounds reports MISS rather than passing quietly. @@ -71,6 +81,14 @@ deletion is the result, not the failure. - **#32** — verifying lens coverage rather than self-reporting it. The honest version of what #31 tried. +- **#33** — the drift self-check's false kills. H severity and retroactive; its + mechanical fix (a `driftcheck` marker) is unbuilt. +- **The autonomous self-merge path never displays review coverage.** `dev_session.sh + merge` gates on `mergeable` alone, so the `review evidence:` line — the whole + remaining value of #31 — is invisible on exactly the path + `autonomous-session-playbook.md` argues review independence matters most on. Known + and documented in `workflows/pr-watch.md`, deliberately not fixed at the end of a + seven-round review; #32 is the real answer. - **Adopter upgrades are written but not run.** `~/Documents/openkitchen-devkit-upgrade.md` and `~/Documents/cs-toolkit-devkit-adoption.md`. Held at the operator's request until #31 landed — it has now landed, so both are unblocked.