Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions docs/kit-friction-log.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,3 +98,53 @@ failure — closed by PR #28).
that made the fix obvious. Worth keeping in mind when weighing whether a guard is worth
its friction — this one cost ten seconds and prevented a narrative-file conflict with the
wrap-up PR. No change proposed.

## 2026-07-26 — inbox

> **At the 150-line budget.** A `triage-friction-log` sweep is required before the
> next entry — the two H entries below are issue-shaped and #33 is already filed.

- **Mutation testing this repo reports false kills — filed as [#33](https://github.com/topij/agentic-dev-kit/issues/33) (severity: H).**
`kit_doctor`'s self-check rehashes every kit-owned file, so any byte change to an
engine fails it and every mutant looks killed. The **mechanism** is trivially
reproducible (change one comment in an engine; only the manifest test fails). The
**figure** — a lens reporting 17/17 killed on #31, and 7 survivors once that test
was excluded — is attested rather than independently measured; the 17 are enumerated
nowhere. Those 7 were closed inside #31. Filed at H because it is **retroactive**:
mutation evidence cited across #25, #28, #29 and #31 may be worthless wherever the
reviewer did not exclude that test, and "N mutants died" was used as a reason to stop
reviewing. A false-negative testing tool is worse than none, because it is used to
justify confidence. #33's mechanical fix (a `driftcheck` marker) is not built; only
the panel doc's prose warning ships.

- **Concurrent review lenses in one working tree destroy each other's work (severity: H).**
On #31 the adversarial lens mutated `pr_watch.py` to test it; the correctness lens,
running at the same time, saw those mutations as an external process corrupting the
repo and ran `git checkout --` to "restore" it. They fought for ~10 minutes. One
lens's results were unreliable; when I stopped the other it left a live mutant behind
(`if False and _PANEL_LENS_NAMES:`) that **silently disabled a guard in my working
tree**, and I caught it only because a test failed citing an error string that should
no longer have existed. Already fixed as contract item 7 (isolated worktrees) and
`.gitignore`/`init.sh` entries, so this is recorded for the pattern rather than as an
open item: **any doctrine that says "run N reviewers concurrently" owes them
isolation**, and mine did not until it bit.

- **Deleting a check reintroduced the bug it was masking (severity: M).**
The roster check removed from #31 was the only thing catching `,` as punctuation in
`--lenses` — so `"adversarial, focused on the merge gate"` (an honest way to record
ONE lens) rendered as two, suppressing the one-lens warning that was the field's
entire remaining value. The commit that deleted it quoted that exact input as an
example of what it still blocked. **Fixed inside #31** — recorded for the pattern. **Lesson worth generalising:** when removing a
mechanism as unfit, enumerate what it was rejecting and confirm each case is either
still rejected elsewhere or deliberately allowed — the deletion commit did neither.

- **Four rounds of tightening a matcher is the signal to delete it, and the rule
already says so (severity: M, pattern).**
`safety-critical-changes.md` rule 1: *"Treat 'we tightened the matcher' as a stopgap,
not a fix."* On #31 I tightened it four times before accepting that. The adversarial
panel told me by round 3 that the artifact was unverifiable from the engine — same
actor, same invocation, nothing bound to what ran — and I built two more epicycles
before acting on it. **Proposed fix:** rule 1 could name a threshold ("a second
tightening of the same matcher is a design signal, not a bug fix") so the decision
point is written down rather than requiring the author to notice it.

90 changes: 87 additions & 3 deletions docs/kit-handoff.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,94 @@
> Older session blocks graduate to [`kit-handoff-history.md`](kit-handoff-history.md) once
> this file crosses its line budget (`scripts/check_doc_budget.py`).

Last updated: 2026-07-25 — Phase 3b shipped: the queued-vs-unavailable ambiguity is
closed on both surfaces, and the receipt now says what it does *not* cover.
Last updated: 2026-07-26 — the fallback review is a panel, and the attempt to make
its claim machine-verifiable was built four times and then deleted.

## Latest session — 2026-07-26 (#26, overnight)

**Theme —** Built `review.fallback_panel`, then spent four rounds trying to make the
receipt's coverage claim verifiable *by the engine*, then deleted all of it. The
deletion is the result, not the failure.

- **#31 merged, closing #26.** `review.fallback_panel` is the primary substitute when
a bot can't review: one isolated, fresh-context reviewer per lens.
`fallback_commands` stays as the explicitly degraded one-lens mode. The lens
*contract* — fresh context, raw diff, no author framing, execute rather than only
read, mutation-test, report-don't-fix — is the part worth having written down, and
lives in the new kit-owned `docs/agentic-dev-kit/fallback-review-panel.md`.
- **`pr_followup_hook` now names the panel.** It fires on every `gh pr create`/`ready`,
which made it the most-read statement of fallback policy in the kit — and it was
advertising the degraded mode.
- **#32 filed** — the design that would actually verify coverage (each lens recording
its own receipt from its own context), with all four defeats written up.
- **#33 filed, and it is the one to read first.** `kit_doctor`'s drift self-check
rehashes every kit-owned file, so *any* byte change to an engine fails it — which
makes a mutation-testing run report every mutant as killed while nothing behavioural
caught anything. Its proposed fix (a `driftcheck` pytest marker) is **not built**;
only a prose warning in the panel doc ships.

## Latest session — 2026-07-25 (Phase 3b)
**Decided**

- **Four tightenings of a matcher is the signal to delete it.** `safety-critical-
changes.md` rule 1 says treat "we tightened the matcher" as a stopgap. I tightened
it four times — source equality, lens names, a required roster, a counted roster —
and each was defeated by the next round, the last by one decorated character in a
field the caller writes themselves. What ships records the claim and labels it a
claim.
- **Report, never gate — now for a fourth field.** `--lenses` joins `signal`,
`bot_signal` and `coverage`. All four make an omission legible; none blocks.

**Learned**

- **Mutation testing this repo reports FALSE KILLS.** `kit_doctor`'s self-check
rehashes every kit-owned file, so any byte change fails it — a run can report 100%
killed while nothing behavioural caught anything. **I verified the mechanism
directly** — disabling a behaviour outright fails only the manifest test. The
*figure* "17/17 reported, 7 survived when excluded" is a reviewing lens's report,
restated: the 17 mutants are enumerated nowhere, so treat it as attested, not
measured. **This invalidates mutation evidence cited in #25, #28, #29 and #31
itself** wherever the reviewer did not exclude that test. Contract item 5 now warns
about it; #33 tracks the mechanical fix. (Those 7 survivors were themselves closed
inside #31 — 7/7 caught by named tests once real coverage was added — so nothing is
known to be live on `main` because of this.)
- **Two review lenses in one working tree corrupt each other.** One mutates files to
test them; the other reads that as external corruption and `git checkout --`s it.
Stopping one left a live mutant behind that silently disabled a guard. Contract
item 7: isolated worktrees.
- **Deleting a check can reintroduce a bug it was masking.** The roster check was the
only thing catching comma-as-punctuation in `--lenses`; removing it brought back the
exact forgery the commit before it claimed to block, plus an honest input
misrendering. **Fixed inside #31** (`_countable_lenses` counts entries that look like
lens names, not prose) — recorded for the pattern, not as a live defect.
- **A silent `str.replace()` no-op let me assert a fix that never landed** — twice.
Every substitution in this session's later rounds reports MISS rather than passing
quietly.

**Open, and owned by nothing yet**

- **#32** — verifying lens coverage rather than self-reporting it. The honest version
of what #31 tried.
- **#33** — the drift self-check's false kills. H severity and retroactive; its
mechanical fix (a `driftcheck` marker) is unbuilt.
- **The autonomous self-merge path never displays review coverage.** `dev_session.sh
merge` gates on `mergeable` alone, so the `review evidence:` line — the whole
remaining value of #31 — is invisible on exactly the path
`autonomous-session-playbook.md` argues review independence matters most on. Known
and documented in `workflows/pr-watch.md`, deliberately not fixed at the end of a
seven-round review; #32 is the real answer.
- **Adopter upgrades are written but not run.** `~/Documents/openkitchen-devkit-upgrade.md`
and `~/Documents/cs-toolkit-devkit-adoption.md`. Held at the operator's request until
#31 landed — it has now landed, so both are unblocked.

▶ Next: **run the OpenKitchen upgrade** (`~/Documents/openkitchen-devkit-upgrade.md`).
It is a real adopter on a pre-#8 install that cannot migrate its own config until
`init.sh` and `kitconfig.py` are copied in, and its pre-push hook is not installed.
cs-toolkit is the larger job — an *adoption*, not an upgrade, plus a fixer change from
`done` to `converged` that will otherwise wedge an unattended nightly loop.

______________________________________________________________________

## Earlier session — 2026-07-25 (Phase 3b)

**Theme —** Fixed #19 + #23 together, and then spent most of the session discovering
that **the fix rounds were more dangerous than the original diff**. Seven review rounds
Expand Down
Loading