feat(lint): gate tests/ on the prose rules, and cap the comment run as well as the file total

[uipreliga]

Applies the prose rules this repo enforces on src/coder_eval to tests/, turns the gate on for it, and — with tests/lint/ finally in scope — fixes the one gate rule that turned out to be measuring the wrong thing. Follows #177, which did the same for src/, and is rebased onto #176.

Four things happen here.

1. The gate now scans tests/ as well as src/

tests/lint/prose_budget.py scans a _ROOTS tuple instead of one hard-coded root, reports repo-relative paths, and fails when a root does not exist — in the scan and in --assert-code-unchanged, so a renamed root can never pass on zero files. _python_files is the single traversal; main prints collect_failures(). Phase 8 flips the default to (src/coder_eval, tests).

That put the essay police on their own beat for the first time. 67 docstrings over 150 prose words and 5 files over their comment budget, cleared:

Phase	Scope	Result
3	30 rule docstrings in tests/lint/rules/	8,347 → 3,328 prose words
4	11 doc-surface lint tools + 4 TestCExxx classes	4,976 → 1,928 prose words
5	_scrub.py, _bracket_clock.py, opencode_fixtures.py	5 essays cleared
6	14 plain test modules	15 essays + 3 comment budgets
7	all of tests/	~120 HISTORY lines, 16 long test docstrings

Across the branch, tests/lint/ goes from 18,265 to 11,668 docstring prose words and all of tests/ from 105,305 to 95,722.

Each docstring keeps its contract and hazards; the defect story moves behind a resolving Rationale: <path> § <heading> pointer. Rule rationale now lives in the new .claude/notes/lint-rules.md (one section per CE id, split verbatim out of .claude/notes/README.md); test rationale goes to the subsystem note it belongs to.

Dangling citations fixed: harbor/packager.py and tests/test_harbor_packager.py cited a module docstring that no longer held the claim; tests/test_harbor_agent.py cited one that never did.

Prose-only, proved

Every prose phase passes --assert-code-unchanged, which compares the docstring-stripped AST and the directive comments against the base.

Gate holes the final review found, and closed here

Three ways the tool could pass without checking what it claims (each fixed test-first):

• assert_code_unchanged compared directive comments one way, so adding a # noqa passed a "prose-only" proof.
• main() ran the ordinary gate for any argv, so a mistyped --assert-code-unchanged exited 0 having proved nothing.
• The file list came from git diff --name-only, which never names an untracked file, so a new .py file full of code passed.

Two further gaps needed design work and are deferred in .claude/harness-candidates.md: narrative after an Args: block that follows a pointer, and pointer targets outside .claude/notes/.

2. Rebased onto #176, which brought five new essays

396c22cc (the reports split) landed three new modules and reworded two more, all over the 150-word bar once this branch's gate sees them:

tests/lint/rules/_layers.py                       432w -> 137w
tests/lint/rules/ce066_no_report_imports_in_core   313w -> 121w
tests/lint/pricing_mirror.py                       311w -> 143w
tests/test_custom_lint.py::TestRuffExternal...     246w ->  93w
tests/lint/rules/no_cli_imports_in_core.py         235w -> 117w

New notes sections carry what moved: CE004, CE065, CE066, _layers, TestRuffExternalCoversEveryRule. Three merge conflicts resolved: ce053's docstring (kept the slimmed side), harness-candidates.md (kept both appends) and the Makefile (kept pricing-mirror and the new docs-budget help text).

3. CE023 deleted — it guarded a package that no longer exists

src/coder_eval/proxy/ is gone; CHANGELOG.md:2487 records deleting the coder_eval.proxy.pricing shim. The rule banning imports of it stayed behind, so it could not fire, and its docstring still described the shim as present.

Removed from all five places a rule lives: the module, its runner.py import and ALL_RULES entry, its test class, and its id in [tool.ruff.lint] external. The id stays retired, as runner.py already requires for CE062 — both are now recorded in .claude/notes/lint-rules.md § The CE id space, which replaces the 14-line comment that used to carry it.

46 → 45 registered rules.

4. The comment budget was inverted, so the rule is now the comment RUN

With tests/lint/ in scope I measured what MAX(20, 0.15 × lines) on a file's total comment lines was actually doing. It was backwards:

file	shape	old rule
isolation/docker_runner.py	229 comments, 62 runs of 1–2 lines, longest run 6	229/229 — blocked
tests/test_regrade.py	one 16-line paragraph	29% of budget — fine
src/coder_eval/criteria/base.py	one 13-line paragraph	31% of budget — fine

It taxed short annotations pinned to the line they explain, and waved through the paragraphs that .claude/notes/ exists to hold. Four files had settled at exactly 100.0% of budget (docker_runner.py 229/229, criteria/agent_judge.py 65/65, evaluation/sub_agent.py 34/34, config.py 33/33) — nobody lands on 229/229 by accident; the budget had become a target.

So a third rule is added, and it is the primary one:

No own-line comment run may exceed 8 lines.

A run reads through one blank line, so splitting a paragraph does not duck it, and through none at two, which is the separation PEP 8 already puts between a banner and its section. It caps a shape, with no per-file allowance: carry as many one-line notes as earn their place.

Measured before choosing 8: 78.5% of runs in the tree are 1–2 lines, 2.5% are 6 or more, and the longest was 16. The cap is below the current maximum, as a cap rule must be. 24 runs across 19 files were shortened or moved to .claude/notes/ behind a pointer — 6 in src/, 18 in tests/.

The file total is kept as the backstop underneath, unchanged at MAX(20, 0.15 × lines): a file can pass the run cap with every block short and still be mostly commentary. test_it_catches_what_the_run_cap_cannot pins that seam — thirty three-line blocks pass the run cap and fail the total — so neither rule can later be deleted as redundant without a red test.

make docs-budget now enforces three rules on both roots: comment run ≤ 8 lines, file total ≤ MAX(20, 0.15 × lines), docstring ≤ 150 prose words.

Worth a human's eye

• The four files still at 100% of their total budget are frozen: a comment added to one has to be paid for by deleting another, or by raising the ratio. Recorded in .claude/notes/README.md with the measurements, so the next person to hit it sees the trade rather than bumping the number blind.
• src/ is touched, in 6 files — all of them one over-long comment run each, moved to notes behind a pointer. No behaviour change.
• .claude/notes/README.md — docs: move design rationale out of src/ into .claude/notes, gate it, and slim CLAUDE.md #177 had rewritten that section to say no prose summary is kept there; this branch points it at lint-rules.md. The rule file owns invariant/scope/blind spots, lint-rules.md owns the motivating defect, and the rule file wins on disagreement.
• Prose moved rather than shrank. tests/ loses ~12,700 docstring words and .claude/notes/ gains ~12,400. .claude/ has no prose budget, so the repo total is roughly flat; that is the design (rationale has one home) but it is worth stating plainly rather than claiming a reduction.

Verification

make verify: 5,816 passed, 2 skipped, 92.63% coverage, 671 custom lint rules — ruff format, ruff check, pyright, lint suite, prose gate, full test run. The gate reports 0 essays, 0 over-long runs and 0 files over budget with both roots in scope.

🤖 Generated with Claude Code

[github-actions]

Claude finished @uipreliga's task in 1m 31s —— View job

---

Code Review In Progress

Todo List:

• Read code review guidelines from .github/code_review.md
• Read project conventions from CLAUDE.md
• Get full PR diff with git diff origin/main...HEAD
• Review key changed files:
  • tests/lint/prose_budget.py (the gate implementation)
  • .claude/notes/lint-rules.md (new rule rationale file)
  • Sample rule files to verify prose cleanup
• Perform cross-file consistency checks
• Check "what's missing" analysis
• Verify prose-only proof claim
• Write comprehensive review feedback

Starting review now...

[bai-uipath]

Approve — execution is clean: the three holes in the prose-only proof were real and are fixed test-first, the CE023 deletion is complete in all five places a rule lives, and I verified the prose-only claim independently against main (the six src/ files are AST-identical). The notes below are about the disposal mechanism, not the gate.

• The bin isn't free once CI pins it. Nearly 500 Rationale: pointers now fail make docs-budget when a target file or heading goes missing, so .claude/notes/ can no longer be emptied, re-headed or pruned without going red. That's the opposite of what a scratch directory is for. Fix: pick one — drop pointer resolution from the gate and let the notes be an actual bin, or delete the narrative outright instead of relocating it. Moving ~12,400 words into a directory CI won't let you clean out is the worst of both.
• lint-rules.md is a second catalogue with a tie-break clause. It reverses the "no prose summary is kept here" rule written one merge earlier, and "the rule file wins on disagreement" concedes they'll diverge. The one-sentence defect summary the slimmed rule docstrings already carry is the part anyone editing a rule actually needs. Not a blocker, but decide this one separately from the rest of the diff.
• The run cap is already saturated. 21 runs sit at exactly 8 lines and none above, so the "a budget with an allowance becomes a target" argument used to replace the file total applies to its replacement on day one. Each of those is one line from a gate failure on an unrelated edit.

Minor, all optional: two blank lines still split a run at module scope, so the paragraph split the docstring says can't duck the cap partly can (ruff format closes it inside functions only); one new pointer in the harbor packager resolves to a notes section that doesn't discuss the thing it cites; a renamed root raises a traceback rather than a gate failure line.

Worth splitting the CE023 deletion and the run-cap rewrite out of prose PRs in future — both are small, independently reviewable, and get lost in 139 files.

[uipreliga]

Thanks for the review. I checked each point against the tree. Fixes are in ed1c01c9 (prose only) and d20d40e6 (gate).

1. Pointer resolution makes .claude/notes/ impossible to prune — not changed.
.claude/notes/ is not a scratch directory. CLAUDE.md makes it the home for design rationale, and the code points into it. The resolution check stops dangling citations, which were a real defect in this PR (479f1a3f had to repair them). When you rename a heading, the gate names each pointer that breaks, and one sed fixes them. I agree with the other half: a paragraph that only records history should be deleted, not moved. That is a per-paragraph decision, not a reason to drop the check.

2. lint-rules.md as a second catalogue — fixed.
The file holds the defect stories moved out of the rule docstrings, not a summary of them. But you were right that the content split was not clean: many sections also restated the scope, exemptions and blind spots. I removed each such sentence where the rule file already states it (about 2,000 words over most sections). Where the rule file does not state it, I kept it, so nothing is lost. I deleted the tie-break clause from README.md. No heading changed, and every Rationale: pointer still resolves.

3. The run cap is already full — fixed.
Confirmed: 21 runs sat at exactly 8 lines. I cut all 21 to 7 lines or fewer, and the cap stays at 8. The longest run is now 7 lines. The notes record the change. The file total still has four files at 100% (config.py, criteria/agent_judge.py, evaluation/sub_agent.py, isolation/docker_runner.py). The notes already say this, and I did not change it here.

Minor notes — all confirmed and fixed:

• Two blank lines at module scope do split a run: 4 lines, two blanks, then 5 lines measure as [(1,4),(7,5)]. It is now a stated blind spot in the own_comment_runs docstring, and CLAUDE.md no longer claims more than the gate does.
• The pre_run pointer in harbor/packager.py pointed to § Harbor export, which does not mention pre_run. No section covers it, so I removed the pointer. The comment explains itself.
• A renamed root now prints one line and exits 1, with no traceback. There is a new test for it.

Splitting PRs: agreed for the future.

make verify passes, and --assert-code-unchanged reports code changes only in tests/lint/prose_budget.py and tests/test_prose_budget.py.

🤖 Generated with Claude Code