refactor(reports): generate the pricing mirror and split the reports layer

[uipreliga]

Implements c/2026-09-14-reports-consolidation.md (five phases), then c/2026-09-14-reports-consolidation-review-fixes.md (four phases) for the residual issues a review of the first found. 15 commits.

What changes

1. Pricing is generated, not copied. evalboard/lib/pricing.generated.ts is rendered from pricing.py by make pricing-mirror and guarded by CE065. Deletes the regex parser, its meta-guard, the DELIBERATELY_UNMIRRORED set, that set's staleness guard, and the comment asking the next reader to keep it honest — five layers of bookkeeping that still shipped a real bug: claude-sonnet-5, gpt-5.6-sol, gpt-5.6-terra and gpt-5.6-luna sat in the exemption set under "the evalboard never runs them" while appearing ~32k / ~2k / ~17k / ~2k times in the run corpus, so every one of those runs rendered — for cost with nothing failing. The one rule the set encoded that was not bookkeeping — that three OpenRouter models must stay unpriced so runs.ts apportions the provider's real bill — becomes ModelPricing.per_request_billing, data beside the rate it qualifies.

2. Two verified DRY fixes, and 18 of 19 function-local imports hoisted. SLOW_PARAMS_PREVIEW_CHARS and TokenUsage.total_tokens each have one definition again. The 19th import (criteria, which runs pkgutil auto-discovery with registry side effects) stays deferred and is pinned by a subprocess test rather than remembered.

3. reports_stats.py splits three ways. It was three unrelated modules sharing a file, and the orchestrator imported turn_time_buckets / visible_turn_count from it during a run. Now: stats.py (dependency-free by contract — asserted by parsing its AST), result_metrics.py (what the orchestrator reads mid-run), and the report helpers stay. The run.json row serializer moves to run_record.py; its old home was the only reason orchestration/batch.py reached into the reports layer at all.

4. reports/ becomes a package (markdown / html / experiment / junit / helpers), format_ms moves to durations.py, and CE066 pins the layering: core may import only the package's public writers. An allowlist, not a denylist. The package publishes 16 names, not 25: nine were only ever imported by sibling modules, which now import each other directly.

5. Documentation truth-up, plus a ## The reports package section in .claude/architecture-notes.md recording the layering, why each extracted module sits where it does, and the rejected section-data layer with the measurement behind the rejection (only 1 of 4 "duplicated" section pairs shares an input shape).

6. The layering rules now see the whole core layer. CE004 and CE066 share one predicate, _layers.is_core_path. It was a denylist — ten directory names plus a top-level-module regex — and isolation/ was not on it, so isolation/docker_runner.py, the driver: docker evaluation path, was invisible to both rules. It is now an anchored allowlist: everything under src/coder_eval/ is core except cli/ and reports/, so a new subpackage is core by default. Measured over the real tree: +3 core files (isolation/__init__.py, isolation/docker_runner.py, resources/__init__.py), 0 removals, 0 new violations. TestCoreLayerMembership pins the non-core set against the filesystem.

7. Lint-id bookkeeping is enforced, and CLAUDE.md's tree with it.

   • [tool.ruff.lint] external was missing CE044 and CE065. Both are @pytest.mark.lint classes, and the existing guard derived rule ids from ALL_RULES alone — blind to exactly that half. It now unions both registries and asserts both directions, so a stale id for a deleted rule fails too.
   • New CE067: CLAUDE.md's directory tree must name every top-level package member, and must name nothing that does not exist. The tree had drifted both ways — a phantom optimize/ row, and missing errors/ and plugins.py (the plugin SPI "Adding a New Agent" points at). All three are fixed.

Notable

• All 21 definitions moved in Phase 3 are byte-identical to their pre-move form, verified by AST comparison. No rendered report byte changes; a characterization test pins the full 47-key run.json row against a pre-move snapshot.
• Generating the table deliberately adds gpt-5.4-mini, gpt-5.4-nano, gpt-5.4-pro and gpt-5.5-pro to the frontend — those four were pure drift bookkeeping, not a product rule.
• Fixes a latent packaging break found in passing: .gitignore's bare reports/ (meant for run output) also matched src/coder_eval/reports/, leaving __init__.py untracked and producing a wheel with zero files under coder_eval/reports/.
• CE004 had the same import blindness as CE066. It matched only node.module, so from ..cli import x — this codebase's dominant idiom — fired on nothing, and had since the rule was written. Both rules now share one matcher covering all three spellings: absolute, relative, and from coder_eval import cli, which binds the package itself and was missed longest.
• Two lint-harness tests were silently passing or not firing, and are fixed: four CE004 fixtures built paths with no src/ segment (three stopped firing under the anchor, one passed vacuously), and nothing pinned the trailing separator that keeps a module like reports_legacy.py core — deleting it left all 738 tests across test_custom_lint.py and test_lint_runner.py green.
• The HTML slowest-commands truncation branch (reports/html.py:825) was the one uncovered line in the renderer. It now has a truncation and a boundary test, both mutation-checked.
• TestRuffExternalCoversEveryRule now carries the lint mark like its siblings. make verify and the Ubuntu lint job still run it; the Windows job (-m "not live and not lint") no longer does.
• Two deferred items are recorded in .claude/harness-candidates.md rather than done here, because each changes a rule's scope: CE004 inherits CE066's reports/ exemption through the shared predicate (latent — nothing in reports/ imports cli), and the anchored src/coder_eval/ regex is still duplicated across seven rule modules.

Plan errors, corrected by measurement

• The first plan's criterion "import coder_eval.reports does not import claude_agent_sdk" is unachievable: models/agent_config.py imports ClaudeAgentOptions and every report module needs models. The tests assert what is true instead — durations.py is SDK-free, and reports no longer imports the SDK-shaped formatting module.
• The first plan's "exactly four names from .stats" contradicted its own Changes body; five is correct (fmt_mean_sd needs stddev).
• The second plan would have added a new parity class beside TestRuffExternalCoversEveryRule, leaving two checks over one list. The existing class was extended instead.

Verification

At 7a8951f5: ruff format/check clean · pyright 0 errors (src + tests configs) · 673 custom lint · 5730 tests at 92.66% coverage · evalboard 803 tests / 54 files / build ✓. At 0fabe3fb, before the follow-up commits: make pricing-mirror idempotent on a clean tree · wheel ships all 6 coder_eval/reports/ files. No follow-up commit touches src/, .gitignore or evalboard/.

🤖 Generated with Claude Code

[github-actions]

Claude finished @uipreliga's task in 1m 18s —— View job

---

Code Review in Progress

Todo List:

• Read .github/code_review.md for review standards
• Read CLAUDE.md for project conventions
• Run git diff origin/main...HEAD to see full changes
• Review architectural changes (pricing mirror generation, reports package split)
• Check cross-file consistency
• Verify custom lint rules (CE004, CE065, CE066)
• Review test coverage and characterization tests
• Complete "What's Missing" analysis
• Provide comprehensive review feedback

Starting code review now...

[uipreliga]

@claude please review this PR in full, at head 2f9bb50f (17 commits against main). The previous review stopped at "Code Review in Progress" and never posted findings. It also ran on 0fabe3fb, so it did not see the last 9 commits: the core-layer predicate allowlist, the CE004 scope fix, the ruff external sensor, CE067, and the HTML truncation tests.

[github-actions]

Claude finished @uipreliga's task in 1m 28s —— View job

---

Code Review in Progress

Todo List:

• Read .github/code_review.md for review standards
• Read CLAUDE.md for project conventions
• Get full diff with git diff origin/main...HEAD
• Review changed files with full context
• Perform cross-file consistency checks
• Analyze design decisions and what's missing
• Format and post comprehensive review findings

Starting review of PR #176 at commit 2f9bb50f (17 commits)...

[bai-uipath]

Fix what you agree with, otherwise lgtm. The split itself is clean, and most of the +3187 is relocation rather than new code. The layering work is the real find here: CE004 has been dead since it was written, and CE066 encodes violations that exist today rather than hypothetical ones.

Worth cutting

• The pricing mirror. The four ids it adds to the frontend table were not drift: they were enumerated exemptions in the parity test this PR deletes, which carried its own stale-membership guard. The generator keeps only per-request billing as an exemption axis, so it reproduces three of the seven exclusions and silently overrides the other four. Latent today, since no experiment config references those models. Fix: carry the exemption set into the mirror, or drop the mirror and add the four ids to the frontend table.
• durations.py. format_ms was already shared before this PR: both renderers imported it from the formatting module, so there was no duplication to remove. The module's own stated rationale also doesn't hold, since the reports package still pulls the SDK in through models. Fix: leave the function where it was, delete the module and its test.
• CE067 and the architecture-notes expansion. Neither is about reports, and the notes additions land in a file the docs PR deletes.

Before merge

• The plugin's framework-contract test. coder_eval_uipath/eval_runner/tests/skills/test_framework_contract.py importorskips the renamed module, so the only guard on the run.json row shape goes green by skipping instead of failing. One-line companion.
• Pick a minor bump at dispatch. Four public modules are deleted and reports becomes a package with no aliases, while the release default is patch.

Minor, all optional: reports/helpers.py inherits a sixth of the module this PR dissolves, under a name that invites the same accumulation; the layering rule's relative matcher is depth-blind (latent, nothing nested today); importing the reports package now pulls all five submodules for any one name.

Land this before the docs PR either way: that PR's prose gate rejects two of the modules added here.

[uipreliga]

Thanks — went through each item. Updated after rebasing onto main, which had moved under this branch (#174/#177) and settled two of these by itself.

Fixed

The pricing mirror — you were right, and I was wrong to push back. Fixed in 2c47ee5.

My earlier reply argued the generated table was a harmless superset, because the four ids carried a don't bother rationale rather than a correctness constraint. That was the wrong test. Whatever the rationale, the hand-copy did not price them and the generator did, so the refactor changed frontend behaviour — latent only because no experiment config references those models today. Reproducing the surface is the generator's job; deciding it may as well be wider was not a call to make inside this PR.

What survives from that reply is that the exemption set encoded two different claims, so they now survive separately rather than collapsing back into one list:

• A routed model must not be priced is a fact about the RATE — an estimate would displace the captured actual per-call cost. Stays on ModelPricing.per_request_billing.
• A heavy frontier variant is not worth pricing on the board is a fact about the FRONTEND. That is DELIBERATELY_UNMIRRORED, beside the generator.

The generated table now reproduces the hand-copy's 52 keys exactly — I diffed the key sets against main: no additions, no removals.

On my "carrying it forward reinstates the bookkeeping that shipped the bug": half right, and it pointed at the wrong fix. What shipped the claude-sonnet-5 / gpt-5.6-* bug was not having an exemption set but letting one go stale unnoticed. So the deleted test's staleness guard comes back as _assert_exemptions_are_live — an id that has left pricing.py now fails make pricing-mirror and CE065, instead of surviving as a comment asking the next reader to re-check it. CE065 asserts both axes from their declared sources, and the evalboard's consumption test pins the four as unpriced.

The layering rule's relative matcher is depth-blind. Real, fixed in 4fb00b9. imports_package compared node.module against the bare package name whenever node.level was non-zero, so from .reports import x inside orchestration/ — i.e. coder_eval.orchestration.reports — read as the reports layer, and from ...reports import x from a sub-package read as it too although it escapes coder_eval entirely. _absolute_module now resolves the dots against the importing file's own package, so both rules compare one fully-qualified name in either spelling. Pinned for both spellings.

One consequence worth naming: test_ce004_scope_is_every_module_outside_cli probed every module path with from ..cli import run_command, which after this fix is correct only from inside a sub-package. It moved to the absolute spelling — that test measures scope, and a relative probe measures depth.

CE067 and the architecture-notes expansion. CE067 is deleted, and main decided it rather than I did. I had argued it should land beside the drift it catches — but #174/#177 replaced CLAUDE.md's fenced directory tree with ls plus selective bullets, and CE067's whole invariant was "that tree names every top-level package member". The invariant is now false by design, and the bullets that replaced it are deliberately not exhaustive, so rebuilding the rule would fight a deliberate upstream choice. The half that is still real — no bullet may name a path that does not exist — is recorded in .claude/harness-candidates.md instead.

The notes expansion moved with the same restructure: .claude/architecture-notes.md is gone, so the reports-layer rationale now lives in .claude/notes/reporting.md, and I retargeted the stale reports_stats / reports_junit references in notes/timing.md and notes/orchestration.md while there.

Your prose-gate prediction also landed exactly. make docs-budget failed on the two modules you named: run_record.py at 40 own-line comments against a budget of 32, and result_metrics.py::_turn_tool_union_ms as a 198-word docstring essay. Both now carry main's own trimmed wording for the prose they inherited, including its .claude/notes/ pointers.

The plugin's framework-contract test. Right, and a plain rename would not have fixed it. coder-eval-uipath pins >=0.9.5,<0.12.0, so pointing importorskip at coder_eval.run_record makes it skip today instead of after the pin bump — the same silent green, just moved. The companion instead gates the skip on the optional dependency (coder_eval), resolves the serializer from run_record with a reports_experiment fallback while the pin still allows a pre-split release, and hard-fails when it is in neither home. Verified green against the currently installed pre-split coder-eval and against this branch's layout. It ships as its own PR on that repo.

Your call

The bump. Agreed on the reasoning — four public modules deleted, no aliases. Owner's call is patch: nothing outside this repo imports them, and the one in-house consumer is the contract test above, which now resolves either name.

Declining

durations.py. Your premise is correct and I had it wrong in the commit framing: format_ms was already shared on main, both renderers importing it from formatting.py, so the move removed no duplication. But the module never claimed to. What it claims is narrower and, I think, still true: durations.py is SDK-free, and reports no longer reaches through an SDK-shaped module for a 14-line formatter. It states in its own docstring that this does not make the package SDK-free, because models/agent_config.py imports ClaudeAgentOptions — and the tests assert only the narrow claim. So the rationale holds as written; reverting is churn with no correctness gain. Happy to drop it if you'd still rather not carry the module.

reports/helpers.py. Fair, but a rename doesn't stop accumulation — splitting it by consumer would, and that's follow-up work rather than a merge blocker.

Eager submodule import. That's what a package __init__ re-exporting a public surface does. Lazy loading would trade a real import cost for __getattr__ indirection, and nothing here sits on a hot path.

The CodeQL duplicate-json comment was already handled in 8fb5c2c.

---

Edited after the rebase: the pricing-mirror item moved from Declining to Fixed, retracting my earlier disagreement on it, and the CE067 item moved too — main deleted its premise. durations.py is the one item I am still declining.