Skip to content

score: add harness-enforcement dimensions (Claude Code hooks, evals, memory, cost, GitHub hygiene, deploy targets) #311

Description

@stackbilt-admin

Problem

charter score grades six categories — Agent config, Grounding, Architecture, Testing, Governance, Freshness. It measures whether agent instructions are well-formed, but under-measures whether the harness around them is enforced.

Concretely, on a real repo (aegis, scoring 87/100 B today):

  • Claude Code hooks are invisible. detectHookFiles in packages/cli/src/commands/score.ts reads only .husky/, .githooks/, .git/hooks/ and core.hooksPath. A repo with seven PreToolUse/PostToolUse/Stop hooks wired in .claude/settings.json scores identically to one with none. aegis reports "1 hook(s) wired" — that is the git pre-commit, counted correctly — while seven Claude Code hooks in .claude/settings.json stay invisible to it.
  • No GitHub hygiene checksPULL_REQUEST_TEMPLATE.md, CODEOWNERS, ISSUE_TEMPLATE/ are not looked at. All three are missing in aegis and nothing surfaces it.
  • No Eval Coverage, Memory Persistence or Cost Efficiency category.
  • Deploy targets are not a category. wrangler.toml appears in score.ts only as a config file feeding freshness, never as a Cloudflare-integration check. Same for vercel.json, netlify.toml, fly.toml.

Prior art

affaan-m/ECC (MIT, 934195f) ships scripts/harness-audit.js: a zero-dependency, deterministic 12-category rubric with --format json, reproducible per commit, and a distinct "consumer" mode for repos that use a harness rather than ship one. Its categories are Tool Coverage, Context Efficiency, Quality Gates, Memory Persistence, Eval Coverage, Security Guardrails, Cost Efficiency, GitHub Integration, plus deploy-target categories that activate only when a marker file is present (Vercel, Netlify, Cloudflare, Fly).

Run against aegis it returns 28/39 with five failing checks — three genuinely actionable (the three GitHub hygiene files above), two specific to ECC's own install layout.

The conditional-category design is the part worth borrowing: max_score varies with which categories apply, so a Worker repo is not penalised for lacking vercel.json.

Proposal

  1. Teach detectHookFiles to read .claude/settings.json (and settings.local.json) hook entries alongside git hooks, reporting the two separately — they enforce different things at different moments.
  2. Add GitHub hygiene checks (PULL_REQUEST_TEMPLATE.md, CODEOWNERS, ISSUE_TEMPLATE/) to the Governance category.
  3. Add deploy-target checks that activate on a marker file, so max_score stays proportional to what the repo actually is.
  4. Consider Eval Coverage and Cost Efficiency as new categories, or as signals inside Testing and Governance.

Items 1–3 are additive and do not change existing scores for repos that already pass. Item 4 changes the denominator and should land behind a rubric version bump.

Not proposed

Vendoring ECC's script. The rubric design is the borrowable part; the implementation assumes ECC's own directory layout.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions