Skip to content

feat: add cb health-check command (extract → eval → gap report)#78

Open
rajnavakoti wants to merge 1 commit into
mainfrom
feat/26-health-check
Open

feat: add cb health-check command (extract → eval → gap report)#78
rajnavakoti wants to merge 1 commit into
mainfrom
feat/26-health-check

Conversation

@rajnavakoti

Copy link
Copy Markdown
Contributor

Summary

Packages the full pipeline into a single command — cb health-check ./docs/ runs extract → eval → gap report and prints a knowledge-health summary. No prior cb init required. This is the "free health check" flow for conference/design-partner outreach.

Exit code is a deployment gate: 0 if CLEAN coverage ≥ threshold, else 1.

What it does

  1. Extract entities from the docs folder (synthesizes a minimal seed if none given)
  2. Generate eval questions (seed + doc + persona) and run coverage eval
  3. Produce an HTML gap report
  4. Summarize: entity count, coverage %, per-persona breakdown, top-5 gaps (by severity), cost estimate

Changes

  • context_blocks/cli.py — new health-check command (mostly orchestration of existing pieces)
  • context_blocks/retrieval/evals.py — extracted a shared run_coverage_eval() helper (KB load + embed + retrieval + coverage report), so the eval flow is reusable
  • tests/unit/test_health_check.py — new

Test plan

  • Pass above threshold → exit 0, HTML report written
  • Below threshold → exit 1
  • Missing docs dir → exit 1 with error
  • Synthesizes a seed when none provided
  • Summary shows entities + coverage
  • Full unit suite: 205 passed, no regressions

Follow-ups (not in this PR)

  • Refactor the existing cb eval command to also use run_coverage_eval() (DRY) — kept separate here to keep the change low-risk.
  • Extraction cost isn't tracked upstream, so the cost line reflects the eval estimate only (labelled as such).
  • PDF report generation is a separate issue (per feat: health check packaging — cb health-check command #26 scope).

Closes #26

🤖 Generated with Claude Code

Package the full pipeline into one command: given a folder of docs,
extract entities, run coverage evals, and produce a gap report — no
prior 'cb init' required. Exit code reflects a configurable coverage
threshold, usable as a deployment gate.

- New `cb health-check` command in cli.py
- Shared run_coverage_eval() helper in evals.py (KB load + embed +
  retrieval + coverage report), reusable by cb eval later
- Synthesizes a minimal seed when none is provided
- Tests: pass/fail threshold exit codes, seed synthesis, summary
  output, and missing-docs validation

Closes #26

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: health check packaging — cb health-check command

1 participant