feat(eval): pack --variance-probe — measure the variance an OCH pack removes (Move 2)

[theagenticguy]

What

Implements spec 010 (Move 2) — codehub code-pack --variance-probe <task-file>. The probe is the empirical instrument behind the decision-equivalence contract (Move 6): if the OCH pack genuinely pins a coding agent's retrieval decision, the agent's answer wanders less run-to-run. The probe turns that claim into a number.

Given a task — a fixed triple (repo @ commit, instruction, success_oracle) — it runs an agent N times (default 10) in each of two arms (with-pack / without-pack), holding commit/instruction/agent/model fixed, and reports the dispersion delta + token overhead. Grounded against arXiv:2606.26979 ("deterministic anchoring halves run-to-run variance at ~10% more tokens").

New package: @opencodehub/eval

The package was a stub (Python harness extracted to opencodehub-testbed so the published set ships free of test-time deps). This is its first TS content, and it honors that intent: private, dep-light (pure JS + node:child_process + zod/yaml, no Python, no heavy deps), and force-bundled into the CLI tarball via tsup noExternal (verified: 0 surviving external imports in dist/index.js).

• task loader — YAML/JSON + Zod, strict, fail-fast (an expensive experiment never starts on a bad task)
• dispersion stats — distinct-output ratio / Bernoulli pass-rate stddev / judge-score stddev; pure, exhaustively unit-covered
• oracle scoring — output_hash (zero-config) | assertion (default, objective) | judge (LLM-panel, interface-ready)
• AgentRunner interface + the v1 direct-CLI runner
• deterministic report — canonicalJson, no clock/run-id (R6): two probe runs over the same captured outcomes serialize byte-identically

Bedrock inference (hard constraint, spec 010 §4a)

Both agents route inference through Amazon Bedrock, grounded against current docs (not recalled):

• Claude Code → CLAUDE_CODE_USE_BEDROCK=1 + AWS_REGION + us.-prefixed ANTHROPIC_MODEL inference profile; claude -p "<prompt>" --output-format json --model <id>; reads .result / .usage / .total_cost_usd.
• Codex → first-party amazon-bedrock provider; codex exec --json -c model_provider=amazon-bedrock -m <model> --skip-git-repo-check "<prompt>"; final answer = last agent_message, tokens = last turn.completed.usage.

Notably, neither CLI exposes temperature/seed — which is consistent with the design: within-arm sampling nondeterminism is the variance being measured.

CLI

codehub code-pack --variance-probe <task-file> [--runs N] [--harness claude|codex] [--aws-region R] [--model ID] [--json]. The command generates the pack once, assembles it into packContext, and runs the experiment. On-demand only — never a CI gate (§8). Example task at packages/eval/examples/variance-task.yaml.

Scope / deferrals

• omnigent-backed multi-agent runner → v2, behind the same AgentRunner interface (CLI-first, per the approved spec).
• The replay / decision-equivalence structural check is the other half of Move 6 (spec 011, separate).

Validation

• biome ci . ✓ (709 files, 0 errors)
• tsc -b full workspace ✓
• full build ✓ — @opencodehub/eval inlined into the CLI bundle (0 external imports)
• full test suite ✓ — all 18 packages fail 0; +69 new eval tests, +3 new CLI command tests
• pre-commit (biome, commitlint, banned-strings) + pre-push (verdict, typecheck, test) hooks ✓

Adds eval to the commitlint scope-enum (new workspace package).

🤖 Generated with Claude Code