Stability measurement suite: answer, reasoning, and counterfactual layers (spec + rung-0 implementation) - #159
Open
MaxGhenis wants to merge 7 commits into
Open
Stability measurement suite: answer, reasoning, and counterfactual layers (spec + rung-0 implementation)#159MaxGhenis wants to merge 7 commits into
MaxGhenis wants to merge 7 commits into
Conversation
Three layers: answer stability (repeats + variance decomposition against household-sampling SE), reasoning stability (mechanism-extraction judge with gold-set/cross-judge validation), counterfactual consistency (+$1,000 head-wage twins vs PolicyEngine true deltas). Spec adopted after a 5-reviewer adversarial workflow (36 findings); load-bearing findings re-verified directly: LiteLLM cache replay would have faked repeat stability, +$1k produces zero binary flips and 247 nonzero amount deltas, judge is Tier-2 classification not >90% extraction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…s (TDD) - policybench/stability.py: pair/row flip metrics with scenario-cluster bootstrap CIs, chi-square variance decomposition with the CI-based sampling-dominance decision rule, pooled roster estimator, spend-ledger cache-contamination guard, multi-dir runs loading. - policybench/counterfactual.py: Scenario-level twin builder (+$1k head wages), both-arm truth deltas rounded to cents, nonzero-stratum delta metrics with the max($1, 10%) band, per-direction sign recall, binary flip counts (never rates), large-response detection with exact binomial CI and suppression, zero-delta baseline, row-matched noise floor and signal-vs-noise bootstrap test. - Harness: eval-no-tools-repeated no longer enables the LiteLLM disk cache (repeat replay would fake stability); serving condition (POLICYBENCH_TOOL_CHOICE / CHUNK_OVERRIDE) now part of the response contract fingerprint; repeat sets write runs_metadata.json and reject mixed model groups per directory. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Mechanism-label extraction judge scoped to one explanation at a time (no pairs, no comparison, no arithmetic), committed few-shot anchors from the reference explanations, disk-deduplicated async extraction that never touches the LiteLLM cache, deterministic numeric-claim channel with half-up rounding and prediction exclusion, answer-stable pair frames with verbatim short-circuit, headline + joint/standardized/nonidentical/ attenuation companions and suppression rule, reference alignment, and the validation statistics the review required (Wilson CI, Gwet AC1, Krippendorff alpha over Jaccard, dominant-set share, gold-set eval). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ctual-manifest/report, stability-cost-plan Command layer in policybench/stability_report.py (testable without argparse): cache guard + runs-metadata fingerprint pooling check + serving-config diff against the board registry for layer 1; judge extraction with reference anchors, cross-judge and determinism validation, gold-set gate with CI-aware status and headline withholding for layer 2; twin manifest with both-arm references, truth-delta summary and frozen-reference drift, delta report with noise floor and provider-fingerprint mismatch counts for layer 3; cost ladder from logged usage with price-override fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- annotations/stability_reasoning_gold_set.csv: 90 reference explanations (5 per output group, seed 20260818) developer-labeled against the mechanism taxonomy for the judge validity gate; prevalence from thresholds_rates 59/90 down to period_annualization 2/90. - Spec runbook now covers all three layers with the implemented CLI (reasoning-stability, counterfactual-manifest/report, stability-cost-plan) and records the free rung-0 dry run: the truth arm reproduces 247 nonzero amount rows / 0 binary flips / 87.55% zero share through the shipped commands, and the cost plan prices 30 models x 4 arms at $1,302 + ~$140 judge (Fable run-level usage flagged unpriced). - docs/runbook.md cross-references the stability spec and its cache-free repeat rule. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rebased onto main's 2026-08-22 board refreeze: top-8 re-derived (ox-alpha fourth, grok-4.6 eighth; grok-4.5 and fable drop out), rung totals recomputed via stability-cost-plan ($1,337 priced + unpriced ox-alpha + Fable ≈$216, judge ≈$149), ox-alpha reported unpriced rather than guessed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds the three-layer stability measurement suite motivated by Shergadwala's "The Stability Trap" (FAccT '26, arXiv:2601.11783) — LLM judges reach >99% verdict agreement while reasoning stability collapses — and Stewart's counterfactual interrogation rights (arXiv:2603.22716). Spec:
docs/stability_spec.md. No paid runs in this PR: everything ships against mocks/fixtures plus free local PolicyEngine computation; every paid rung is gated on explicit approval.Process: adversarial verify before implementation
Per the task requirement, the metric definitions were attacked by a 5-reviewer adversarial workflow (statistical validity, literature fidelity with both papers fetched, PolicyBench integration, counterfactual design, ops/cost) before any implementation. 36 findings; the load-bearing ones were independently re-verified against code and data, and all reshaped the spec. The two that mattered most:
eval-no-tools-repeatedenabled the LiteLLM disk cache with no run-specific key material, so repeats 2..K would have replayed run 1's responses — "perfect stability" as a cache artifact, the exact failure the suite exists to catch. Fixed: repeats run cache-free, andstability-reporthard-fails on anycache_hit: truespend-ledger record.Layer 1 — answer stability
row_stability_by_model: K-invariant pair-level flip metrics (answer/verdict/coverage flips, "consistently wrong, differently wrong" on amount rows), scenario-cluster bootstrap CIs on every pooled rate.stability_variance_decomposition: run-score std vs household-sampling SE with an explicit chi-square decision rule (K=3 detectable floor ≈0.23 is printed, not hidden) plus a pooled roster estimator — this turns the manuscript's unmeasured "sampling dominates run-to-run variation" claim (paper/index.qmd, Uncertainty) into a measured one either way.POLICYBENCH_TOOL_CHOICE/POLICYBENCH_CHUNK_OVERRIDE) now enters the resume-metadata fingerprint (note: in-flight pre-change partial runs cannot resume — none known); repeat sets writeruns_metadata.jsonand reject mixed model groups per directory.Layer 2 — reasoning stability
Headline: "right answer, unstable reasoning" rate — among repeat pairs where the answer is stable and exact-correct, the share whose explanations invoke different mechanisms. Judge design per the paper's own findings: the LLM extracts discrete taxonomy labels from one explanation at a time (never compares, never computes); all pairing/agreement/numeric parsing is deterministic code. The spec is honest that 12-label extraction is the paper's Tier-2 regime (35–83% observed stability), so reliability is carried by the validation battery, not asserted: committed 90-row developer-labeled gold set (
annotations/stability_reasoning_gold_set.csv, pending maintainer review), cross-judge agreement, temp-0 determinism floor, Gwet AC1 + Krippendorff alpha (κ only where prevalence supports it), CI-aware gates that withhold the headline on failure, and pre-registered fallbacks. Composition-adjusted companions (joint rate over all pairs, direct-standardized rate, min-n suppression) address the conditioning-collider concern; reference explanations (1,984/1,984 coverage) anchor a reference-alignment metric.Layer 3 — counterfactual consistency
Scenario-level +$1,000 head-wage twins; both arms' references computed with one engine version (never mixing frozen and regenerated references — snapshot used policyengine-us 1.755.4, installed is 1.723.0, base drift 1,983/1,984 within $1);
pred_deltavstrue_deltawith the max($1, 10%) band, per-direction sign recall, large-response detection with exact binomial CI and suppression, zero-delta baseline (87.55% by construction), and a row-matched noise floor from layer-1 repeats with a named bootstrap test for "distinguishable from repeat noise". Per-arm provider fingerprints stand in for Stewart's pinned-version requirement.CLI + rung-0 dry run (free, already exercised)
stability-report,reasoning-stability(--deterministic-onlyfor the judge-free channels),counterfactual-manifest,counterfactual-report,stability-cost-plan. The truth arm reproduces the measured delta distribution exactly through the shipped commands; the cost plan prices the 32-model roster (post-#157 refreeze) at ≈$1,337/4 arms priced + Fable ≈$216 + unpriced ox-alpha, judge ≈$149. Ladder: rung 1 pilot ≈$8 model + ≈$17 judge (2 cheap models); rung 2 top-8 ≈$423+; rung 3 full roster or fold into the #139 v2 protocol. Boring before billed — nothing here spends.Testing
61 new tests across 5 files (all external calls mocked; chi-square/agreement statistics pinned to hand-computed reference values); full suite 673 passed post-rebase onto the 32-model refreeze; ruff clean. A final subagent bug-hunt was attempted twice and died both times on account session limits — reported here rather than silently skipped; the maintainer-agent's own computational review (bootstrap model-alignment, cost arithmetic, CLI dry runs) stands in.
Frozen board scores and
app/data are untouched (the runbook explicitly re-pointsanalyze --app-data-outputaway fromapp/src/data.json).🤖 Generated with Claude Code