Phase 1 diagnostics: localize FALSE→MISLEADING softening + audit evidence packs (P67.2)#44
Closed
aRealGem wants to merge 1 commit into
Closed
Phase 1 diagnostics: localize FALSE→MISLEADING softening + audit evidence packs (P67.2)#44aRealGem wants to merge 1 commit into
aRealGem wants to merge 1 commit into
Conversation
…dence packs Two $0 diagnostic scripts + their artifacts + the findings doc (P67.2 Phase 1): - diagnose_softening.py (offline, stdlib-only): buckets every milder-than-gold outcome by pipeline stage (UNANIMOUS_SOFT / ARBITER_SOFT / TIE_ABSTAIN / CRM114_SOFT) across all committed eval artifacts and both full-publish runs, joining full-run rows to gold by normalized claim text (F6). Asserts pca.yaml criterion=label_mismatch so the 2-1-tally-identifies-the-arbiter theorem (F2) is guarded. - audit_evidence_packs.py (live Brave/FactCheck, zero LLM): rebuilds the packs for the 4 gold-FALSE sids exactly as the eval does, keeps pre-cap items, and runs systematic + gist counter-evidence probes. FINDINGS (SOFTENING_DIAGNOSIS.md): UNANIMOUS_SOFT dominates; calib moves one seat to FALSE on 0342/0556 but the arbiter hedges it back (ARBITER_SOFT); TIE_ABSTAIN secondary (0020); CRM-114 never soften-flips. F4's inference is REFUTED: the packs already contain explicit refuting fact-checks. Root cause: on absolute claims (zero/only/ended/obliterated) the calib rubric's rule (3) instructs MISLEADING for what gold+fact-checkers call FALSE. Phase 3 lever = absolute-claim rule in the calib prompt + CRM-114 criteria, NOT retrieval. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
This was referenced Jul 19, 2026
aRealGem
added a commit
that referenced
this pull request
Jul 19, 2026
…ence packs (P67.2) [reopen of #44] (#49) * Track B eval harness: frontier PCA roster + open-book A/B artifacts - hydramind/specs/rosters.yaml: add `frontier` roster (P=claude-sonnet, C=claude-sonnet, A=claude-opus) alongside the dev roster. - eval/benchmarks/run_pca_frontier.py: closed-book frontier runner (forks run_pca_devlot; reads verdict_gold.train.jsonl; emits `status` so score_verdict.py consumes output directly). - eval/benchmarks/score_layerb_vs_gold.py: add --roster/--limit flags so the open-book eval can run any spec roster on a row subset. - Commit the four open-book eval artifacts (dev/frontier x plain/calib, all +CRM-114) behind the Track B conclusion: no model-tier lift; dev+calib decided-acc 0.5625 beats frontier+calib 0.50 at ~1/7 the cost; FALSE->MISLEADING severity-softening is roster-independent (<=1/4 in every config). Full findings on kanban P67/P67.2. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Phase 1 diagnostics: localize FALSE->MISLEADING softening + audit evidence packs Two $0 diagnostic scripts + their artifacts + the findings doc (P67.2 Phase 1): - diagnose_softening.py (offline, stdlib-only): buckets every milder-than-gold outcome by pipeline stage (UNANIMOUS_SOFT / ARBITER_SOFT / TIE_ABSTAIN / CRM114_SOFT) across all committed eval artifacts and both full-publish runs, joining full-run rows to gold by normalized claim text (F6). Asserts pca.yaml criterion=label_mismatch so the 2-1-tally-identifies-the-arbiter theorem (F2) is guarded. - audit_evidence_packs.py (live Brave/FactCheck, zero LLM): rebuilds the packs for the 4 gold-FALSE sids exactly as the eval does, keeps pre-cap items, and runs systematic + gist counter-evidence probes. FINDINGS (SOFTENING_DIAGNOSIS.md): UNANIMOUS_SOFT dominates; calib moves one seat to FALSE on 0342/0556 but the arbiter hedges it back (ARBITER_SOFT); TIE_ABSTAIN secondary (0020); CRM-114 never soften-flips. F4's inference is REFUTED: the packs already contain explicit refuting fact-checks. Root cause: on absolute claims (zero/only/ended/obliterated) the calib rubric's rule (3) instructs MISLEADING for what gold+fact-checkers call FALSE. Phase 3 lever = absolute-claim rule in the calib prompt + CRM-114 criteria, NOT retrieval. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Cass (Claude Code agent) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The two $0 diagnostics from the post-Track-B roadmap (Phase 1), their committed artifacts, and the findings doc. Stacked on #41 (reads the four open-book eval artifacts committed there). The full-run join also reads
metrics/pca_runs/*.jsonfrom #40's branch at runtime (not committed here).diagnose_softening.py(offline, stdlib-only): buckets every milder-than-gold outcome by the pipeline stage that produced it —UNANIMOUS_SOFT/ARBITER_SOFT/TIE_ABSTAIN/CRM114_SOFT— across all eval configs + both full-publish runs (joined to gold by claim text, per F6; assertspca.yaml criterion=label_mismatchto guard the F2 theorem).audit_evidence_packs.py(live Brave/FactCheck, zero LLM, ~24 queries): rebuilds the packs for the 4 gold-FALSE sids exactly as the eval does, keeps pre-cap drops visible, and runs systematic + gist counter-evidence probes.Findings (
SOFTENING_DIAGNOSIS.md)_CALIB_PROCEDURErule (3) explicitly instructs MISLEADING for "overstating a true underlying fact". The models are complying with the prompt, not failing it.Phase 3 lever (pre-registered decision rule): absolute-claim rule in the calib prompt + CRM-114 criteria hardening — not query augmentation.
Test plan
diagnose_softening.pyneeds no network at all).🤖 Generated with Claude Code