Skip to content

review-bench fixture maintenance round 2: fold C2-run adjudications + resolve u2 disagreement #40

Description

@SSFSKIM

Problem & intent

C2's scored runs (post-#36 fixtures) surfaced a second round of adjudicated genuine-unseeded defects plus one cross-engine disagreement on a promoted truth entry. Fold them into tests/review-bench/ truth/fixtures the same way #36 did, so the NEXT scored run-id needs no escape-hatch exclusions.

Items (details in results/2026-07-26-c2-scores.json new_unseeded_candidates_excluded and the per-run notes.md):

  • case4 source-dir symlink probe — cross-engine convergent (codex r3, argus plain r2, argus high r2 [P1]); genuine, fix or promote
  • case4 staging-filesystem free-space gap (argus high r2 [P2] PLAUSIBLE)
  • case4 find -type f drops symlinked archive files (both argus runs [P3])
  • case3 expected_revision: null skips validation (codex r3 [P2]); adjudicate then fix or promote
  • case1 null config entry crashes load_tenant_limits (argus high r2 [P3])
  • case1 explicit-null tenant stringified to "None" (argus high r2 [P3])
  • case4-u2 disagreement: codex finds the remote-umask exposure every run; an argus high verifier actively REFUTED it in r2. Pull the refutation reasoning from the run transcript, decide whether u2 stands as truth (and whether the verifier text needs a nudge) — do not silently drop either side.

Constraints

Success criteria

  • A fresh engine run on the updated fixtures produces no finding that is neither truth-matched nor a deliberate bait, with no known-item exclusion list.

Open questions

  • u2: keep as truth, demote, or split? (needs the transcript reasoning first)

Decision log

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions