fix(knowledge): gradeFor refuses a passing check for having a precise expectation

[drewstone]

Summary

gradeFor refuses an expectation carrying three or more key=value tokens and returns uncheckable — after the check has already run and exited 0, and before the output is compared. A check whose output matches its expectation byte for byte is therefore discarded as ungraded.

Measured on @tangle-network/agent-knowledge@10.7.0, 2026-08-23: 173 of 1001 rung≥4 claims with an expectation (17.3%) are refused this way.

The rule

function expectationRefusalNote(expect) {
  if (!expect?.trim()) return NO_EXPECTATION_NOTE
  if (expect.includes("\n") || expect.includes("\r")) return MULTILINE_EXPECTATION_NOTE
  if (expect.trim().split(/\s+/).filter((token) => token.includes("=")).length >= 3) return MULTI_VALUE_EXPECTATION_NOTE
}

Called from gradeFor at a point where execution.exitCode === 0 is already established:

if (mustBeCheckable) {
  const note = expectationRefusalNote(evidence.expect)
  if (note) return { verdict: "uncheckable", note }
}
if (evidence.expect && !output.includes(evidence.expect)) return { verdict: ... }
return { verdict: "verified" }

The refusal sits directly above the comparison that would have passed.

Worked example

A formally certified code-distance result. The check runs drat-trim over two UNSAT proofs and re-verifies two witness words with numpy:

expect:     SMALL_MITTEN_CERTIFIED proofs_verified=true dZ_lb=10 dX_lb=10 dZ_witness=10OK dX_witness=10OK
outputTail: SMALL_MITTEN_CERTIFIED proofs_verified=true dZ_lb=10 dX_lb=10 dZ_witness=10OK dX_witness=10OK
verdict:    uncheckable

Re-executed independently: same output, exit 0, 7 m 15 s. output.includes(expect) is true. The verdict is uncheckable solely because five tokens contain =.

Why the rule inverts its own intent

A guard against vague expectations should reject expectations that are easy to satisfy accidentally. This rejects the opposite. Ranked by how hard they are to match by chance, the refused expectations are the strongest in our corpus:

GRID OK cells=8 WIN=1 PARETO=6 NEGATIVE=1
CELL steane7|pL=0.10 delta=-8 disc=(9,1) p=0.021484 -> WIN
SCORE cells=4 verdict=MIXED detail=150-30-10:no-loss:NEGATIVE;...

Each names several independent quantities at once, so a false pass requires every one of them to coincide. Meanwhile expect: OK — one token, no = — sails through and is satisfied by any output containing "OK" anywhere.

The rule makes a claim less gradable the more precisely it is stated, which is backwards, and the effect is silent: the verdict is uncheckable, the same word used for a claim that carries no check at all, so a fleet operator reading a grade summary cannot tell "the author never wrote a check" from "the check ran and passed".

Suggested fix

Compare first, refuse second. If output.includes(expect) the claim is verified regardless of token shape — a passing exact-match is not a vague expectation.

If a guard against kitchen-sink expectations is still wanted, it should key on something that actually indicates weakness (an expectation shorter than N characters, or one that matches a large fraction of arbitrary outputs), not on the count of = characters. And if such an expectation must be refused, please give it a distinct verdict from "no check exists" — those are opposite failures.

[drewstone]

Correction to the measurement in the issue body: the counts there double-counted mirrored copies of the same run.

Deduplicated by page id across all lab paths:

DISTINCT rung>=4 claims with an expect: 835
  refused by the >=3 key=value rule:    146  (17.5%)

The rate is unchanged at roughly one claim in six (17.3% -> 17.5%); the absolute count is 146, not 173. Everything else in the report stands, including the worked example, which was a single claim and unaffected by the duplication.

[drewstone]

Measured the actual distribution of expectation strength in this store, which makes the inversion
quantitative rather than illustrative.

Across the 181 verified rung>=4 claims of the director-v2 arm:

expectation shape	count	share
strong — 2+ key=value tokens	74	40%
one key=value	48	26%
prose / other	34	18%
bare number (10, 22, 18, 0)	12	6%
single word (LP_STRUCT_OK, RECOVERY_NONE)	9	4%
bare sha256 digest	4	2%

25 of 181 (13%) carry an expectation a false pass could satisfy by accident. expect: 10
matches any output containing the characters 10 anywhere — a stray line number, a timestamp, a
count of something unrelated. Real examples in the store: expect: 22 twice in
qc-lp-nonabelian-dominance-r1, expect: 18 and expect: 20 in
surgery-lowweight-basis-lp20-r1, expect: 0 in tourney-reservoir-core-r1.

The gradeFor rule refuses the 74 strongest and admits all 25 weakest. A claim needing
four numbers to coincide simultaneously is the one that cannot pass by accident; a claim needing
the substring 10 is the one that can. The guard is applied in exactly the wrong direction, and
the cost measured earlier — 146 of 835 distinct claims, 17.5% — falls entirely on the precise end.

If the intent is to prevent an author from writing an expectation so specific it can only match
their own recorded output, the discriminator is not token COUNT. A recorded digest or a
hand-copied float has that property at one token; a multi-key measurement line does not have it at
five. Length is uncorrelated with the failure mode the rule is trying to catch.

[drewstone]

The sharpest case yet: the strongest result this fleet produced today was refused by this rule.

An escalation asked for a predicate separating the concept classes that violate a ceil(log2|C|)
lower bound from those that do not, over an exhaustive enumeration at |X|=4. It found one:

expect : PERFECT_DNF TP=660 FP=0 TN=3323 FN=0 SINGLE_LITERAL_MIN_ERR=463
output : PERFECT_DNF TP=660 FP=0 TN=3323 FN=0
         SINGLE_LITERAL_MIN_ERR=463
verdict: uncheckable

A DNF predicate classifying all 3,983 canonical classes with zero false positives and zero false
negatives (660 + 3323 = 3983, the exact census), plus the control showing the best single literal
errs on 463 so the disjunction is genuinely required. Five key=value tokens, every one of which a
false pass would have to satisfy simultaneously.

expectationRefusalNote refuses it for carrying three or more of them.

Two defects stack here. Our local upgrade in tools/oracle.mjs rescues a refused expectation
when the check exits 0 and output.includes(expect) — but the check prints the two facts on
separate lines while the expectation joins them with a space, so the substring test fails and the
rescue does not fire. The claim is correct, re-executed by hand, and lands as uncheckable.

That second half is worth noting independently of this issue: a whitespace difference between a
multi-line output and a single-line expectation is not a disagreement about anything, and any
comparison that treats it as one will keep producing verdicts like this. Normalising runs of
whitespace before the containment test would fix it without weakening the check.

Restating the measurement from my earlier comment, now with this instance in it: of 181 verified
claims in this arm, the 74 strongest carry 2+ key=value tokens and the 25 weakest rest on a bare
number or single word. This rule refuses the strong end and admits the weak end.