A claim about a refusal cannot be verified, because a correct refusal exits nonzero
gradeFor in @tangle-network/agent-knowledge consults the exit code and returns before the
expectation is ever compared:
if (execution.exitCode !== 0) {
return { verdict: UNRUNNABLE_SIGNATURES.test(output) ? 'unrunnable' : 'contradicted' }
}
if (mustBeCheckable) {
const note = expectationRefusalNote(evidence.expect)
if (note) return { verdict: 'uncheckable', note }
}
if (evidence.expect && !output.includes(evidence.expect)) {
return { verdict: output === '' ? 'silent-check' : 'contradicted' }
}
return { verdict: 'verified' }
So for any check whose CORRECT behaviour is a nonzero exit, the expect string is dead code.
Reproduction, measured 2026-08-23
h2-director-v2-ygadget-orbit-under-4067-r1 records a negative control: a gate must REFUSE when a
named artifact is absent. Its check re-runs that gate and asserts the refusal.
check : python3 "$DISCOVERY_RUN_WORKSPACE/lead-1-s1/check_c1_locate.py"
expect : REFUSED=1
exit : 3
output : ... REFUSAL_JSON REFUSED=1 reason=ARTIFACT_ABSENT failing_row_index=None RECORD_OK=True ...
verdict : contradicted
The check ran, did exactly what the claim says, printed the expectation verbatim, and was refuted
for it. Two claims in that one run; a store-wide scan finds only these two, but the scan can only
see the stored 300-character output tail, so it is a lower bound.
Why this is worth fixing rather than working around
The affected class is negative controls, refusal gates, and "prove the guard fires" claims — the
checks that establish an instrument is trustworthy before any positive result rests on it. A grader
that refutes them is penalising the practice it most wants. Authors learn to make such checks exit
0 and encode the refusal only in stdout, which discards the exit code as a signal everywhere else.
Suggested shape
The exit code and the expectation answer different questions. When an expect is present, it is
the claim's own statement of what success looks like, and it should be consulted first: if the
output contains the expectation, the claim is verified whatever the exit code, because the author
has said in advance which observation settles it. Reserve the exit-code branch for claims that
carry no expectation, where it is the only available signal.
The UNRUNNABLE_SIGNATURES test should still take precedence, so a check that could not run is
never rescued by a coincidental substring.
A claim about a refusal cannot be verified, because a correct refusal exits nonzero
gradeForin@tangle-network/agent-knowledgeconsults the exit code and returns before theexpectation is ever compared:
So for any check whose CORRECT behaviour is a nonzero exit, the
expectstring is dead code.Reproduction, measured 2026-08-23
h2-director-v2-ygadget-orbit-under-4067-r1records a negative control: a gate must REFUSE when anamed artifact is absent. Its check re-runs that gate and asserts the refusal.
The check ran, did exactly what the claim says, printed the expectation verbatim, and was refuted
for it. Two claims in that one run; a store-wide scan finds only these two, but the scan can only
see the stored 300-character output tail, so it is a lower bound.
Why this is worth fixing rather than working around
The affected class is negative controls, refusal gates, and "prove the guard fires" claims — the
checks that establish an instrument is trustworthy before any positive result rests on it. A grader
that refutes them is penalising the practice it most wants. Authors learn to make such checks exit
0 and encode the refusal only in stdout, which discards the exit code as a signal everywhere else.
Suggested shape
The exit code and the expectation answer different questions. When an
expectis present, it isthe claim's own statement of what success looks like, and it should be consulted first: if the
output contains the expectation, the claim is verified whatever the exit code, because the author
has said in advance which observation settles it. Reserve the exit-code branch for claims that
carry no expectation, where it is the only available signal.
The
UNRUNNABLE_SIGNATUREStest should still take precedence, so a check that could not run isnever rescued by a coincidental substring.