Skip to content

fix(knowledge): gradeFor refutes a refusal claim because a correct refusal exits nonzero #1010

Description

@drewstone

A claim about a refusal cannot be verified, because a correct refusal exits nonzero

gradeFor in @tangle-network/agent-knowledge consults the exit code and returns before the
expectation is ever compared:

if (execution.exitCode !== 0) {
  return { verdict: UNRUNNABLE_SIGNATURES.test(output) ? 'unrunnable' : 'contradicted' }
}
if (mustBeCheckable) {
  const note = expectationRefusalNote(evidence.expect)
  if (note) return { verdict: 'uncheckable', note }
}
if (evidence.expect && !output.includes(evidence.expect)) {
  return { verdict: output === '' ? 'silent-check' : 'contradicted' }
}
return { verdict: 'verified' }

So for any check whose CORRECT behaviour is a nonzero exit, the expect string is dead code.

Reproduction, measured 2026-08-23

h2-director-v2-ygadget-orbit-under-4067-r1 records a negative control: a gate must REFUSE when a
named artifact is absent. Its check re-runs that gate and asserts the refusal.

check   : python3 "$DISCOVERY_RUN_WORKSPACE/lead-1-s1/check_c1_locate.py"
expect  : REFUSED=1
exit    : 3
output  : ... REFUSAL_JSON REFUSED=1 reason=ARTIFACT_ABSENT failing_row_index=None RECORD_OK=True ...
verdict : contradicted

The check ran, did exactly what the claim says, printed the expectation verbatim, and was refuted
for it. Two claims in that one run; a store-wide scan finds only these two, but the scan can only
see the stored 300-character output tail, so it is a lower bound.

Why this is worth fixing rather than working around

The affected class is negative controls, refusal gates, and "prove the guard fires" claims — the
checks that establish an instrument is trustworthy before any positive result rests on it. A grader
that refutes them is penalising the practice it most wants. Authors learn to make such checks exit
0 and encode the refusal only in stdout, which discards the exit code as a signal everywhere else.

Suggested shape

The exit code and the expectation answer different questions. When an expect is present, it is
the claim's own statement of what success looks like, and it should be consulted first: if the
output contains the expectation, the claim is verified whatever the exit code, because the author
has said in advance which observation settles it. Reserve the exit-code branch for claims that
carry no expectation, where it is the only available signal.

The UNRUNNABLE_SIGNATURES test should still take precedence, so a check that could not run is
never rescued by a coincidental substring.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions