Skip to content

fix(gate): stop text-based failure guessing on echoed content - #17

Open
HMWKR wants to merge 1 commit into
fivetaku:mainfrom
HMWKR:fix/gate-echoed-content-false-positive
Open

fix(gate): stop text-based failure guessing on echoed content#17
HMWKR wants to merge 1 commit into
fivetaku:mainfrom
HMWKR:fix/gate-echoed-content-false-positive

Conversation

@HMWKR

@HMWKR HMWKR commented Aug 9, 2026

Copy link
Copy Markdown

Problem

detect_failure runs FAILURE_RE over the full response text whenever no explicit exit/success key is present. Two common shapes of successful calls match it:

  • Edit/Write responses echo file content — editing a doc that contains words like "failure" or "error:" records a tool failure.
  • Bash stdout echoes grepped/cat'ed textgrep failed src/ output matches FAILURE_RE on success.

Once one lands in the ledger, gate_post_tool emits fablize gate observed a tool failure… on that call, and with repeats the disclosure guard escalates — we observed a session where every successful Edit produced the warning (alert fatigue → the gate trains the model to ignore it, the exact failure mode harness-lint warns about).

Fix

When no explicit exit/success key is present:

  • a string tool_response matching anchored ^(Error: )?Exit code [1-9] is a failure — this is the shape Claude Code actually delivers for failed Bash calls (confirmed against real transcripts: "Exit code 128\nfatal: …");
  • otherwise text guessing runs only for Bash and only on the stderr field.

verification_record keeps full-text matching, because test runners like pytest print failures to stdout and that signal is worth keeping for verification commands.

Verification

  • Existing suites pass: test_gate.py (6), test_gate_robustness.py (12), test_recovery.py, test_shadow*.py.
  • 11 bidirectional cases: echoed-content paths silent (Edit echo, Bash stdout grep, Read doc); string exit-code, dict exit-code, and stderr failure paths still fire; verification_record semantics unchanged.
  • A second-vendor read-only review pass caught the string-form Exit code regression in an earlier draft of this patch; it is covered here with the anchored match.

🤖 Generated with Claude Code

detect_failure previously ran FAILURE_RE over the full response text
whenever no explicit exit/success key was present. Edit/Write responses
echo file content and Bash stdout echoes grepped text, so successful
calls containing words like 'failure' or 'error:' were recorded as tool
failures, cascading 'gate observed a tool failure' warnings for the rest
of the session (observed 2026-08-09).

Now, when no explicit exit/success key is present:
- a string tool_response matching anchored '^(Error: )?Exit code [1-9]'
  is a failure (the shape Claude Code actually delivers for failed Bash
  calls, confirmed against real transcripts);
- otherwise text guessing runs only for Bash and only on the stderr
  field.
verification_record keeps full-text matching because test runners like
pytest print failures to stdout.

Existing suites (gate 6, robustness 12, recovery, shadow/M3/M4) pass;
11 bidirectional cases verified (echoed-content paths silent; string
exit-code, dict exit-code, and stderr failure paths still fire;
verification_record semantics kept). Reviewed by a second-vendor
read-only pass which caught the string-form regression.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants