fix(gate): stop text-based failure guessing on echoed content - #17
Open
HMWKR wants to merge 1 commit into
Open
Conversation
detect_failure previously ran FAILURE_RE over the full response text whenever no explicit exit/success key was present. Edit/Write responses echo file content and Bash stdout echoes grepped text, so successful calls containing words like 'failure' or 'error:' were recorded as tool failures, cascading 'gate observed a tool failure' warnings for the rest of the session (observed 2026-08-09). Now, when no explicit exit/success key is present: - a string tool_response matching anchored '^(Error: )?Exit code [1-9]' is a failure (the shape Claude Code actually delivers for failed Bash calls, confirmed against real transcripts); - otherwise text guessing runs only for Bash and only on the stderr field. verification_record keeps full-text matching because test runners like pytest print failures to stdout. Existing suites (gate 6, robustness 12, recovery, shadow/M3/M4) pass; 11 bidirectional cases verified (echoed-content paths silent; string exit-code, dict exit-code, and stderr failure paths still fire; verification_record semantics kept). Reviewed by a second-vendor read-only pass which caught the string-form regression.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
detect_failurerunsFAILURE_REover the full response text whenever no explicit exit/success key is present. Two common shapes of successful calls match it:grep failed src/output matchesFAILURE_REon success.Once one lands in the ledger,
gate_post_toolemitsfablize gate observed a tool failure…on that call, and with repeats the disclosure guard escalates — we observed a session where every successful Edit produced the warning (alert fatigue → the gate trains the model to ignore it, the exact failure modeharness-lintwarns about).Fix
When no explicit exit/success key is present:
tool_responsematching anchored^(Error: )?Exit code [1-9]is a failure — this is the shape Claude Code actually delivers for failed Bash calls (confirmed against real transcripts:"Exit code 128\nfatal: …");stderrfield.verification_recordkeeps full-text matching, because test runners like pytest print failures to stdout and that signal is worth keeping for verification commands.Verification
test_gate.py(6),test_gate_robustness.py(12),test_recovery.py,test_shadow*.py.verification_recordsemantics unchanged.Exit coderegression in an earlier draft of this patch; it is covered here with the anchored match.🤖 Generated with Claude Code