agent: tell the model when bash output was truncated - #680
Open
JonasColmsjo wants to merge 1 commit into
Open
Conversation
agent_bash_observation() clips long output to a <head -N file> or <tail -N file> block and prints output_path, but never states that anything was dropped. The model is left to infer truncation from the tag name alone, and smaller models do not make that inference: they read the fragment as the complete output and answer confidently from a partial result. Observed with a 5000-line command whose last line decided the answer. The full output is already on disk and the read tool can open it, so the fix is only to say so. After the head/tail block, emit a line stating how many of how many lines were shown and naming the file to read, and document the convention in the tools system prompt. Warn only on real truncation. The head branch is also taken while a job is still running, and at that point nothing has been dropped -- warning there would claim "the first 5 of 5 lines" and teach the model to distrust output that is intact. The warning goes into the model-visible observation. The terminal echo path (agent_bash_publish_observation) still publishes only the head/tail body, so interactive output is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
agent_bash_observation()clips long bash output to a<head -N file>or<tail -N file>block and printsoutput_path=..., but it never states thatanything was dropped. The model has to infer truncation from the tag name alone.
Larger models usually make that inference. Smaller/quantized ones do not — they
read the fragment as the complete output and answer confidently from a partial
result. I hit this with a command whose answer depended on its last line: the
model saw the first 100 of 5000 lines and never realised the rest existed.
The full output is already on disk and the
readtool can open it. Nothing toldthe model it needed to.
Change
After the head/tail block, emit one line stating how many of how many lines were
shown and naming the file to read, and document the convention in the tools
system prompt so the model knows head/tail blocks are never the whole output.
Warn only on real truncation. The head branch is also taken while a job is still
running, and at that point nothing has been dropped — warning there would claim
"the first 5 of 5 lines" and teach the model to distrust output that is intact.
The warning goes into the model-visible observation. The terminal echo path
(
agent_bash_publish_observation()) still publishes only the head/tail body, sointeractive output is unchanged.
Verification
Built with
make cuda-spark(GB10 / sm_121), no new warnings, and exercisedagainst DeepSeek-V4-Flash on a DGX Spark:
seq 1 5000— model quotes back:WARNING: output truncated. Only the first 100 of 5000 lines are shown above; the rest was NOT shown. Read /tmp/ds4_agent_output_SBvTZt before drawing conclusions that depend on the full output.seq 1 5— model reportsNO-WARNING(no false positive on short output)In agent runs the model then reads
output_pathinstead of guessing.