fix(prompt): restore two dropped qualifiers from the reference Doing-tasks section#748
Merged
Merged
Conversation
…tasks section
Both are faithful-port restorations of text present in
typescript/src/constants/prompts.ts and lost in the Python port — not new
guidance.
1. "Do not create files unless they're absolutely necessary" was missing
the reference's "for achieving your goal" AND its rationale ("as this
prevents file bloat and builds on existing work more effectively").
Stripped of both, a bloat-avoidance heuristic reads as a flat ban on
creating files.
Cost, measured on terminal-bench 2.1 (torch-tensor-parallelism,
2026-07-25): the instruction was "Create the file
/app/parallel_linear.py", /app was empty, and the container had no
Python to verify against. clawcodex spent all 8 of its steps hunting
for an interpreter — two Bash probes, a ToolSearch, then four
filesystem-wide Globs — and never wrote the file. Claude Code ran the
same probes, concluded nothing was runnable, wrote the deliverable on
its 5th action, and scored 1.0.
2. The "If an approach fails" bullet lost its closing clause scoping
escalation to genuine blockage. Restored WITHOUT naming
AskUserQuestion, which the reference interpolates there: that tool is
unregistered on the headless surface, and a prompt naming an
unavailable tool is its own failure mode (fix-git, 2026-07-25 — asked
twice, then handed the task back to a user who could not exist).
Note what is NOT here: no instruction telling the agent to produce the
artifact when the environment cannot be verified. The reference has no
such line — Claude Code's behavior there is emergent — and inventing one
is what a previous A/B on this benchmark already falsified.
Tests: 8843 passed; both restorations are revert-sensitive.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ericleepi314
added a commit
that referenced
this pull request
Jul 26, 2026
…ance (#750) Iteration 2 on the trajectory-step gap against the latest Claude Code (same model, same effort). Iteration 1 (#749) took the subset mean from 25.5 to 13.2 steps against CC's 7.5; this addresses the largest remaining structural difference. Measured on the same tasks: clawcodex emits more than one tool call per assistant turn in **5.7%** of steps, the latest Claude Code in **18.1%**. Every independent pair it does not batch is an extra step, so that ratio is most of what remains between the two step counts. Reading the trajectories side by side shows it plainly — on regex-log CC issued 8 calls, all Bash, batching probes ("for c in node deno perl ruby; do command -v $c; done") and creating files with heredocs; clawcodex issued 17, alternating Write then Bash and probing one command at a time. Cause is a truncated port. The reference bullet has three parts: You can call multiple tools in a single response. If you intend to call multiple tools and there are no dependencies between them, make all independent tool calls in parallel. Maximize use of parallel tool calls where possible to increase efficiency. However, if some tool calls depend on previous calls to inform dependent values, do NOT call these tools in parallel and instead call them sequentially. ... clawcodex kept only the first sentence. What was dropped is what turns a permission into a practice: the imperative to maximize, and the dependency carve-out. The carve-out matters as much as the push — without a statement of when NOT to parallelize, "call them in parallel" is risky advice a careful model will mostly decline, which is what the 5.7% looks like. Same defect class as #748 (dropped qualifiers on the file-creation bullet), and general rather than benchmark-specific: batching independent work is better agent behavior everywhere. Tests: 8843 passed. The new test pins all three clauses, since restoring only the push would leave the model without the safety half. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two faithful-port restorations of text that exists in
typescript/src/constants/prompts.tsand was lost in the Python port. Not new prompt engineering.The bug
Both qualifiers were dropped. "For achieving your goal" is what licenses creating a file when the goal is the file; the rationale is what scopes the rule to bloat-avoidance. Stripped of both, a heuristic reads as a flat ban.
Measured cost — terminal-bench 2.1
torch-tensor-parallelism, 2026-07-25. The instruction was literally "Create the file /app/parallel_linear.py";/appwas empty and the container had no Python to verify against. clawcodex spent all 8 of its steps hunting for an interpreter (two Bash probes →ToolSearch→ four filesystem-wideGlobs) and never wrote the file. Claude Code ran the same probes, concluded nothing was runnable, wrote the deliverable on its 5th action, and scored 1.0.Verified on the real task
Re-ran
torch-tensor-parallelismagainst this branch, same model/effort (claude-opus-5, xhigh):reward 0.0 → 1.0, 0 errors.
Clean comparison — identical environment both times (
python3: command not found, empty/app). Only the ordering changed:ToolSearch→ 4×Glob→ timeout, nothing writtenWrite /app/parallel_linear.py(3rd action) → then resumed investigatingThat is Claude Code's ordering: deliverable first, verification after.
Second restoration
The "If an approach fails" bullet lost its closing clause scoping escalation to genuine blockage. Restored without naming
AskUserQuestion, which the reference interpolates there — that tool is now unregistered on the headless surface (#747), and a prompt naming an unavailable tool is its own failure mode (fix-git, 2026-07-25: asked twice, then handed the task back to a user who could not exist).What is deliberately NOT here
No instruction telling the agent to produce the artifact when the environment cannot be verified. The reference has no such line — Claude Code's behaviour there is emergent — and inventing one is precisely what a previous A/B on this benchmark already falsified. This worked by removing a wrong instruction, not by adding a right one.
Caveats
n=1 on the task-level result. The re-run also carried #747's changes (no
AskUserQuestion, empty-turn guard), so while the trace makes the causal story clear — the file gets written at action 3, before any of that could matter — a full rerun is what would size the real effect.Suite: 8843 passed. Both restorations are revert-sensitive (reverting the wording fails the intended test).
🤖 Generated with Claude Code