perf(agents): hand the agent amico-run's absolute path — 28x faster on the task that needed it - #248
Draft
Rchari1 wants to merge 1 commit into
Draft
perf(agents): hand the agent amico-run's absolute path — 28x faster on the task that needed it#248Rchari1 wants to merge 1 commit into
Rchari1 wants to merge 1 commit into
Conversation
…n the task that needed it
AGENTS.md said bare `amico-run` and relied on PATH. When PATH missed, the agent
went looking for the binary, and looking is expensive: profiling one session's
history found 128 unbounded filesystem searches totalling 417 seconds, individual
ones at 123s.
A new {{AMICO_RUN}} substitution (alongside the existing {{JULIA_PROJECT}} and
{{TEMPLATE_PATH}}) resolves to the launcher's absolute path at session prep, and
the guidance forbids `which` and the find/pip fallbacks. Degrades safely: with no
resolvable bin dir it substitutes the bare command, i.e. today's behaviour.
MEASURED, A/B, same model (bedrock moonshotai.kimi-k2.5), same prompt, only
AGENTS.md differing — 4 baseline vs 7 treatment trials on "run amico-run and show
its usage":
turns tool calls wall (median) `which` calls timeouts
baseline 6 10 226.6s 4 2 of 4
treatment 2 1 8.0s 0 0 of 7
~28x on median wall, 3x fewer turns, 10x fewer tool calls, and the baseline failed
to finish at all in half its trials. The mechanism is visible in the transcripts:
baseline ran `which amico-run`, then `ls /usr/local/bin`, `ls ~/.amico/bin`,
`find /usr`, `find /opt`, `pip list`…; treatment ran the absolute path once.
WHAT DID NOT WORK, and is trimmed accordingly: an earlier draft carried a long
batching instruction with the profiling numbers in it. A/B on the six-independent-
reads task showed NO measurable effect, because both models already batch —
kimi-k2.5 emitted [6,0] in all six trials either way, and deepseek-v4-flash-free
was noise (turns 3,3,2 baseline vs 4,2,3 treatment). Since it ships on every turn,
it is cut to one sentence: +160 tok/turn instead of +313, keeping the intent
without paying for prose that bought nothing measurable.
Caveat on effect size: under the trimmed wording, one of three re-runs regressed
to 5 turns (from 2). The substitution is what does the work; the prose is a
backstop. `which` calls stayed at 0 across all 7 treatment trials under both
wordings.
Tests: 3 new — the substitution, the safe degradation to a bare command, and the
two instructions. A pre-existing guard ("leaves no unknown {{...}} placeholder")
caught the new placeholder immediately and now covers it. 853 passed, tsc clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rchari1
marked this pull request as draft
August 4, 2026 13:32
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The finding
AGENTS.mdsaid bareamico-runand relied on PATH. When PATH missed, the agent went looking for the binary — and looking is expensive. Profiling one session's history:which amico-run || find /Users/raghavchari -type f -name "amico-run", andfind ~/.julia -name "Piccolo" -type dat 126sThe fix
A new
{{AMICO_RUN}}substitution (alongside the existing{{JULIA_PROJECT}}/{{TEMPLATE_PATH}}) resolves to the launcher's absolute path at session prep, and the guidance forbidswhichplus thefind/pip listfallbacks.Degrades safely: with no resolvable bin dir it substitutes the bare command — exactly today's behaviour.
Measured, A/B
Same model (
amazon-bedrock/moonshotai.kimi-k2.5), same prompt, onlyAGENTS.mddiffering. Task: "Run amico-run to print its usage/help output." 4 baseline vs 7 treatment trials.whichcalls~28× on median wall, 3× fewer turns, 10× fewer tool calls — and the baseline failed to finish at all in half its trials (363s+, one of them 11 consecutive single-tool turns).
The mechanism is visible in the transcripts. Baseline:
Treatment: the absolute path, once.
What did NOT work — and is trimmed accordingly
An earlier draft of this branch carried a long batching instruction quoting the profiling numbers. I A/B'd it on a six-independent-reads task and it showed no measurable effect, because both models already batch:
kimi-k2.5:per_turn = [6, 0]in all six trials, both arms — already optimaldeepseek-v4-flash-free: turns3,3,2baseline vs4,2,3treatment — noiseSince it ships on every turn, I cut it to one sentence: +160 tok/turn instead of +313. The intent stays; the prose that bought nothing measurable doesn't.
This also corrects an over-claim I made while profiling. The 212 "collapsible" read pairs are ~5% of turns, not a 2–3× lever, and the models mostly batch already.
Honest caveat on effect size
Under the trimmed wording, one of three re-runs regressed to 5 turns (from a consistent 2). The substitution is what does the work; the prose is a backstop.
whichcalls stayed at 0 across all 7 treatment trials under both wordings, so the mechanism is robust even where turn count varies.Tests
3 new: the substitution, the safe degradation to a bare command, and the two instructions. A pre-existing guard — "leaves no unknown
{{...}}placeholder after session-prep substitution" — caught the new placeholder immediately; it now covers it.853 passed, 6 skipped,
tsc --noEmitclean, built.Not addressed here
Context is still ~43k tokens per call (base
AGENTS.md~8.1k, a 37-skill index ~1.8k injected every turn regardless of need, plus 25 tool schemas and history). Trimming the always-on surface is the next lever —#244's two-tier skill work already points at it — but it wants its own measurement rather than being bundled in here.