Skip to content

perf(agents): hand the agent amico-run's absolute path — 28x faster on the task that needed it - #248

Draft
Rchari1 wants to merge 1 commit into
mainfrom
rchari/perf-path-and-batching
Draft

perf(agents): hand the agent amico-run's absolute path — 28x faster on the task that needed it#248
Rchari1 wants to merge 1 commit into
mainfrom
rchari/perf-path-and-batching

Conversation

@Rchari1

@Rchari1 Rchari1 commented Aug 3, 2026

Copy link
Copy Markdown
Member

The finding

AGENTS.md said bare amico-run and relied on PATH. When PATH missed, the agent went looking for the binary — and looking is expensive. Profiling one session's history:

  • 128 unbounded filesystem searches, 417 seconds total, individual calls at 123s
  • e.g. which amico-run || find /Users/raghavchari -type f -name "amico-run", and find ~/.julia -name "Piccolo" -type d at 126s

The fix

A new {{AMICO_RUN}} substitution (alongside the existing {{JULIA_PROJECT}} / {{TEMPLATE_PATH}}) resolves to the launcher's absolute path at session prep, and the guidance forbids which plus the find/pip list fallbacks.

Degrades safely: with no resolvable bin dir it substitutes the bare command — exactly today's behaviour.

Measured, A/B

Same model (amazon-bedrock/moonshotai.kimi-k2.5), same prompt, only AGENTS.md differing. Task: "Run amico-run to print its usage/help output." 4 baseline vs 7 treatment trials.

turns tool calls wall (median) which calls timeouts
baseline 6 10 226.6s 4 2 of 4
treatment 2 1 8.0s 0 0 of 7

~28× on median wall, 3× fewer turns, 10× fewer tool calls — and the baseline failed to finish at all in half its trials (363s+, one of them 11 consecutive single-tool turns).

The mechanism is visible in the transcripts. Baseline:

amico-run --help
which amico-run || which amico || echo "Not in PATH"
ls -la /usr/local/bin/amico* …
ls -la ~/.amico/bin/amico* …
find /usr -name "amico*" -type f …
find /opt -name "amico*" -type f …
pip list | grep -i amico …

Treatment: the absolute path, once.

What did NOT work — and is trimmed accordingly

An earlier draft of this branch carried a long batching instruction quoting the profiling numbers. I A/B'd it on a six-independent-reads task and it showed no measurable effect, because both models already batch:

  • kimi-k2.5: per_turn = [6, 0] in all six trials, both arms — already optimal
  • deepseek-v4-flash-free: turns 3,3,2 baseline vs 4,2,3 treatment — noise

Since it ships on every turn, I cut it to one sentence: +160 tok/turn instead of +313. The intent stays; the prose that bought nothing measurable doesn't.

This also corrects an over-claim I made while profiling. The 212 "collapsible" read pairs are ~5% of turns, not a 2–3× lever, and the models mostly batch already.

Honest caveat on effect size

Under the trimmed wording, one of three re-runs regressed to 5 turns (from a consistent 2). The substitution is what does the work; the prose is a backstop. which calls stayed at 0 across all 7 treatment trials under both wordings, so the mechanism is robust even where turn count varies.

Tests

3 new: the substitution, the safe degradation to a bare command, and the two instructions. A pre-existing guard — "leaves no unknown {{...}} placeholder after session-prep substitution" — caught the new placeholder immediately; it now covers it.

853 passed, 6 skipped, tsc --noEmit clean, built.

Not addressed here

Context is still ~43k tokens per call (base AGENTS.md ~8.1k, a 37-skill index ~1.8k injected every turn regardless of need, plus 25 tool schemas and history). Trimming the always-on surface is the next lever — #244's two-tier skill work already points at it — but it wants its own measurement rather than being bundled in here.

…n the task that needed it

AGENTS.md said bare `amico-run` and relied on PATH. When PATH missed, the agent
went looking for the binary, and looking is expensive: profiling one session's
history found 128 unbounded filesystem searches totalling 417 seconds, individual
ones at 123s.

A new {{AMICO_RUN}} substitution (alongside the existing {{JULIA_PROJECT}} and
{{TEMPLATE_PATH}}) resolves to the launcher's absolute path at session prep, and
the guidance forbids `which` and the find/pip fallbacks. Degrades safely: with no
resolvable bin dir it substitutes the bare command, i.e. today's behaviour.

MEASURED, A/B, same model (bedrock moonshotai.kimi-k2.5), same prompt, only
AGENTS.md differing — 4 baseline vs 7 treatment trials on "run amico-run and show
its usage":

                  turns   tool calls   wall (median)   `which` calls   timeouts
  baseline          6         10           226.6s            4          2 of 4
  treatment         2          1             8.0s            0          0 of 7

~28x on median wall, 3x fewer turns, 10x fewer tool calls, and the baseline failed
to finish at all in half its trials. The mechanism is visible in the transcripts:
baseline ran `which amico-run`, then `ls /usr/local/bin`, `ls ~/.amico/bin`,
`find /usr`, `find /opt`, `pip list`…; treatment ran the absolute path once.

WHAT DID NOT WORK, and is trimmed accordingly: an earlier draft carried a long
batching instruction with the profiling numbers in it. A/B on the six-independent-
reads task showed NO measurable effect, because both models already batch —
kimi-k2.5 emitted [6,0] in all six trials either way, and deepseek-v4-flash-free
was noise (turns 3,3,2 baseline vs 4,2,3 treatment). Since it ships on every turn,
it is cut to one sentence: +160 tok/turn instead of +313, keeping the intent
without paying for prose that bought nothing measurable.

Caveat on effect size: under the trimmed wording, one of three re-runs regressed
to 5 turns (from 2). The substitution is what does the work; the prose is a
backstop. `which` calls stayed at 0 across all 7 treatment trials under both
wordings.

Tests: 3 new — the substitution, the safe degradation to a bare command, and the
two instructions. A pre-existing guard ("leaves no unknown {{...}} placeholder")
caught the new placeholder immediately and now covers it. 853 passed, tsc clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Rchari1
Rchari1 marked this pull request as draft August 4, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant