From 1b7eb8257bc0ac8c63933e1b2b5bb070aa1975ca Mon Sep 17 00:00:00 2001 From: Shehab Yasser Date: Fri, 31 Jul 2026 22:22:57 +0300 Subject: [PATCH 1/6] fix: tell the optimizer to wait in bounded steps, not one silent half-hour block A swe-atlas-qna cell died at 71 minutes with `StreamTerminatedError` / "Connection lost", losing an optimization that had already earned a 0.1224 validation score. Nothing was actually broken. Harbor reads a remote agent through one long-lived stdout stream (`harbor/environments/modal.py`: `stdout = await process.stdout.read.aio()`), and a harness only flushes a command's output when that command RETURNS -- every one of opencode's 122 tool events in that run carried `status: "completed"`, none were partial. The instructions told the agent "a foreground call that blocks for half an hour is working correctly -- let it block", so it called `evals wait` and emitted nothing for 9m57s. The idle stream was reaped by the network path. Everything else was healthy: harbor downloaded artifacts out of that same sandbox four seconds before the failure, and `pmset` shows the machine never slept. Measured on that run: median gap between stdout events 6s, maximum gap 24.5 minutes. It is probabilistic rather than a fixed idle timeout -- the same stream survived the 24.5-minute gap and then died after ten minutes -- so the fix is to stop producing long silences at all. `evals wait --timeout N` already does exactly that: it prints the current status and exits 0 on expiry, and is idempotent so you just call it again. The mechanism existed; the instructions simply never pointed at it. This recommends `--detach` plus a `--timeout 300` loop for any long evaluation, capping silence at five minutes. The anti-backgrounding rule is preserved and now reads correctly alongside it: a bounded loop is still foreground and still blocks the whole time, it merely returns and re-enters, so it satisfies that rule rather than bending it. A heartbeat printed from inside `evals wait` was tried first and abandoned: it cannot work, because the harness withholds a tool's output until the tool returns, so the beats would arrive only once the wait was already over. Note the wording deliberately says "the outer trial is not retried" rather than naming a retry budget: `test_compiler_budget_disclosure_toggle` asserts the rendered instruction contains no "budget" when disclosure is off, and the first draft leaked it. Test plan: `test_compiler_budget_disclosure_toggle` passes. The 5 remaining failures in tests/test_v05_harbor_build.py reproduce identically on unmodified origin/main and are unrelated to this change. --- .../harbor/build/templates/instruction.md.j2 | 45 ++++++++++++++----- 1 file changed, 35 insertions(+), 10 deletions(-) diff --git a/vero/src/vero/harbor/build/templates/instruction.md.j2 b/vero/src/vero/harbor/build/templates/instruction.md.j2 index f6a9f678..24177638 100644 --- a/vero/src/vero/harbor/build/templates/instruction.md.j2 +++ b/vero/src/vero/harbor/build/templates/instruction.md.j2 @@ -25,8 +25,10 @@ hidden final evaluation. The trusted evaluation sidecar owns the cases, scoring, --partition {{ selection_partition }} ``` - An evaluation can take many minutes — that is expected; let the call block - and read the result it returns. Iterate cheaply on a subset first + An evaluation can take many minutes. For a **short** one, let the call block + and read the result it returns. For anything longer than a few minutes, use + the bounded wait loop below instead: a single call that blocks in silence for + half an hour can get your run killed. Iterate cheaply on a subset first (`--start 0 --stop N`, or repeated `--case-id ID`). {% if seed_supported %} Pass `--seed N` to reproduce a noisy comparison exactly. @@ -45,19 +47,42 @@ hidden final evaluation. The trusted evaluation sidecar owns the cases, scoring, and `evals diff OLD NEW`. Do not re-run an evaluation to see a number you truncated — look it up. - Add `--detach` **only** to run several evaluations at once: it returns a - `job_id` immediately instead of blocking. Then `evals wait JOB_ID` blocks - until that job finishes and prints its result (or poll `evals status JOB_ID`, - which also shows elapsed time). + Add `--detach` to run several evaluations at once, and to keep any long one + from going silent: it returns a `job_id` immediately instead of blocking. + Then wait for it in **bounded** steps rather than one open-ended call: + + ```bash + job=$(evals run --detach ... | grep -o '"job_id": *"[^"]*"' | cut -d'"' -f4) + until evals wait "$job" --timeout 300 | tee /dev/stderr | grep -q '"status": *"complete"'; do :; done + ``` + + `evals wait --timeout N` prints the current status and exits 0 when N seconds + pass, so you simply call it again; it is idempotent by design. Plain + `evals wait JOB_ID` with no `--timeout` blocks until the job finishes, and + `evals status JOB_ID` reports elapsed time. + + **Why bounded, and not one long block.** Your process is read through a + single long-lived stdout stream, and a harness only flushes a command's + output when that command *returns*. A wait that blocks for half an hour + therefore emits nothing for half an hour, and a stream idle that long can be + torn down by the network path even though the machine, the connection and the + sandbox are all healthy. When that happens the run dies with + `StreamTerminatedError` / "Connection lost", the outer trial is not retried, + and the whole optimization is lost. Observed 2026-07-31: a cell + died at 71 minutes, 9m57s into a silent wait, having already earned a 0.1224 + validation score. A `--timeout 300` loop caps that silence at five minutes + and costs nothing, because each return is just one more line of output. **Run every `evals` call in the foreground.** You are a single-shot headless run: nothing exists to deliver a notification or wake you later. If you put a long call in a background task, schedule a wake-up, or say you will "report back when it finishes", the run simply ends there and whatever you have not - submitted is lost. A foreground call that blocks for half an hour is working - correctly — let it block. To wait on two jobs, `evals wait` the first, then - the second. Do not claim an improvement before its comparison baseline has - been scored on the same cases. + submitted is lost. The bounded wait loop above is still foreground: it blocks + the whole time, it just returns and re-enters every few minutes instead of + sitting silent, so it satisfies this rule rather than bending it. A call that + blocks for a few minutes is working correctly — let it block. To wait on two + jobs, wait out the first, then the second. Do not claim an improvement before + its comparison baseline has been scored on the same cases. 4. Use `evals status` to inspect evaluation jobs and allowed evaluation sets{% if disclose_budget %}, and to see remaining budgets{% endif %}. From 134b4efbfaa9e84f1a27f3371cb6c7a01e1f2af0 Mon Sep 17 00:00:00 2001 From: Shehab Yasser Date: Sat, 1 Aug 2026 08:58:24 +0300 Subject: [PATCH 2/6] fix: stop recommending a wait loop that never terminates Greptile flagged that the example loop spins forever on a `failed` or `cancelled` job: `evals wait` returns that terminal status immediately, the loop matched only `complete`, so every subsequent wait also returned immediately. It is worse than reported. On SUCCESS the loop does not terminate either. `wait_command` prints the evaluation *result* on `complete` (`GET /eval/jobs/{id}/result`) and only prints the job record on the non-complete terminal states, so the string the loop grepped for is absent from the output in exactly the case the loop was waiting for. Both arms spin. Shipping a shell one-liner into a prompt was the underlying mistake: it is fragile, it encodes assumptions about output shape that the CLI never promised, and an optimizer copying it verbatim inherits every one of them. Replaced with the two commands plus prose that states the contract: `--timeout N` returns on terminal state or after N seconds, exits 0 either way, and terminality is read from `evals status`, not from the text of the wait output. The failure mode is named explicitly so a model reconstructing a loop does not rebuild the same bug. Placeholder style kept as `JOB_ID` to match the rest of the document; `test_compiler_emits_isolated_canonical_harbor_task` pins that surface and caught the drift when this first used ``. tests/test_v05_harbor_build.py: 41 passed (credentials exported; without them six tests fail on compiler.py:566 credential validation). --- .../harbor/build/templates/instruction.md.j2 | 22 ++++++++++++++----- 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/vero/src/vero/harbor/build/templates/instruction.md.j2 b/vero/src/vero/harbor/build/templates/instruction.md.j2 index 24177638..41cc02b6 100644 --- a/vero/src/vero/harbor/build/templates/instruction.md.j2 +++ b/vero/src/vero/harbor/build/templates/instruction.md.j2 @@ -52,14 +52,24 @@ hidden final evaluation. The trusted evaluation sidecar owns the cases, scoring, Then wait for it in **bounded** steps rather than one open-ended call: ```bash - job=$(evals run --detach ... | grep -o '"job_id": *"[^"]*"' | cut -d'"' -f4) - until evals wait "$job" --timeout 300 | tee /dev/stderr | grep -q '"status": *"complete"'; do :; done + evals run --detach --backend ... --evaluation-set ... --partition ... + # -> prints a job_id + evals wait JOB_ID --timeout 300 + # -> returns within 5 minutes whether or not the job is done; call it again + # while it is still running ``` - `evals wait --timeout N` prints the current status and exits 0 when N seconds - pass, so you simply call it again; it is idempotent by design. Plain - `evals wait JOB_ID` with no `--timeout` blocks until the job finishes, and - `evals status JOB_ID` reports elapsed time. + `evals wait --timeout N` returns when the job reaches a terminal state OR + after N seconds, whichever comes first, and exits 0 either way, so calling it + again is always safe. A job is terminal when `evals status JOB_ID` reports + `complete`, `failed` or `cancelled`. **Check the status; do not loop on the + text of the wait output.** On success `evals wait` prints the evaluation + *result*, not the job record, so a loop that waits for the word "complete" in + that output never terminates, and on a `failed` or `cancelled` job every + further wait returns instantly, turning the same loop into a busy spin that + burns the rest of your run. Plain `evals wait JOB_ID` with no `--timeout` + blocks until the job finishes, which is fine for a short evaluation and is + the thing to avoid for a long one. **Why bounded, and not one long block.** Your process is read through a single long-lived stdout stream, and a harness only flushes a command's From d12cf5bd8c7f0f0d63e2e59b9185283ee520350a Mon Sep 17 00:00:00 2001 From: Shehab Yasser Date: Sat, 1 Aug 2026 11:27:55 +0300 Subject: [PATCH 3/6] fix: bound the optimizer's tool calls from the harness, not the prompt The two commits under this PR diagnosed the failure correctly and then fixed it in the one place that cannot enforce anything. The optimizer's stdout reaches harbor as one long-lived stream, a harness flushes a command's output only when that command returns, and an idle stream gets reaped while the machine, the connection and the sandbox stay healthy. So "how long may one tool call run" and "how long may the stream go silent" are the same question, and the outer trial is not retried when the answer is too long. Asking the optimizer to wait in bounded steps is the right shape at the wrong layer. The evidence is in this branch's own history: the recipe shipped in the first commit spun forever on both arms and had to be replaced in the second. A prompt is advisory, a model reconstructing the loop from memory reintroduces the failure, and nothing in the system notices when it does. Both harnesses already expose the bound as a setting, so set it. HARNESS_TOOL_TIMEOUT_SECONDS = 300 goes out per harness through harbor's --ae seam: opencode reads OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS, claude-code reads BASH_DEFAULT_TIMEOUT_MS and BASH_MAX_TIMEOUT_MS. Harnesses with no verified knob (codex, mini-swe-agent) are sent nothing rather than a variable they ignore. It is placed ahead of the build's own agent_env so a build can raise or drop the cap by naming the same variable, since harbor's parse_env_vars keeps the last value for a key. opencode's is a default only: packages/opencode/src/tool/shell.ts resolves `flags.bashDefaultTimeoutMs ?? 2 * 60 * 1000` and then `params.timeout ?? defaultTimeoutMs` with no clamp, so a model naming its own timeout still escapes it. It covers the case that actually killed the run, where the instruction said to let the call block and the model therefore never named one, and it lowers the number quoted in the tool description the model reads. claude-code's MAX is a true ceiling. The cap alone would be worse than nothing: a five-minute kill on a thirty-minute `evals run` hands the model opencode's own "retry with a larger timeout" message, which restores the silence and burns an evaluation. So `evals run` now always starts a job and polls it internally on a 240s bound, 255s worst case, returning the job record with its job_id when the bound expires. `evals wait` resumes, and the evaluation keeps running in the sidecar throughout. Same evaluation as before: the sidecar drives POST /eval and POST /eval/jobs through the same _execute_tracked_job, same budget, same SidecarEvaluationResult. The detached path records failures on the job record instead of raising, so _await_evaluation_job reads the reason back off it and raises a ClickException, preserving today's non-zero exit. The instruction template loses the recipe, the contract essay and the incident narrative that the two earlier commits added, about 45 lines, and states the behavior in four. The number it quotes is read from WAIT_TIMEOUT_SECONDS through the template context rather than restated, so prompt and code cannot drift. SKILL.md and docs/guide.md follow. test_harbor_cli_builds_canonical_selection pinned the old contract that a plain run posts to /eval; it now pins /eval/jobs. That was the only existing test the change broke. Test plan: 490 passed, 15 skipped (485 before, five new tests: the per-harness bound including the HARNESS_TOOL_TIMEOUT_SECONDS > WAIT_TIMEOUT_SECONDS ordering, a build overriding the cap through agent_env, and the three _await_evaluation_job paths, bound expiry, completion and failure). Credentials must be exported or five unrelated tests fail on compiler.py credential validation. Not verified end to end. The failure is probabilistic, the same stream in the dead run survived a 24.5-minute gap and then died after ten, so a live A/B would need many full optimization runs to say anything. What is verified is that the variables reach the harness and that the CLI returns inside the cap. Co-Authored-By: Claude Opus 5 (1M context) --- vero/docs/guide.md | 14 ++- vero/src/vero/evals_cli.py | 53 ++++++-- vero/src/vero/harbor/build/compiler.py | 4 + .../harbor/build/templates/instruction.md.j2 | 57 ++------- vero/src/vero/harbor/cli.py | 118 ++++++++++++++++-- vero/src/vero/skills/evals/SKILL.md | 22 ++-- vero/tests/test_v05_cli.py | 107 ++++++++++++++++ vero/tests/test_v05_harbor_http.py | 80 +++++++++++- 8 files changed, 376 insertions(+), 79 deletions(-) diff --git a/vero/docs/guide.md b/vero/docs/guide.md index 1cdf6969..cbe22d8f 100644 --- a/vero/docs/guide.md +++ b/vero/docs/guide.md @@ -96,11 +96,15 @@ build.yaml --output task` compiles without running, for inspection. > Harbor constructs the agent; a raw `harbor run` would let the agent adapter > read the upstream key from its own host process first. -Inside the container the agent evaluates candidates with `evals run ---detach`, then `evals status JOB` / `evals result JOB` / `evals status` (via `VERO_EVAL_URL`). -Detached evaluations are **durable jobs** — the candidate version is captured -before the command returns, so ending the agent process can't lose or race a -running measurement. +Inside the container the agent evaluates candidates with `evals run` (via +`VERO_EVAL_URL`), which waits for the result and, if the evaluation outlives +that wait's bound, hands back a `job_id` for `evals wait JOB`; `--detach` +returns the id immediately instead, for evaluating concurrently. Either way the +evaluation is a **durable job**: the candidate version is captured before the +command returns, so ending the agent process can't lose or race a running +measurement. The bound matters because vero also caps how long one of the +optimizer's tool calls may run (`HARNESS_TOOL_TIMEOUT_SECONDS`), so an `evals` +call has to return on its own terms rather than be killed mid-evaluation. ### How the boundaries hold diff --git a/vero/src/vero/evals_cli.py b/vero/src/vero/evals_cli.py index 006e7ca1..4503f53a 100644 --- a/vero/src/vero/evals_cli.py +++ b/vero/src/vero/evals_cli.py @@ -26,6 +26,25 @@ CONTEXT_DIRECTORY = ".evals" _CELL_WIDTH = 48 +# How long a blocking `evals run` / `evals wait` waits before returning what it +# has so far. +# +# Not ergonomics: the optimizer's whole process is read through one long-lived +# stdout stream, and an agent harness only flushes a command's output when that +# command *returns*. A call that sits silent long enough for the network path to +# reap that stream kills the trial, and the outer trial is not retried. Observed +# 2026-07-31: a cell died at 71 minutes, 9m57s into a single silent wait, +# discarding a candidate that had already scored 0.1224 on 49 validation cases. +# Returning on a bound is what keeps the stream alive, and each return costs one +# line of output. +# +# Deliberately below HARNESS_TOOL_TIMEOUT_SECONDS (`vero/harbor/cli.py`), the cap +# vero sets on the optimizer's harness: these commands must return on their own +# terms, handing back a job id that can be waited on again, rather than be killed +# by the harness and hand the model a "retry with a larger timeout" message. +WAIT_TIMEOUT_SECONDS = 240.0 +WAIT_POLL_INTERVAL_SECONDS = 15.0 + # -------------------------------------------------------------------------- # Context discovery and shared helpers @@ -301,10 +320,11 @@ def evals() -> None: resources), `candidates/` (prior program versions), and `plan.json` (what you may evaluate, and remaining budget). - Typical loop: `evals plan` -> edit + commit -> `evals run` (blocks and - returns the result) -> `evals diff BASELINE CANDIDATE` -> - `evals cases ID --sort score` -> `evals trace ID CASE`. Add `--detach` only - to run several evaluations at once, then poll `evals status JOB`. + Typical loop: `evals plan` -> edit + commit -> `evals run` (waits for the + result, or returns a `job_id` to `evals wait` on) -> `evals diff BASELINE + CANDIDATE` -> `evals cases ID --sort score` -> `evals trace ID CASE`. Add + `--detach` only to run several evaluations at once, then poll + `evals status JOB`. """ @@ -384,34 +404,41 @@ def status_command(job_id): @click.argument("job_id") @click.option( "--poll-interval", - default=15.0, + default=WAIT_POLL_INTERVAL_SECONDS, show_default=True, type=click.FloatRange(min=1), help="Seconds between status polls.", ) @click.option( "--timeout", + default=WAIT_TIMEOUT_SECONDS, + show_default=True, type=click.FloatRange(min=0), - help="Optional max seconds to wait. On expiry, print the current " - "(still-running) status and exit 0 so you can simply call `evals wait` " - "again. Default: wait until the job finishes.", + help="Max seconds to wait. On expiry, print the current (still-running) " + "status and exit 0, so calling `evals wait` again resumes the wait. The " + "default is bounded on purpose; raising it risks the run (see the docstring).", ) def wait_command(job_id, poll_interval, timeout): - """Block until a detached job finishes, then print its result. + """Wait for a detached job, then print its result. The blocking companion to `evals run --detach`: one call that waits, so you - never hand-roll a poll loop. Idempotent — safe to call again if it returns - while the job is still running (only happens when --timeout is set). + never hand-roll a poll loop. Idempotent, so it is always safe to call again + when it returns while the job is still running. + + The wait is *bounded* by default rather than open-ended. A call that returns + nothing for tens of minutes can get the whole run killed (see + WAIT_TIMEOUT_SECONDS), and re-entering costs one line of output, so the + bound is the default rather than something to opt into. """ request = _sidecar_request() terminal = {"complete", "failed", "cancelled"} - deadline = None if timeout is None else time.monotonic() + timeout + deadline = time.monotonic() + timeout while True: job = request("GET", f"/eval/jobs/{job_id}") status = job.get("status") if isinstance(job, dict) else None if status in terminal: break - if deadline is not None and time.monotonic() >= deadline: + if time.monotonic() >= deadline: click.echo(json.dumps(_enrich_job(job), indent=2)) return time.sleep(poll_interval) diff --git a/vero/src/vero/harbor/build/compiler.py b/vero/src/vero/harbor/build/compiler.py index 9f2cefce..59297a08 100644 --- a/vero/src/vero/harbor/build/compiler.py +++ b/vero/src/vero/harbor/build/compiler.py @@ -15,6 +15,7 @@ from importlib.metadata import version as distribution_version from pathlib import Path, PurePosixPath +from vero.evals_cli import WAIT_TIMEOUT_SECONDS from vero.evaluation import ( EvaluationBudget, EvaluationLimits, @@ -799,6 +800,9 @@ def compile_harbor_task( ], "exhaust_budget": config.instruct_exhaust_budget, "disclose_budget": config.disclose_budget, + # Read from the CLI constant rather than restated here, so the number the + # instruction quotes cannot drift from the one `evals run` enforces. + "wait_timeout_seconds": WAIT_TIMEOUT_SECONDS, "build_timeout": config.build_timeout_seconds, "verifier_timeout": ( config.verifier_timeout_seconds or max(1, int(config.timeout_seconds)) diff --git a/vero/src/vero/harbor/build/templates/instruction.md.j2 b/vero/src/vero/harbor/build/templates/instruction.md.j2 index 41cc02b6..89cb9267 100644 --- a/vero/src/vero/harbor/build/templates/instruction.md.j2 +++ b/vero/src/vero/harbor/build/templates/instruction.md.j2 @@ -25,11 +25,11 @@ hidden final evaluation. The trusted evaluation sidecar owns the cases, scoring, --partition {{ selection_partition }} ``` - An evaluation can take many minutes. For a **short** one, let the call block - and read the result it returns. For anything longer than a few minutes, use - the bounded wait loop below instead: a single call that blocks in silence for - half an hour can get your run killed. Iterate cheaply on a subset first - (`--start 0 --stop N`, or repeated `--case-id ID`). + An evaluation can take many minutes. `evals run` waits for it and prints the + result. If it is still running after {{ wait_timeout_seconds | int }}s the + command returns the job record instead, carrying a `job_id`; the evaluation + keeps running and `evals wait JOB_ID` resumes the wait. Iterate cheaply on a + subset first (`--start 0 --stop N`, or repeated `--case-id ID`). {% if seed_supported %} Pass `--seed N` to reproduce a noisy comparison exactly. {% else %} @@ -47,50 +47,19 @@ hidden final evaluation. The trusted evaluation sidecar owns the cases, scoring, and `evals diff OLD NEW`. Do not re-run an evaluation to see a number you truncated — look it up. - Add `--detach` to run several evaluations at once, and to keep any long one - from going silent: it returns a `job_id` immediately instead of blocking. - Then wait for it in **bounded** steps rather than one open-ended call: - - ```bash - evals run --detach --backend ... --evaluation-set ... --partition ... - # -> prints a job_id - evals wait JOB_ID --timeout 300 - # -> returns within 5 minutes whether or not the job is done; call it again - # while it is still running - ``` - - `evals wait --timeout N` returns when the job reaches a terminal state OR - after N seconds, whichever comes first, and exits 0 either way, so calling it - again is always safe. A job is terminal when `evals status JOB_ID` reports - `complete`, `failed` or `cancelled`. **Check the status; do not loop on the - text of the wait output.** On success `evals wait` prints the evaluation - *result*, not the job record, so a loop that waits for the word "complete" in - that output never terminates, and on a `failed` or `cancelled` job every - further wait returns instantly, turning the same loop into a busy spin that - burns the rest of your run. Plain `evals wait JOB_ID` with no `--timeout` - blocks until the job finishes, which is fine for a short evaluation and is - the thing to avoid for a long one. - - **Why bounded, and not one long block.** Your process is read through a - single long-lived stdout stream, and a harness only flushes a command's - output when that command *returns*. A wait that blocks for half an hour - therefore emits nothing for half an hour, and a stream idle that long can be - torn down by the network path even though the machine, the connection and the - sandbox are all healthy. When that happens the run dies with - `StreamTerminatedError` / "Connection lost", the outer trial is not retried, - and the whole optimization is lost. Observed 2026-07-31: a cell - died at 71 minutes, 9m57s into a silent wait, having already earned a 0.1224 - validation score. A `--timeout 300` loop caps that silence at five minutes - and costs nothing, because each return is just one more line of output. + Add `--detach` **only** to run several evaluations at once: it returns a + `job_id` immediately without waiting. `evals wait JOB_ID` then waits for one + on the same bound, and `evals status JOB_ID` reports its state and elapsed + time without waiting at all. A job is finished when its status is `complete`, + `failed` or `cancelled`; any other status means it is still running, so wait + on it again. **Run every `evals` call in the foreground.** You are a single-shot headless run: nothing exists to deliver a notification or wake you later. If you put a long call in a background task, schedule a wake-up, or say you will "report back when it finishes", the run simply ends there and whatever you have not - submitted is lost. The bounded wait loop above is still foreground: it blocks - the whole time, it just returns and re-enters every few minutes instead of - sitting silent, so it satisfies this rule rather than bending it. A call that - blocks for a few minutes is working correctly — let it block. To wait on two + submitted is lost. A call that waits for minutes and hands back a + still-running job is working correctly: wait on it again. To wait on two jobs, wait out the first, then the second. Do not claim an improvement before its comparison baseline has been scored on the same cases. diff --git a/vero/src/vero/harbor/cli.py b/vero/src/vero/harbor/cli.py index 90562c56..66502b73 100644 --- a/vero/src/vero/harbor/cli.py +++ b/vero/src/vero/harbor/cli.py @@ -11,12 +11,18 @@ import subprocess import sys import tempfile +import time import urllib.error import urllib.request from pathlib import Path import click +from vero.evals_cli import ( + WAIT_POLL_INTERVAL_SECONDS, + WAIT_TIMEOUT_SECONDS, + _enrich_job, +) from vero.evaluation import ( CaseIds, CaseRange, @@ -231,6 +237,42 @@ def _load_env_file(path: Path) -> dict[str, str]: # the gateway token cap are the intended limits. OPENCODE_STEP_LIMIT = 1000 +# The ceiling vero puts on a single optimizer tool call, in seconds. +# +# The optimizer's stdout reaches harbor as one long-lived stream, and an agent +# harness only flushes output when a tool call *returns*. "How long may one tool +# call run" is therefore the same question as "how long may that stream go +# silent", and an idle stream gets reaped by the network path while the machine, +# the connection and the sandbox all stay healthy. The outer trial is not +# retried, so the whole optimization goes with it: on 2026-07-31 a cell died at +# 71 minutes, 9m57s into one silent call, discarding a candidate that had +# already scored 0.1224 on 49 validation cases. +# +# Configured on the harness rather than requested in the instruction, because +# the instruction is advisory and this is not. Telling the optimizer to wait in +# bounded steps leaves it free to ignore the advice, and free to reconstruct the +# loop wrongly. Sits above WAIT_TIMEOUT_SECONDS so the evals CLI always returns +# first on its own terms; the cap is the backstop for everything else the +# optimizer runs. +HARNESS_TOOL_TIMEOUT_SECONDS = 300 + +# How each harness spells "bound one tool call". Only knobs verified in the +# harness's own source or docs are listed: a harness missing here keeps its own +# default rather than being sent a variable it silently ignores. +_TOOL_TIMEOUT_ENVIRONMENT: dict[str, tuple[str, ...]] = { + # opencode reads a *default* only. `packages/opencode/src/tool/shell.ts` + # resolves `flags.bashDefaultTimeoutMs ?? 2 * 60 * 1000` and then + # `params.timeout ?? defaultTimeoutMs` with no clamp, so a model that names + # its own timeout still escapes the bound. It covers the case that actually + # killed the run -- the instruction said to let the call block, so the model + # never named one -- and it lowers the default quoted in the tool + # description the model reads. + "opencode": ("OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS",), + # claude-code takes both, and its MAX is a true ceiling: a per-call timeout + # from inside the conversation cannot raise it. + "claude-code": ("BASH_DEFAULT_TIMEOUT_MS", "BASH_MAX_TIMEOUT_MS"), +} + # Harnesses that drive the model through litellm rather than a provider SDK. # litellm reads the base URL as _API_BASE; the SDKs read # _BASE_URL. vero sets the SDK names, so a litellm-based harness sees no @@ -375,6 +417,28 @@ def _opencode_gateway_args(agent: str, model: str | None, task: Path) -> list[st return ["--ak", f"opencode_config={json.dumps(payload, separators=(',', ':'))}"] +def _agent_tool_timeout_args(agent: str) -> list[str]: + """Bound one optimizer tool call, so no single call can idle the stream. + + The mechanism the previous fix reached for was the instruction: it told the + optimizer to wait in bounded steps instead of one open-ended block. That is + the right shape and the wrong layer. A prompt cannot enforce anything, the + recipe it shipped had to be corrected twice in review, and a model that + reconstructs the loop from memory reintroduces the failure. The harnesses + already expose the bound as a setting; set it. + + Returns `--ae NAME=VALUE` pairs, which harbor merges into the scoped exec env + wrapping the agent's run phase. Values are milliseconds, the unit every + harness here uses. + """ + + milliseconds = int(HARNESS_TOOL_TIMEOUT_SECONDS * 1000) + arguments: list[str] = [] + for name in _TOOL_TIMEOUT_ENVIRONMENT.get(agent, ()): + arguments.extend(["--ae", f"{name}={milliseconds}"]) + return arguments + + def _outer_app_name_args( environment: str, config_name: str, extra: tuple[str, ...] ) -> list[str]: @@ -806,6 +870,10 @@ def run_command(config_path, agent, model, environment, params, env_file, extra) ] if model is not None: command.extend(["-m", model]) + # Ahead of the build's own agent env, so a build can raise or drop the + # cap by naming the same variable: harbor keeps the last value for a key, + # and a vero default must not silently outrank an explicit choice. + command.extend(_agent_tool_timeout_args(agent)) # Forward the build's declared agent env to the optimizer agent's shell. # Harbor's `--ae KEY=VALUE` populates the agent's extra_env, which harbor # injects into the agent's setup/install exec (scoped_exec_env). Sorted @@ -887,6 +955,43 @@ def inference_gateway_command(config_path, host, port): serve_inference_gateway(config_path=config_path, host=host, port=port) +def _await_evaluation_job(job: dict, timeout: float = WAIT_TIMEOUT_SECONDS) -> dict: + """Wait out a started evaluation job, bounded, and return what to print. + + A blocking `POST /eval` is one HTTP call that can take half an hour and + prints nothing until it returns, which is precisely the silence that killed + an optimization on 2026-07-31. The sidecar drives both entry points through + the same tracked job (`Sidecar._execute_tracked_job`), so starting a job and + polling it is the same evaluation, the same budget and the same + `SidecarEvaluationResult` -- only interruptible. + + Returns the evaluation result once the job completes, or the job record when + the bound expires first, in which case the caller re-enters with + `evals wait JOB_ID` and the evaluation keeps running in the sidecar + regardless. A failed or cancelled job raises, so a bounded run still exits + non-zero carrying the sidecar's own reason, exactly as the blocking call did. + """ + + job_id = job.get("job_id") if isinstance(job, dict) else None + if not job_id: + return job + terminal = {"complete", "failed", "cancelled"} + deadline = time.monotonic() + timeout + while True: + status = job.get("status") + if status in terminal: + break + if time.monotonic() >= deadline: + return _enrich_job(job) + time.sleep(WAIT_POLL_INTERVAL_SECONDS) + job = _request("GET", f"/eval/jobs/{job_id}") + if status == "complete": + return _request("GET", f"/eval/jobs/{job_id}/result") + raise click.ClickException( + f"evaluation job {job_id} {status}: {job.get('error') or 'no reason recorded'}" + ) + + @harbor.command("eval") @click.option( "--backend", "backend_id", required=True, @@ -1018,15 +1123,12 @@ def evaluate_command( limits=EvaluationLimits(**limit_values) if limit_values else None, seed=seed, ) + payload = body.model_dump(mode="json") + if detach: + click.echo(json.dumps(_request("POST", "/eval/jobs", payload=payload), indent=2)) + return click.echo( - json.dumps( - _request( - "POST", - "/eval/jobs" if detach else "/eval", - payload=body.model_dump(mode="json"), - ), - indent=2, - ) + json.dumps(_await_evaluation_job(_request("POST", "/eval/jobs", payload=payload)), indent=2) ) diff --git a/vero/src/vero/skills/evals/SKILL.md b/vero/src/vero/skills/evals/SKILL.md index 3381c081..b9435112 100644 --- a/vero/src/vero/skills/evals/SKILL.md +++ b/vero/src/vero/skills/evals/SKILL.md @@ -40,20 +40,26 @@ you truncated — it is on disk. ## Blocking vs detached -By default `evals run` **blocks** until scoring finishes and returns the result -— one call, nothing to track. Evaluations can take many minutes; that is -expected, so let it block. This is what you want almost always. +By default `evals run` **waits** for scoring and returns the result: one call, +nothing to track. This is what you want almost always. Evaluations can take many +minutes, and a wait is *bounded*: if the evaluation is still running when the +bound expires, the command returns the job record (with its `job_id`) instead of +the result and exits 0. Nothing is lost when that happens, the evaluation keeps +running in the sidecar, and `evals wait JOB_ID` resumes waiting for it. Waiting +again is always safe. Use `--detach` **only** to run several evaluations concurrently: it returns a -`job_id` immediately instead of blocking. Then `evals wait JOB_ID` blocks until -that job finishes and prints its result; or poll `evals status JOB_ID`, which -now also reports `elapsed_seconds` (and `requested_cases` for a subset) so you -can see it is progressing. To wait on two jobs, wait the first, then the second. +`job_id` immediately without waiting. Then `evals wait JOB_ID` waits for one on +the same bound and prints its result; or poll `evals status JOB_ID`, which also +reports `elapsed_seconds` (and `requested_cases` for a subset) so you can see it +is progressing. A job is finished when its status is `complete`, `failed` or +`cancelled`. To wait on two jobs, wait the first, then the second. Run every `evals` call in the **foreground**. If you are a headless single-shot run, nothing can wake you: putting a long call in a background task, scheduling a wake-up, or promising to "report back when it finishes" ends the run right -there. A call that blocks for half an hour is working correctly. +there. A call that waits for minutes and hands back a still-running job is +working correctly; wait on it again. ## Run options worth knowing (`evals run --help` for all) diff --git a/vero/tests/test_v05_cli.py b/vero/tests/test_v05_cli.py index 335a52a0..f7191350 100644 --- a/vero/tests/test_v05_cli.py +++ b/vero/tests/test_v05_cli.py @@ -416,6 +416,113 @@ def test_opencode_gets_a_step_limit_that_does_not_truncate_the_search(tmp_path): assert _opencode_gateway_args("claude-code", "claude-sonnet-5", tmp_path) == [] +def test_the_harness_is_configured_to_bound_one_tool_call(): + """A silent tool call is what kills a run, so the bound goes on the harness. + + The optimizer's stdout reaches harbor as one long-lived stream and a harness + flushes a command's output only when that command returns, so the length of + one tool call *is* the length of the stream's silence. On 2026-07-31 a cell + died at 71 minutes, 9m57s into a single silent call, discarding a candidate + already scored at 0.1224 on 49 validation cases. + + The first fix for this asked the optimizer, in the instruction, to wait in + bounded steps. A prompt enforces nothing: the recipe it shipped had to be + corrected twice in review, and a model reconstructing the loop from memory + reintroduces the failure. Both harnesses expose the bound as a setting. + """ + from vero.evals_cli import WAIT_TIMEOUT_SECONDS + from vero.harbor.cli import HARNESS_TOOL_TIMEOUT_SECONDS, _agent_tool_timeout_args + + milliseconds = str(HARNESS_TOOL_TIMEOUT_SECONDS * 1000) + + # opencode exposes a default only: packages/opencode/src/tool/shell.ts reads + # `flags.bashDefaultTimeoutMs ?? 2 * 60 * 1000`, then `params.timeout ?? + # defaultTimeoutMs` with no clamp. It covers the case that actually killed + # the run, where the instruction said to let the call block so the model + # never named a timeout of its own. + assert _agent_tool_timeout_args("opencode") == [ + "--ae", + f"OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS={milliseconds}", + ] + + # claude-code takes a true ceiling as well: a per-call timeout chosen inside + # the conversation cannot raise BASH_MAX_TIMEOUT_MS. + claude = _agent_tool_timeout_args("claude-code") + assert claude[0::2] == ["--ae", "--ae"] + assert claude[1::2] == [ + f"BASH_DEFAULT_TIMEOUT_MS={milliseconds}", + f"BASH_MAX_TIMEOUT_MS={milliseconds}", + ] + + # A harness whose knob is not verified is left at its own default rather + # than handed a variable it silently ignores. + assert _agent_tool_timeout_args("codex") == [] + assert _agent_tool_timeout_args("mini-swe-agent") == [] + + # The evals CLI has to return on its own terms first. If the harness killed + # the call instead, the model would be handed opencode's "retry with a + # larger timeout" message, take the advice, and restore the silence. + assert HARNESS_TOOL_TIMEOUT_SECONDS > WAIT_TIMEOUT_SECONDS + + +def test_a_build_can_override_the_tool_call_bound_it_is_given(tmp_path, monkeypatch): + """vero's cap is a default, not a seizure of the variable. + + Harbor keeps the last `--ae` value for a key, so ordering decides who wins. + A build that legitimately needs a longer single call has to be able to say + so through `agent_env`, the same way it declares any other agent variable. + """ + from vero.harbor import build as harbor_build + from vero.harbor import cli as harbor_cli + + config_path = tmp_path / "build.yaml" + config_path.write_text("name: org/task\n", encoding="utf-8") + + class _Config: + harbor_requirement = "harbor[modal]==0.20.0" + agent_env = {"OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS": "900000"} + optimizer_harbor_args: list[str] = [] + extra_harbor_args: list[str] = [] + name = "vero/stub-benchmark" + + monkeypatch.setattr(harbor_build, "load_harbor_build_config", lambda *a, **k: _Config()) + monkeypatch.setattr(harbor_build, "compile_harbor_task", lambda config, output: output) + monkeypatch.setattr(harbor_cli.shutil, "which", lambda name: "/usr/bin/uvx") + monkeypatch.setattr(harbor_cli, "_compiled_run_environment", lambda task, overrides: {}) + + recorded: list[list[str]] = [] + + def _record(command, env=None): + recorded.append(command) + return subprocess.CompletedProcess(command, 0) + + monkeypatch.setattr(harbor_cli.subprocess, "run", _record) + + result = CliRunner().invoke( + main, + [ + "harbor", + "run", + "--config", + str(config_path), + "--agent", + "opencode", + "--model", + "anthropic/claude-sonnet-5", + "--yes", + ], + ) + + assert result.exit_code == 0, result.output + command = recorded[0] + default = ( + "OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS=" + f"{harbor_cli.HARNESS_TOOL_TIMEOUT_SECONDS * 1000}" + ) + override = "OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS=900000" + assert command.index(default) < command.index(override) + + def test_litellm_harnesses_get_the_gateway_url_under_the_name_they_read(tmp_path): """litellm reads _API_BASE; the provider SDKs read _BASE_URL. diff --git a/vero/tests/test_v05_harbor_http.py b/vero/tests/test_v05_harbor_http.py index 6c9e14dc..0bc4b3a5 100644 --- a/vero/tests/test_v05_harbor_http.py +++ b/vero/tests/test_v05_harbor_http.py @@ -530,7 +530,11 @@ def fake_request(method, path, *, payload=None, headers=None): assert result.exit_code == 0, result.output assert captured["method"] == "POST" - assert captured["path"] == "/eval" + # Even a plain (non-detached) run starts a job and waits on it, rather than + # holding one open-ended `POST /eval` that prints nothing until it returns. + # Same evaluation either way: the sidecar drives both entry points through + # `_execute_tracked_job`. + assert captured["path"] == "/eval/jobs" assert captured["payload"]["evaluation_set"]["selection"] == { "kind": "ids", "ids": ["a", "b"], @@ -571,6 +575,80 @@ def fake_request(method, path, *, payload=None, headers=None): assert requests[2][:2] == ("GET", "/eval/jobs/job-1/result") +def test_a_plain_run_hands_back_a_waitable_job_when_its_bound_expires(monkeypatch): + """The wait ends before the harness would kill it, and loses nothing. + + The optimizer's harness caps one tool call, so `evals run` has to return on + its own terms while the evaluation is still running: it prints the job + record, enriched with elapsed time so the next call can see progress, and + exits 0. `evals wait JOB_ID` resumes. Were the harness to time the call out + instead, the model would be told to retry with a larger timeout and would + walk straight back into the silence that killed the trial. + """ + from vero.harbor.cli import _await_evaluation_job + + requests = [] + + def fake_request(method, path, *, payload=None, headers=None): + requests.append((method, path)) + raise AssertionError("an expired bound must not poll again") + + monkeypatch.setattr("vero.harbor.cli._request", fake_request) + running = { + "job_id": "job-1", + "status": "running", + "created_at": "2026-07-31T00:00:00+00:00", + } + + awaited = _await_evaluation_job(running, timeout=0) + + assert awaited["job_id"] == "job-1" + assert awaited["status"] == "running" + assert awaited["elapsed_seconds"] > 0 + assert requests == [] + + +def test_a_finished_job_is_awaited_into_its_result(monkeypatch): + """A job that is already terminal costs no extra wait and no poll interval.""" + from vero.harbor.cli import _await_evaluation_job + + requests = [] + + def fake_request(method, path, *, payload=None, headers=None): + requests.append((method, path)) + return {"metrics": {"score": 0.1224}} + + monkeypatch.setattr("vero.harbor.cli._request", fake_request) + + awaited = _await_evaluation_job({"job_id": "job-1", "status": "complete"}) + + assert awaited == {"metrics": {"score": 0.1224}} + assert requests == [("GET", "/eval/jobs/job-1/result")] + + +def test_a_failed_job_still_fails_the_command_with_the_sidecar_reason(monkeypatch): + """Detaching internally must not turn a failed evaluation into a success. + + The blocking `POST /eval` path raised through to a non-zero exit; the job + path records the reason on the job record instead, so waiting has to read it + back off the record and raise. + """ + from vero.harbor.cli import _await_evaluation_job + + def fake_request(method, path, *, payload=None, headers=None): + raise AssertionError("a failed job has no result to fetch") + + monkeypatch.setattr("vero.harbor.cli._request", fake_request) + + with pytest.raises(click.ClickException) as failure: + _await_evaluation_job( + {"job_id": "job-1", "status": "failed", "error": "backend refused the partition"} + ) + + assert "job-1" in str(failure.value) + assert "backend refused the partition" in str(failure.value) + + def test_harbor_finalize_cli_writes_only_rewards(tmp_path, monkeypatch): token_file = write_admin_token(tmp_path / "token", "admin-secret") output = tmp_path / "logs/reward.json" From d29c8728d96504d5f575779fa06268df883a4ce2 Mon Sep 17 00:00:00 2001 From: Shehab Yasser Date: Sat, 1 Aug 2026 11:46:24 +0300 Subject: [PATCH 4/6] fix: stop every benchmark from switching the new tool-call cap off Caught while pulling the dead run's harbor config to pick a repro cell. Its agent env carried BASH_DEFAULT_TIMEOUT_MS=39600000 and BASH_MAX_TIMEOUT_MS= 39600000, eleven hours, straight out of swe-atlas-qna's build.yaml. Every benchmark sets the same pair, and the previous commit places vero's cap ahead of the build's agent_env so the build wins. So the fix landed as a no-op on every claude-code cell in the suite. The opencode path was unaffected, since those names are claude-code's and opencode ignores them, which is the only reason the cell that died is a valid test of it. The raises were themselves a fix for the mirror-image failure. Claude Code's default 10-minute Bash cap truncated an evaluation, the optimizer moved to --detach plus background-poll and ended its turn, and a headless --print run is never re-woken, so the search died there (officeqa run #2). Raising the cap above a full validation pass bought that back and bought the silence instead: one call blocking for hours is one stream idling for hours. Both failures are handled below the config now. `evals run` returns inside its own bound carrying a job_id, so no evaluation needs a long call and the optimizer is never pushed into detach-and-end-turn; vero caps the tool call itself. So the pair is removed from all seven build files (six benchmarks plus the atlas gpt54mini variant), each keeping the history in the comment that replaces it, since deleting a setting without its reason invites its return. test_no_benchmark_re_raises_the_tool_call_bound pins it, derived from _TOOL_TIMEOUT_ENVIRONMENT rather than a literal list so a harness added there is covered without touching the test. It asserts on the loaded config's agent_env, which is where the override would actually take effect. CONFIGURATION.md's per-benchmark row becomes a uniform 300s and says the number now comes from vero rather than the build, with the footnote and the agent_env bullet rewritten to record why the old sizing existed and why it went. Same for terminal-bench's README row. Test plan: 497 passed, 18 skipped (493/16 before; the new test contributes four passes and two skips, officeqa and browsecomp-plus skipping on unvendored tasks). Credentials must be exported. Co-Authored-By: Claude Opus 5 (1M context) --- harness-engineering-bench/CONFIGURATION.md | 42 ++++++++++++------- .../browsecomp-plus/baseline/build.yaml | 28 ++++++++----- .../gaia/baseline/build.yaml | 28 ++++++++----- .../officeqa/baseline/build.yaml | 25 +++++++---- .../baseline/build.gpt54mini.yaml | 28 ++++++++----- .../swe-atlas-qna/baseline/build.yaml | 28 ++++++++----- .../tau3/baseline/build.yaml | 28 ++++++++----- .../terminal-bench/README.md | 2 +- .../terminal-bench/baseline/build.yaml | 18 +++++--- vero/tests/test_v05_benchmark_configs.py | 28 +++++++++++++ 10 files changed, 169 insertions(+), 86 deletions(-) diff --git a/harness-engineering-bench/CONFIGURATION.md b/harness-engineering-bench/CONFIGURATION.md index 8d77ecfe..70f46008 100644 --- a/harness-engineering-bench/CONFIGURATION.md +++ b/harness-engineering-bench/CONFIGURATION.md @@ -184,7 +184,7 @@ benchmark can be checked against the others at a glance. | declared `[verifier] timeout_sec` | 300 | 300 | 900 | 300 | 300 | 360–12000 ✦ | n/a (registry dataset) | | declared `build_timeout_sec` | 300 | 600 | 600 | 600 | 7200 | 600 | n/a (registry dataset) | | verifier_timeout_seconds ‖ | 14400 | 54000 | 176400 | 158400 | 75600 | 64800 | 28800 | -| BASH_MAX_TIMEOUT_MS (tool) ¤ | 3600 s | 10800 s | 39600 s | 32400 s | 14400 s | 28800 s | n/a | +| optimizer tool-call cap ¤ | 300 s | 300 s | 300 s | 300 s | 300 s | 300 s | 300 s | | harness_user | harness | harness | null ‡ | null ‡ | null ‡ | harness | harness | | task_services_use_upstream | false | false | true (rubric judge) | true (user-sim + grader) | true (answer judge) | false (own tests) | false | | task-specific extras | — | `--no-force-build` (prebuilt corpus image) | `keepalive` --ek (ENTRYPOINT images) | `TAU2_*` model pins | pinned 2.2 GB BM25 index | per-task declared timeouts ✦ | registry dataset; `expose_case_resources: false`; sampled variant ◈ | @@ -305,14 +305,25 @@ build's own comment): 3.3–5.1× the worst measured cost of 1.33M/case-run for for `max_concurrency: 24` and for `n_attempts: 3` on the held-out target, so raising a benchmark to a 3× finalize needs no timeout change. - **Case budgets** are 4× the partition size, i.e. four full passes. -- **Optimizer `agent_env`** (now on all five): inner evals take 15–30 min, but - Claude Code caps a single Bash call at `BASH_MAX_TIMEOUT_MS` (default - 600000=10min), which forces the agent into `--detach` + background-poll + - end-turn — and in headless `--print` mode, ending the turn ends the run. Set - `BASH_MAX_TIMEOUT_MS`/`BASH_DEFAULT_TIMEOUT_MS` above a worst-case full - validation eval so the agent can block on one in a single call. +- **Optimizer `agent_env`** (now on all five): inner evals take 15–30 min, and no + benchmark sets a Bash timeout any more. Both ways of sizing one were wrong. The + 10-minute default forces the agent into `--detach` + background-poll + + end-turn, and in headless `--print` mode ending the turn ends the run; raising + it above a full validation eval, which is what these configs did until + 2026-08-01, makes one tool call sit silent for hours, and harbor reads the + optimizer through a single long-lived stdout stream that gets reaped when it + idles. Run #2 died the first way and a swe-atlas-qna cell died the second, at + 71 minutes, 9m57s into one wait. Both are now handled below the config: `evals + run` returns inside its own bound with a `job_id` to wait on again, so no + evaluation needs a long call, and vero caps the tool call itself + (`HARNESS_TOOL_TIMEOUT_SECONDS`, `vero/harbor/cli.py`). **Do not set + `BASH_MAX_TIMEOUT_MS`/`BASH_DEFAULT_TIMEOUT_MS` (or opencode's + `OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS`) in a build**: a build's + `agent_env` is forwarded after vero's, harbor keeps the last value for a key, + and the run would look configured while the cap did nothing. + `test_no_benchmark_re_raises_the_tool_call_bound` enforces this. `ENABLE_BACKGROUND_TASKS`/`FORCE_AUTO_BACKGROUND_TASKS=0` are defence in depth - only — **they gate *automatic* backgrounding and do not remove the Bash tool's + only: **they gate *automatic* backgrounding and do not remove the Bash tool's `run_in_background` parameter**, which the model can still choose, and run #2 did. Only the optimizer instruction actually forbids it. - **`infrastructure_max_attempts: 3`** applies only to trusted finalization @@ -455,13 +466,14 @@ newly clipping ~1% of cases whose p99 sat at 608); that is the benchmark's intent, and suggestively the gaia agent's own `MAX_TURNS` cap lands right at ~608 s, i.e. it was written against the declared 600. -¤ Bash tool cap for the optimizer, set above that benchmark's widest single -blocking eval — a full validation pass, `ceil(val_cases / 24) × case_timeout` -worst case. `BASH_DEFAULT_TIMEOUT_MS` is set equal to it so an eval invoked -without an explicit `timeout` still blocks rather than being truncated: a -truncated eval is what pushed run #2's optimizer into backgrounding and ended the -run. The agent additionally wraps its own calls (`timeout 1750`, `timeout 3000` -observed), so this is a ceiling, not the expected duration. +¤ Cap on a single optimizer tool call, uniform across benchmarks because it is +set by vero, not by the build (`HARNESS_TOOL_TIMEOUT_SECONDS`, +`vero/harbor/cli.py`). It is a bound on *silence*, not on evaluation length: an +evaluation runs in the sidecar for as long as it needs, while `evals run` +returns inside a 240 s bound with a `job_id` to wait on again. Until 2026-08-01 +each benchmark instead raised `BASH_MAX_TIMEOUT_MS`/`BASH_DEFAULT_TIMEOUT_MS` +above its widest single blocking eval so one evaluation fit in one call; see the +`agent_env` bullet above for why that was replaced. ¶ `evaluation` and `finalization` each get this cap independently — they are separate scopes with separate tokens and separate ledgers, so the numbers do not diff --git a/harness-engineering-bench/browsecomp-plus/baseline/build.yaml b/harness-engineering-bench/browsecomp-plus/baseline/build.yaml index e419c6b3..c0c4bd36 100644 --- a/harness-engineering-bench/browsecomp-plus/baseline/build.yaml +++ b/harness-engineering-bench/browsecomp-plus/baseline/build.yaml @@ -143,18 +143,24 @@ instruct_exhaust_budget: true task_services_use_upstream: true harness_user: null # Optimizer-agent env (forwarded to the harbor claude-code agent as --ae KEY=VALUE). -# Claude Code's Bash tool caps a single call at BASH_MAX_TIMEOUT_MS (default -# 600000=10min), well under one inner eval, which pushed the officeqa optimizer -# into --detach + background-poll + end-turn -- and a headless --print run is -# never re-woken, so the search died there. Raise the cap so a whole eval fits in -# one blocking call. The background-task vars are defence in depth only: they gate -# *automatic* backgrounding and do NOT remove the Bash tool's run_in_background -# parameter, which the model can still choose. The instruction forbids that. +# This block used to raise BASH_MAX_TIMEOUT_MS/BASH_DEFAULT_TIMEOUT_MS to hours so +# a whole eval fit in one blocking Bash call: the 10-minute default had pushed the +# officeqa optimizer into --detach + background-poll + end-turn, and a headless +# --print run is never re-woken, so the search died there. A real trade, and it +# bought the opposite failure. Harbor reads the optimizer through one long-lived +# stdout stream, and a harness flushes a command's output only when the command +# returns, so an hours-long call is an hours-long silence and the idle stream gets +# reaped: a swe-atlas-qna cell died that way at 71 minutes on 2026-07-31, 9m57s +# into one wait, discarding a candidate already scored 0.1224. +# Neither cap belongs here now. `evals run` returns inside its own bound carrying a +# job_id to wait on again, so the optimizer is never forced to detach and end its +# turn, and vero sets the tool-call cap itself (HARNESS_TOOL_TIMEOUT_SECONDS in +# vero/harbor/cli.py). Setting either variable here would silently switch that off: +# harbor keeps the last value for a key and this block is applied after vero's. +# The background-task vars are defence in depth only: they gate *automatic* +# backgrounding and do NOT remove the Bash tool's run_in_background parameter, +# which the model can still choose. The instruction forbids that. agent_env: - # Above this benchmark's widest single eval: a full validation pass is - # ceil(66/24) x 3600 = 10800s worst case. - BASH_MAX_TIMEOUT_MS: "14400000" - BASH_DEFAULT_TIMEOUT_MS: "14400000" # same as max: an un-timed eval must still block ENABLE_BACKGROUND_TASKS: "0" FORCE_AUTO_BACKGROUND_TASKS: "0" # Harnesses installed with `uv tool install` (mini-swe-agent, swe-agent) diff --git a/harness-engineering-bench/gaia/baseline/build.yaml b/harness-engineering-bench/gaia/baseline/build.yaml index 551aab25..6a3dffce 100644 --- a/harness-engineering-bench/gaia/baseline/build.yaml +++ b/harness-engineering-bench/gaia/baseline/build.yaml @@ -112,18 +112,24 @@ secrets: # candidate harness runs as an unprivileged uid, unable to read held-out state. harness_user: harness # Optimizer-agent env (forwarded to the harbor claude-code agent as --ae KEY=VALUE). -# Claude Code's Bash tool caps a single call at BASH_MAX_TIMEOUT_MS (default -# 600000=10min), well under one inner eval, which pushed the officeqa optimizer -# into --detach + background-poll + end-turn -- and a headless --print run is -# never re-woken, so the search died there. Raise the cap so a whole eval fits in -# one blocking call. The background-task vars are defence in depth only: they gate -# *automatic* backgrounding and do NOT remove the Bash tool's run_in_background -# parameter, which the model can still choose. The instruction forbids that. +# This block used to raise BASH_MAX_TIMEOUT_MS/BASH_DEFAULT_TIMEOUT_MS to hours so +# a whole eval fit in one blocking Bash call: the 10-minute default had pushed the +# officeqa optimizer into --detach + background-poll + end-turn, and a headless +# --print run is never re-woken, so the search died there. A real trade, and it +# bought the opposite failure. Harbor reads the optimizer through one long-lived +# stdout stream, and a harness flushes a command's output only when the command +# returns, so an hours-long call is an hours-long silence and the idle stream gets +# reaped: a swe-atlas-qna cell died that way at 71 minutes on 2026-07-31, 9m57s +# into one wait, discarding a candidate already scored 0.1224. +# Neither cap belongs here now. `evals run` returns inside its own bound carrying a +# job_id to wait on again, so the optimizer is never forced to detach and end its +# turn, and vero sets the tool-call cap itself (HARNESS_TOOL_TIMEOUT_SECONDS in +# vero/harbor/cli.py). Setting either variable here would silently switch that off: +# harbor keeps the last value for a key and this block is applied after vero's. +# The background-task vars are defence in depth only: they gate *automatic* +# backgrounding and do NOT remove the Bash tool's run_in_background parameter, +# which the model can still choose. The instruction forbids that. agent_env: - # Above this benchmark's widest single eval: a full validation pass is - # ceil(66/24) x 600 = 1800s worst case. - BASH_MAX_TIMEOUT_MS: "3600000" - BASH_DEFAULT_TIMEOUT_MS: "3600000" # same as max: an un-timed eval must still block ENABLE_BACKGROUND_TASKS: "0" FORCE_AUTO_BACKGROUND_TASKS: "0" # Harnesses installed with `uv tool install` (mini-swe-agent, swe-agent) diff --git a/harness-engineering-bench/officeqa/baseline/build.yaml b/harness-engineering-bench/officeqa/baseline/build.yaml index 6702a552..30b5302c 100644 --- a/harness-engineering-bench/officeqa/baseline/build.yaml +++ b/harness-engineering-bench/officeqa/baseline/build.yaml @@ -117,19 +117,26 @@ secrets: harness_user: harness # Optimizer-agent env (forwarded to the harbor claude-code agent as --ae KEY=VALUE). -# Inner evals take 15-30 min; Claude Code's Bash tool caps a single call at -# BASH_MAX_TIMEOUT_MS (default 600000=10min), which forced the agent into -# --detach + background-poll + end-turn (it then ended its turn waiting for a -# notification that never re-wakes a headless --print run). Raise the cap so the -# agent can block on a whole eval in one Bash call. +# Inner evals take 15-30 min, and this block used to raise BASH_MAX_TIMEOUT_MS / +# BASH_DEFAULT_TIMEOUT_MS to hours so one of them fit in a single blocking Bash +# call: the 10-minute default forced the agent into --detach + background-poll + +# end-turn, and a headless --print run is never re-woken, so the search died +# there. A real trade, and it bought the opposite failure. Harbor reads the +# optimizer through one long-lived stdout stream, and a harness flushes a +# command's output only when the command returns, so an hours-long call is an +# hours-long silence and the idle stream gets reaped: a swe-atlas-qna cell died +# that way at 71 minutes on 2026-07-31, 9m57s into one wait, discarding a +# candidate already scored 0.1224. +# Neither cap belongs here now. `evals run` returns inside its own bound carrying +# a job_id to wait on again, so the optimizer is never forced to detach and end +# its turn, and vero sets the tool-call cap itself (HARNESS_TOOL_TIMEOUT_SECONDS +# in vero/harbor/cli.py). Setting either variable here would silently switch that +# off: harbor keeps the last value for a key and this block is applied after +# vero's. # The background-task vars are kept for defence in depth but do NOT actually # disable the Bash tool's run_in_background parameter -- the model can still # choose it, and run #2 did. The instruction is what forbids it. agent_env: - # Above this benchmark's widest single eval: a full validation pass is - # ceil(98/24) x 1800 = 9000s worst case. - BASH_MAX_TIMEOUT_MS: "10800000" - BASH_DEFAULT_TIMEOUT_MS: "10800000" # same as max: an un-timed eval must still block ENABLE_BACKGROUND_TASKS: "0" # gates auto-backgrounding only (see above) FORCE_AUTO_BACKGROUND_TASKS: "0" # Harnesses installed with `uv tool install` (mini-swe-agent, swe-agent) diff --git a/harness-engineering-bench/swe-atlas-qna/baseline/build.gpt54mini.yaml b/harness-engineering-bench/swe-atlas-qna/baseline/build.gpt54mini.yaml index 6052e9ce..d04bdadc 100644 --- a/harness-engineering-bench/swe-atlas-qna/baseline/build.gpt54mini.yaml +++ b/harness-engineering-bench/swe-atlas-qna/baseline/build.gpt54mini.yaml @@ -202,18 +202,24 @@ task_services_use_upstream: true # audit of every candidate version. Proper fix: per-role egress isolation. harness_user: null # Optimizer-agent env (forwarded to the harbor claude-code agent as --ae KEY=VALUE). -# Claude Code's Bash tool caps a single call at BASH_MAX_TIMEOUT_MS (default -# 600000=10min), well under one inner eval, which pushed the officeqa optimizer -# into --detach + background-poll + end-turn -- and a headless --print run is -# never re-woken, so the search died there. Raise the cap so a whole eval fits in -# one blocking call. The background-task vars are defence in depth only: they gate -# *automatic* backgrounding and do NOT remove the Bash tool's run_in_background -# parameter, which the model can still choose. The instruction forbids that. +# This block used to raise BASH_MAX_TIMEOUT_MS/BASH_DEFAULT_TIMEOUT_MS to hours so +# a whole eval fit in one blocking Bash call: the 10-minute default had pushed the +# officeqa optimizer into --detach + background-poll + end-turn, and a headless +# --print run is never re-woken, so the search died there. A real trade, and it +# bought the opposite failure. Harbor reads the optimizer through one long-lived +# stdout stream, and a harness flushes a command's output only when the command +# returns, so an hours-long call is an hours-long silence and the idle stream gets +# reaped: a swe-atlas-qna cell died that way at 71 minutes on 2026-07-31, 9m57s +# into one wait, discarding a candidate already scored 0.1224. +# Neither cap belongs here now. `evals run` returns inside its own bound carrying a +# job_id to wait on again, so the optimizer is never forced to detach and end its +# turn, and vero sets the tool-call cap itself (HARNESS_TOOL_TIMEOUT_SECONDS in +# vero/harbor/cli.py). Setting either variable here would silently switch that off: +# harbor keeps the last value for a key and this block is applied after vero's. +# The background-task vars are defence in depth only: they gate *automatic* +# backgrounding and do NOT remove the Bash tool's run_in_background parameter, +# which the model can still choose. The instruction forbids that. agent_env: - # Above this benchmark's widest single eval: a full validation pass is - # ceil(49/24) x 10800 = 32400s worst case. - BASH_MAX_TIMEOUT_MS: "39600000" - BASH_DEFAULT_TIMEOUT_MS: "39600000" # same as max: an un-timed eval must still block ENABLE_BACKGROUND_TASKS: "0" FORCE_AUTO_BACKGROUND_TASKS: "0" # Harnesses installed with `uv tool install` (mini-swe-agent, swe-agent) diff --git a/harness-engineering-bench/swe-atlas-qna/baseline/build.yaml b/harness-engineering-bench/swe-atlas-qna/baseline/build.yaml index 52e66d45..c4acd7d4 100644 --- a/harness-engineering-bench/swe-atlas-qna/baseline/build.yaml +++ b/harness-engineering-bench/swe-atlas-qna/baseline/build.yaml @@ -191,18 +191,24 @@ task_services_use_upstream: true # audit of every candidate version. Proper fix: per-role egress isolation. harness_user: null # Optimizer-agent env (forwarded to the harbor claude-code agent as --ae KEY=VALUE). -# Claude Code's Bash tool caps a single call at BASH_MAX_TIMEOUT_MS (default -# 600000=10min), well under one inner eval, which pushed the officeqa optimizer -# into --detach + background-poll + end-turn -- and a headless --print run is -# never re-woken, so the search died there. Raise the cap so a whole eval fits in -# one blocking call. The background-task vars are defence in depth only: they gate -# *automatic* backgrounding and do NOT remove the Bash tool's run_in_background -# parameter, which the model can still choose. The instruction forbids that. +# This block used to raise BASH_MAX_TIMEOUT_MS/BASH_DEFAULT_TIMEOUT_MS to hours so +# a whole eval fit in one blocking Bash call: the 10-minute default had pushed the +# officeqa optimizer into --detach + background-poll + end-turn, and a headless +# --print run is never re-woken, so the search died there. A real trade, and it +# bought the opposite failure. Harbor reads the optimizer through one long-lived +# stdout stream, and a harness flushes a command's output only when the command +# returns, so an hours-long call is an hours-long silence and the idle stream gets +# reaped: a swe-atlas-qna cell died that way at 71 minutes on 2026-07-31, 9m57s +# into one wait, discarding a candidate already scored 0.1224. +# Neither cap belongs here now. `evals run` returns inside its own bound carrying a +# job_id to wait on again, so the optimizer is never forced to detach and end its +# turn, and vero sets the tool-call cap itself (HARNESS_TOOL_TIMEOUT_SECONDS in +# vero/harbor/cli.py). Setting either variable here would silently switch that off: +# harbor keeps the last value for a key and this block is applied after vero's. +# The background-task vars are defence in depth only: they gate *automatic* +# backgrounding and do NOT remove the Bash tool's run_in_background parameter, +# which the model can still choose. The instruction forbids that. agent_env: - # Above this benchmark's widest single eval: a full validation pass is - # ceil(49/24) x 10800 = 32400s worst case. - BASH_MAX_TIMEOUT_MS: "39600000" - BASH_DEFAULT_TIMEOUT_MS: "39600000" # same as max: an un-timed eval must still block ENABLE_BACKGROUND_TASKS: "0" FORCE_AUTO_BACKGROUND_TASKS: "0" # Harnesses installed with `uv tool install` (mini-swe-agent, swe-agent) diff --git a/harness-engineering-bench/tau3/baseline/build.yaml b/harness-engineering-bench/tau3/baseline/build.yaml index 1d2a0985..2663b457 100644 --- a/harness-engineering-bench/tau3/baseline/build.yaml +++ b/harness-engineering-bench/tau3/baseline/build.yaml @@ -138,18 +138,24 @@ task_services_use_upstream: true # Re-enable once task-service credentials are delivered off the harness env. harness_user: null # Optimizer-agent env (forwarded to the harbor claude-code agent as --ae KEY=VALUE). -# Claude Code's Bash tool caps a single call at BASH_MAX_TIMEOUT_MS (default -# 600000=10min), well under one inner eval, which pushed the officeqa optimizer -# into --detach + background-poll + end-turn -- and a headless --print run is -# never re-woken, so the search died there. Raise the cap so a whole eval fits in -# one blocking call. The background-task vars are defence in depth only: they gate -# *automatic* backgrounding and do NOT remove the Bash tool's run_in_background -# parameter, which the model can still choose. The instruction forbids that. +# This block used to raise BASH_MAX_TIMEOUT_MS/BASH_DEFAULT_TIMEOUT_MS to hours so +# a whole eval fit in one blocking Bash call: the 10-minute default had pushed the +# officeqa optimizer into --detach + background-poll + end-turn, and a headless +# --print run is never re-woken, so the search died there. A real trade, and it +# bought the opposite failure. Harbor reads the optimizer through one long-lived +# stdout stream, and a harness flushes a command's output only when the command +# returns, so an hours-long call is an hours-long silence and the idle stream gets +# reaped: a swe-atlas-qna cell died that way at 71 minutes on 2026-07-31, 9m57s +# into one wait, discarding a candidate already scored 0.1224. +# Neither cap belongs here now. `evals run` returns inside its own bound carrying a +# job_id to wait on again, so the optimizer is never forced to detach and end its +# turn, and vero sets the tool-call cap itself (HARNESS_TOOL_TIMEOUT_SECONDS in +# vero/harbor/cli.py). Setting either variable here would silently switch that off: +# harbor keeps the last value for a key and this block is applied after vero's. +# The background-task vars are defence in depth only: they gate *automatic* +# backgrounding and do NOT remove the Bash tool's run_in_background parameter, +# which the model can still choose. The instruction forbids that. agent_env: - # Above this benchmark's widest single eval: a full validation pass is - # ceil(150/24) x 3600 = 25200s worst case. - BASH_MAX_TIMEOUT_MS: "32400000" - BASH_DEFAULT_TIMEOUT_MS: "32400000" # same as max: an un-timed eval must still block ENABLE_BACKGROUND_TASKS: "0" FORCE_AUTO_BACKGROUND_TASKS: "0" # Harnesses installed with `uv tool install` (mini-swe-agent, swe-agent) diff --git a/harness-engineering-bench/terminal-bench/README.md b/harness-engineering-bench/terminal-bench/README.md index 49948bff..c7f51a80 100644 --- a/harness-engineering-bench/terminal-bench/README.md +++ b/harness-engineering-bench/terminal-bench/README.md @@ -123,7 +123,7 @@ Derived from the split and the declared budgets, at `max_concurrency: 24`: | widest single search eval | 24,000s | 36 validation ÷ 24 = 2 waves × slowest validation task (12,000s) | | `timeout_seconds` | 43,200 | above worst-case finalize | | `verifier_timeout_seconds` | 64,800 | finalize + a `rescore_top_k: 3` validation pass | -| optimizer `BASH_MAX_TIMEOUT_MS` | 28,800,000 | above the widest single eval, so one evaluation fits in a single blocking foreground call | +| optimizer tool-call cap | 300 s | set by vero (`HARNESS_TOOL_TIMEOUT_SECONDS`), not here; bounds how long one call may sit silent, while `evals run` returns inside it with a `job_id` to wait on again | ## The seed agent diff --git a/harness-engineering-bench/terminal-bench/baseline/build.yaml b/harness-engineering-bench/terminal-bench/baseline/build.yaml index 4cc5fa79..99b52643 100644 --- a/harness-engineering-bench/terminal-bench/baseline/build.yaml +++ b/harness-engineering-bench/terminal-bench/baseline/build.yaml @@ -125,13 +125,19 @@ secrets: - WANDB_BASE_URL harness_user: harness +# This block used to raise BASH_MAX_TIMEOUT_MS/BASH_DEFAULT_TIMEOUT_MS to 28800000 +# so the optimizer could block on one evaluation in a single foreground call, a +# headless run that backgrounds a long call never being re-woken. That is still +# true, and it bought the opposite failure: harbor reads the optimizer through one +# long-lived stdout stream, a harness flushes a command's output only when the +# command returns, and the idle stream gets reaped. A swe-atlas-qna cell died that +# way at 71 minutes on 2026-07-31, 9m57s into one wait. +# Neither cap belongs here now. `evals run` returns inside its own bound carrying a +# job_id to wait on again, so a foreground call never has to run for hours, and +# vero sets the tool-call cap itself (HARNESS_TOOL_TIMEOUT_SECONDS in +# vero/harbor/cli.py). Setting either variable here would silently switch that off: +# harbor keeps the last value for a key and this block is applied after vero's. agent_env: - # Above this benchmark's widest single search eval: a full validation pass is - # ceil(36/24) = 2 waves x 12,000s = 24,000s worst case. The optimizer must be - # able to block on one evaluation in a single foreground call -- a headless run - # that backgrounds a long call is never re-woken and the search dies there. - BASH_MAX_TIMEOUT_MS: "28800000" - BASH_DEFAULT_TIMEOUT_MS: "28800000" ENABLE_BACKGROUND_TASKS: "0" FORCE_AUTO_BACKGROUND_TASKS: "0" UV_TOOL_BIN_DIR: "/home/agent/.local/bin" diff --git a/vero/tests/test_v05_benchmark_configs.py b/vero/tests/test_v05_benchmark_configs.py index 55e6dbaa..5e9f68b8 100644 --- a/vero/tests/test_v05_benchmark_configs.py +++ b/vero/tests/test_v05_benchmark_configs.py @@ -123,6 +123,34 @@ def test_upstream_rerouting_benchmarks_keep_their_agent_on_the_gateway(benchmark ) +@pytest.mark.parametrize("benchmark", BENCHMARKS) +def test_no_benchmark_re_raises_the_tool_call_bound(benchmark): + """A build's agent_env is applied after vero's, so these keys switch it off. + + Every one of these configs used to raise BASH_MAX_TIMEOUT_MS and + BASH_DEFAULT_TIMEOUT_MS to hours, so one evaluation fit in a single blocking + Bash call. That was a fix for a real failure (a headless run that detaches + and ends its turn is never re-woken) and it bought the opposite one: the + hours-long call is an hours-long silence on harbor's stdout stream, the idle + stream gets reaped, and the trial dies without a retry. + + Setting them here again would be invisible: harbor keeps the last value for + a key and the build's agent_env is forwarded after vero's cap, so the run + would look configured while the cap did nothing. Both failures are now + handled below the config, by `evals run` returning inside its own bound and + by HARNESS_TOOL_TIMEOUT_SECONDS, so a benchmark has no reason to touch them. + """ + from vero.harbor.cli import _TOOL_TIMEOUT_ENVIRONMENT + + governed = {name for names in _TOOL_TIMEOUT_ENVIRONMENT.values() for name in names} + declared = governed.intersection(_config(benchmark).agent_env) + + assert not declared, ( + f"{benchmark} sets {sorted(declared)} in agent_env, which overrides vero's " + f"tool-call cap; remove it, or raise HARNESS_TOOL_TIMEOUT_SECONDS instead" + ) + + @pytest.mark.parametrize("benchmark", BENCHMARKS) def test_target_model_is_the_only_model_the_evaluation_scope_allows(benchmark): """The measurement substrate is fixed: one target model, allow-listed. From 96b0a53cf72c39db55235ffe801dbee6c253864e Mon Sep 17 00:00:00 2001 From: Shehab Yasser Date: Sat, 1 Aug 2026 12:14:10 +0300 Subject: [PATCH 5/6] fix: fail `evals wait` on a dead job, like the run it resumes Greptile's re-review, and correct. `_await_evaluation_job` raises on `failed` and `cancelled`, but `wait_command` printed the record and exited 0. The asymmetry existed before, harmlessly: `evals wait` was reachable only behind an explicit `--detach`. Making every long `evals run` hand back a job_id put it on the common path, so the most likely place to observe a failed evaluation became the one place that reported success. Raise the same ClickException with the sidecar's own reason. Bound expiry is untouched and still exits 0 with the still-running record: that is the case the caller is meant to re-enter, and a non-zero exit there would read as a broken evaluation. The instruction and SKILL.md gain one clause each, so the optimizer is told the exit code carries the answer rather than being left to infer terminality from a status field. Test plan: 498 passed, 18 skipped (497/18 before). Co-Authored-By: Claude Opus 5 (1M context) --- vero/src/vero/evals_cli.py | 15 +++++++++---- .../harbor/build/templates/instruction.md.j2 | 3 ++- vero/src/vero/skills/evals/SKILL.md | 3 ++- vero/tests/test_v05_evals_cli.py | 22 +++++++++++++++++++ 4 files changed, 37 insertions(+), 6 deletions(-) diff --git a/vero/src/vero/evals_cli.py b/vero/src/vero/evals_cli.py index 4503f53a..0894b9c8 100644 --- a/vero/src/vero/evals_cli.py +++ b/vero/src/vero/evals_cli.py @@ -429,6 +429,11 @@ def wait_command(job_id, poll_interval, timeout): nothing for tens of minutes can get the whole run killed (see WAIT_TIMEOUT_SECONDS), and re-entering costs one line of output, so the bound is the default rather than something to opt into. + + A job that ended in `failed` or `cancelled` exits non-zero naming the reason, + the same as `evals run`. This is the resumption of that command's own wait, + so it has to report a dead evaluation the same way; printing the record and + exiting 0 would let a shell caller read a failure as a finished measurement. """ request = _sidecar_request() terminal = {"complete", "failed", "cancelled"} @@ -442,10 +447,12 @@ def wait_command(job_id, poll_interval, timeout): click.echo(json.dumps(_enrich_job(job), indent=2)) return time.sleep(poll_interval) - if status == "complete": - click.echo(json.dumps(request("GET", f"/eval/jobs/{job_id}/result"), indent=2)) - else: - click.echo(json.dumps(_enrich_job(job), indent=2)) + if status != "complete": + raise click.ClickException( + f"evaluation job {job_id} {status}: " + f"{job.get('error') or 'no reason recorded'}" + ) + click.echo(json.dumps(request("GET", f"/eval/jobs/{job_id}/result"), indent=2)) @evals.command("list") diff --git a/vero/src/vero/harbor/build/templates/instruction.md.j2 b/vero/src/vero/harbor/build/templates/instruction.md.j2 index 89cb9267..e4c2b914 100644 --- a/vero/src/vero/harbor/build/templates/instruction.md.j2 +++ b/vero/src/vero/harbor/build/templates/instruction.md.j2 @@ -52,7 +52,8 @@ hidden final evaluation. The trusted evaluation sidecar owns the cases, scoring, on the same bound, and `evals status JOB_ID` reports its state and elapsed time without waiting at all. A job is finished when its status is `complete`, `failed` or `cancelled`; any other status means it is still running, so wait - on it again. + on it again. `evals wait` prints the result if the job completed and fails + with the reason if it did not, so you never have to read a status to tell. **Run every `evals` call in the foreground.** You are a single-shot headless run: nothing exists to deliver a notification or wake you later. If you put a diff --git a/vero/src/vero/skills/evals/SKILL.md b/vero/src/vero/skills/evals/SKILL.md index b9435112..50fe1c55 100644 --- a/vero/src/vero/skills/evals/SKILL.md +++ b/vero/src/vero/skills/evals/SKILL.md @@ -53,7 +53,8 @@ Use `--detach` **only** to run several evaluations concurrently: it returns a the same bound and prints its result; or poll `evals status JOB_ID`, which also reports `elapsed_seconds` (and `requested_cases` for a subset) so you can see it is progressing. A job is finished when its status is `complete`, `failed` or -`cancelled`. To wait on two jobs, wait the first, then the second. +`cancelled`; `evals wait` prints the result if it completed and fails with the +reason if it did not. To wait on two jobs, wait the first, then the second. Run every `evals` call in the **foreground**. If you are a headless single-shot run, nothing can wake you: putting a long call in a background task, scheduling diff --git a/vero/tests/test_v05_evals_cli.py b/vero/tests/test_v05_evals_cli.py index 090844e2..34d0ec4a 100644 --- a/vero/tests/test_v05_evals_cli.py +++ b/vero/tests/test_v05_evals_cli.py @@ -362,3 +362,25 @@ def test_wait_timeout_returns_still_running_and_enriched(monkeypatch): payload = json.loads(result.output) assert payload["status"] == "running" assert payload["requested_cases"] == 3 + + +def test_wait_on_a_dead_job_exits_non_zero_like_the_run_it_resumes(monkeypatch): + """`evals run` raises on a failed job, so its resumption has to as well. + + Now that a plain `evals run` returns a job_id whenever an evaluation + outlives its bound, `evals wait` is where most failures are actually + observed. Printing `status: failed` and exiting 0 there would let a shell + caller, and an optimizer reading only the exit code, treat a dead + evaluation as a finished measurement. + """ + import vero.harbor.cli as harbor_cli + + def fake_request(method, path, **kw): + assert not path.endswith("/result"), "a dead job has no result to fetch" + return {"job_id": "j", "status": "failed", "error": "backend refused the partition"} + + monkeypatch.setattr(harbor_cli, "_request", fake_request) + result = CliRunner().invoke(evals, ["wait", "j"]) + + assert result.exit_code != 0 + assert "backend refused the partition" in result.output From 28b4a068f3e50bb90a2f94cf29e37bd8f8e2df30 Mon Sep 17 00:00:00 2001 From: Shehab Yasser Date: Sat, 1 Aug 2026 13:44:36 +0300 Subject: [PATCH 6/6] fix: retry an outer trial that loses its connection to the sandbox The tool-call cap this branch already ships does not prevent StreamTerminatedError, and this branch claimed it did. Two runs on 2026-08-01 each sat through repeated 242s silences (the bounded `evals run` returning on schedule, exactly as designed) and then died during shorter ones, at 104s and 188s. No idle-reap threshold is both above 242 and below 104, so silence was never the trigger. What is: harbor reads the whole agent phase through one Modal stdio stream, and Modal's budget for silently reconnecting it is per *stream*, not per drop. `stream_stdio_max_retries` is 10 for the life of the stream and is never replenished; only the backoff delay resets on a successful chunk (modal/_utils/task_command_router_client.py:764). The backoff sums to about 10s, so a long run exhausts the budget either by accumulating drops or in one outage, and the next drop is fatal whenever it lands. Neither limit is reachable from vero: they are constructor keywords on a client vero does not build. Harbor's own retry is reachable, and already classes StreamTerminatedError as retryable (it is absent from RetryConfig.exclude_exceptions) -- but max_retries defaults to 0, so both runs recorded `Trials 0 | Exceptions 1` instead of trying again. Pass `--max-retries 1`, narrowed with `--retry-include` to StreamTerminatedError and ConnectionError. Harbor's default retries every exception not explicitly excluded, and a retry restarts the optimizer from zero, so an unrestricted one would spend a full second run re-deriving a deterministic crash. Emitted ahead of the build's flags and the caller's, so a later --max-retries wins: the same ordering rule that made the tool-call cap a no-op on every benchmark setting BASH_MAX_TIMEOUT_MS. Asserted, not just intended. This is mitigation, not a cure. If drops accumulate with stream age then a second attempt re-rolls the same dice, at the cost of another full optimizer run. Confirming which of the two exhaustion paths is at work needs MODAL_LOGLEVEL=DEBUG, which logs each reconnect. The tool-call cap stays. It is a bound on how long a run can report nothing, which is worth having on its own, and it is what lets `evals run` hand back a job_id rather than be killed mid-evaluation. Its comments, tests and docs now say that instead of claiming a cure they do not deliver. Test plan: 518 tests collected (517 before), 500 passed, 18 skipped. Co-Authored-By: Claude Opus 5 (1M context) --- harness-engineering-bench/CONFIGURATION.md | 34 +++++++--- vero/docs/harbor-architecture.md | 18 ++++++ vero/src/vero/harbor/cli.py | 73 +++++++++++++++++++--- vero/tests/test_v05_cli.py | 57 ++++++++++++++--- 4 files changed, 158 insertions(+), 24 deletions(-) diff --git a/harness-engineering-bench/CONFIGURATION.md b/harness-engineering-bench/CONFIGURATION.md index 70f46008..1f12ed2a 100644 --- a/harness-engineering-bench/CONFIGURATION.md +++ b/harness-engineering-bench/CONFIGURATION.md @@ -185,6 +185,7 @@ benchmark can be checked against the others at a glance. | declared `build_timeout_sec` | 300 | 600 | 600 | 600 | 7200 | 600 | n/a (registry dataset) | | verifier_timeout_seconds ‖ | 14400 | 54000 | 176400 | 158400 | 75600 | 64800 | 28800 | | optimizer tool-call cap ¤ | 300 s | 300 s | 300 s | 300 s | 300 s | 300 s | 300 s | +| outer-trial retries ¤ | 1 | 1 | 1 | 1 | 1 | 1 | 1 | | harness_user | harness | harness | null ‡ | null ‡ | null ‡ | harness | harness | | task_services_use_upstream | false | false | true (rubric judge) | true (user-sim + grader) | true (answer judge) | false (own tests) | false | | task-specific extras | — | `--no-force-build` (prebuilt corpus image) | `keepalive` --ek (ENTRYPOINT images) | `TAU2_*` model pins | pinned 2.2 GB BM25 index | per-task declared timeouts ✦ | registry dataset; `expose_case_resources: false`; sampled variant ◈ | @@ -308,15 +309,23 @@ build's own comment): 3.3–5.1× the worst measured cost of 1.33M/case-run for - **Optimizer `agent_env`** (now on all five): inner evals take 15–30 min, and no benchmark sets a Bash timeout any more. Both ways of sizing one were wrong. The 10-minute default forces the agent into `--detach` + background-poll + - end-turn, and in headless `--print` mode ending the turn ends the run; raising - it above a full validation eval, which is what these configs did until - 2026-08-01, makes one tool call sit silent for hours, and harbor reads the - optimizer through a single long-lived stdout stream that gets reaped when it - idles. Run #2 died the first way and a swe-atlas-qna cell died the second, at - 71 minutes, 9m57s into one wait. Both are now handled below the config: `evals - run` returns inside its own bound with a `job_id` to wait on again, so no - evaluation needs a long call, and vero caps the tool call itself - (`HARNESS_TOOL_TIMEOUT_SECONDS`, `vero/harbor/cli.py`). **Do not set + end-turn, and in headless `--print` mode ending the turn ends the run; run #2 + died that way. Raising it above a full validation eval, which is what these + configs did until 2026-08-01, is not a fix but a different problem: one tool + call then sits silent for hours, and harbor reads the optimizer through a + single long-lived stdout stream, so nothing distinguishes a working optimizer + from a wedged one for that whole window. Both are now handled below the + config: `evals run` returns inside its own bound with a `job_id` to wait on + again, so no evaluation needs a long call, and vero caps the tool call itself + (`HARNESS_TOOL_TIMEOUT_SECONDS`, `vero/harbor/cli.py`). + + The cap does **not** keep that stream alive, and an earlier version of this + note claimed it did. Modal's budget for silently reconnecting a dropped stream + is per stream and never replenished (10 for its whole life), so a long run + exhausts it and the next drop kills the trial regardless of how quiet the + stream was; two runs on 2026-08-01 survived repeated 242 s silences and then + died during 104 s and 188 s ones. What covers that is `--max-retries 1`, also + set by vero (`HARNESS_TRIAL_RETRIES`). **Do not set `BASH_MAX_TIMEOUT_MS`/`BASH_DEFAULT_TIMEOUT_MS` (or opencode's `OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS`) in a build**: a build's `agent_env` is forwarded after vero's, harbor keeps the last value for a key, @@ -475,6 +484,13 @@ each benchmark instead raised `BASH_MAX_TIMEOUT_MS`/`BASH_DEFAULT_TIMEOUT_MS` above its widest single blocking eval so one evaluation fit in one call; see the `agent_env` bullet above for why that was replaced. +The retry row is vero-set for the same reason (`HARNESS_TRIAL_RETRIES`) and is +narrowed to `StreamTerminatedError`/`ConnectionError`: a lost Modal stdio stream +costs a trial rather than the run, while a deterministic crash still fails once. +It is an *outer*-trial retry, unrelated to `n_attempts` (§) on the held-out +target. Each attempt restarts the optimizer from zero, so raising it multiplies +wall clock and tokens. + ¶ `evaluation` and `finalization` each get this cap independently — they are separate scopes with separate tokens and separate ledgers, so the numbers do not share a pool. `max_requests` is 200 000 on both everywhere (a full officeqa diff --git a/vero/docs/harbor-architecture.md b/vero/docs/harbor-architecture.md index 0bb3ea69..0387b4df 100644 --- a/vero/docs/harbor-architecture.md +++ b/vero/docs/harbor-architecture.md @@ -84,6 +84,24 @@ propagates past `await self._run_verifier()`. Two outer trials died on `StreamTerminatedError` at 71 minutes left nothing, discarding a candidate that had already scored 0.1224 on 49 validation cases. +That last one is transport, not the optimizer. Harbor reads the whole agent +phase through a single Modal stdio stream, and Modal's budget for silently +reconnecting that stream is per *stream* rather than per drop: +`stream_stdio_max_retries` is 10 for its entire life and is never replenished, +with roughly 10s of backoff in total. A long optimizer run exhausts it either by +accumulating drops or in one outage, and the next drop ends the trial. Harbor +already classes `StreamTerminatedError` as retryable but defaults `max_retries` +to 0, so vero passes `--max-retries 1` with `--retry-include` narrowed to +`StreamTerminatedError` and `ConnectionError` (`HARNESS_TRIAL_RETRIES`, +`vero/harbor/cli.py`). A retry restarts the optimizer from zero, which is why +it is one attempt and why it is scoped to transport failures. + +Timing bounds do not help with this, and it is worth recording why, because the +first fix assumed they would. If an idle stream were being reaped, a shorter +maximum silence would prevent the reap. Two runs on 2026-08-01 falsified that: +both sat through repeated 242s silences and then died during shorter ones, at +104s and 188s. No single idle threshold is both above 242 and below 104. + What still runs on that path is **artifact collection**: Harbor calls `_collect_artifacts` from `Trial._recover_outputs` too, and in the failed run it succeeded from the same sandbox moments after the stream died. So the compiled diff --git a/vero/src/vero/harbor/cli.py b/vero/src/vero/harbor/cli.py index 66502b73..20a2b4cb 100644 --- a/vero/src/vero/harbor/cli.py +++ b/vero/src/vero/harbor/cli.py @@ -240,13 +240,17 @@ def _load_env_file(path: Path) -> dict[str, str]: # The ceiling vero puts on a single optimizer tool call, in seconds. # # The optimizer's stdout reaches harbor as one long-lived stream, and an agent -# harness only flushes output when a tool call *returns*. "How long may one tool -# call run" is therefore the same question as "how long may that stream go -# silent", and an idle stream gets reaped by the network path while the machine, -# the connection and the sandbox all stay healthy. The outer trial is not -# retried, so the whole optimization goes with it: on 2026-07-31 a cell died at -# 71 minutes, 9m57s into one silent call, discarding a candidate that had -# already scored 0.1224 on 49 validation cases. +# harness only flushes output when a tool call *returns*, so "how long may one +# tool call run" is the same question as "how long may that stream go silent". +# A cell that goes quiet for tens of minutes reports nothing while it is quiet: +# no progress, no partial trajectory, and no way to tell a working optimizer +# from a wedged one. This bounds that blind window. +# +# It is *not* what keeps the stream alive. Two runs on 2026-08-01 each sat +# through repeated 242s silences and then died anyway during shorter ones (104s +# and 188s), which rules out an idle-reap threshold: no single threshold is both +# above 242 and below 104. See HARNESS_TRIAL_RETRIES for what actually drops the +# stream and what survives it. # # Configured on the harness rather than requested in the instruction, because # the instruction is advisory and this is not. Telling the optimizer to wait in @@ -256,6 +260,41 @@ def _load_env_file(path: Path) -> dict[str, str]: # optimizer runs. HARNESS_TOOL_TIMEOUT_SECONDS = 300 +# How many times harbor may restart a failed outer trial, and on what. +# +# Modal reconnects a dropped stdio stream transparently, but the budget for +# those reconnects is per *stream*, not per drop: `stream_stdio_max_retries` is +# 10, is decremented for the life of the stream, and is never replenished (only +# the backoff delay resets on a successful chunk, see +# modal/_utils/task_command_router_client.py). Harbor reads the whole agent +# phase through one stream, so an hour-long optimizer run gets 10 reconnects +# total and the 11th drop is fatal whenever it lands. The backoff sums to about +# 10s, so one outage longer than that also exhausts the budget in a single +# burst. Neither is reachable from vero: the limits are constructor keywords on +# a client vero does not build. +# +# What *is* reachable is harbor's own retry. It already treats +# StreamTerminatedError as retryable (the exception is absent from +# RetryConfig.exclude_exceptions), but max_retries defaults to 0, so the two +# 2026-08-01 runs recorded `Trials 0 | Exceptions 1` rather than trying again. +# +# One retry, not more: a retry discards the trial directory and restarts the +# optimizer from zero, so each attempt costs a full run's wall clock and tokens. +# This is mitigation, not a cure -- if the drops accumulate with stream age then +# a second attempt re-rolls the same dice -- but it converts the common case of +# a transient blip from "no result" into "a result, late". +HARNESS_TRIAL_RETRIES = 1 + +# Restricted to transport failures a fresh attempt could plausibly fix. Harbor's +# default is to retry *every* exception not explicitly excluded, which would +# spend a second full optimizer run on a deterministic crash. ConnectionError is +# included because Modal raises it from the same reconnect path when the give-up +# is a timeout or a socket error rather than a terminated stream. +_RETRYABLE_TRIAL_EXCEPTIONS: tuple[str, ...] = ( + "ConnectionError", + "StreamTerminatedError", +) + # How each harness spells "bound one tool call". Only knobs verified in the # harness's own source or docs are listed: a harness missing here keeps its own # default rather than being sent a variable it silently ignores. @@ -439,6 +478,23 @@ def _agent_tool_timeout_args(agent: str) -> list[str]: return arguments +def _trial_retry_args() -> list[str]: + """Let harbor restart an outer trial that lost its connection to the sandbox. + + `--retry-include` *replaces* harbor's include set rather than adding to it, + so naming these two is what narrows retries to transport failures. A build or + caller passing its own `--retry-include` later on the command line widens the + set (typer collects repeats), which is the intended direction: a benchmark + may know of another exception worth a second attempt, and none of them should + have to re-state these. + """ + + arguments = ["--max-retries", str(HARNESS_TRIAL_RETRIES)] + for name in _RETRYABLE_TRIAL_EXCEPTIONS: + arguments.extend(["--retry-include", name]) + return arguments + + def _outer_app_name_args( environment: str, config_name: str, extra: tuple[str, ...] ) -> list[str]: @@ -884,6 +940,9 @@ def run_command(config_path, agent, model, environment, params, env_file, extra) command.extend(_opencode_gateway_args(agent, model, task)) command.extend(_litellm_base_url_args(agent, task)) command.extend(_kimi_gateway_args(agent, task)) + # Ahead of both, so a build or a caller can raise, lower or disable the + # retry: a later `--max-retries` wins on click's last-value rule. + command.extend(_trial_retry_args()) # Build-declared outer-trial flags first, so a command-line arg can still # override them (harbor's `--ek` takes the last value for a key). command.extend(config.optimizer_harbor_args) diff --git a/vero/tests/test_v05_cli.py b/vero/tests/test_v05_cli.py index f7191350..998095f5 100644 --- a/vero/tests/test_v05_cli.py +++ b/vero/tests/test_v05_cli.py @@ -417,17 +417,18 @@ def test_opencode_gets_a_step_limit_that_does_not_truncate_the_search(tmp_path): def test_the_harness_is_configured_to_bound_one_tool_call(): - """A silent tool call is what kills a run, so the bound goes on the harness. + """How long a run may report nothing is a harness setting, not a request. The optimizer's stdout reaches harbor as one long-lived stream and a harness flushes a command's output only when that command returns, so the length of - one tool call *is* the length of the stream's silence. On 2026-07-31 a cell - died at 71 minutes, 9m57s into a single silent call, discarding a candidate - already scored at 0.1224 on 49 validation cases. + one tool call *is* the length of the stream's silence, and a cell that goes + quiet for tens of minutes cannot be told from a wedged one. - The first fix for this asked the optimizer, in the instruction, to wait in - bounded steps. A prompt enforces nothing: the recipe it shipped had to be - corrected twice in review, and a model reconstructing the loop from memory + This does not keep the stream alive; see the trial-retry test for what does. + + The first fix asked the optimizer, in the instruction, to wait in bounded + steps. A prompt enforces nothing: the recipe it shipped had to be corrected + twice in review, and a model reconstructing the loop from memory reintroduces the failure. Both harnesses expose the bound as a setting. """ from vero.evals_cli import WAIT_TIMEOUT_SECONDS @@ -471,6 +472,10 @@ def test_a_build_can_override_the_tool_call_bound_it_is_given(tmp_path, monkeypa Harbor keeps the last `--ae` value for a key, so ordering decides who wins. A build that legitimately needs a longer single call has to be able to say so through `agent_env`, the same way it declares any other agent variable. + The same ordering rule has to hold for every vero default on this command + line, so the trial retry is checked here too: emitting it after the build's + own flags is exactly what made the first version of the tool-call cap a + no-op on every benchmark that set BASH_MAX_TIMEOUT_MS. """ from vero.harbor import build as harbor_build from vero.harbor import cli as harbor_cli @@ -481,7 +486,7 @@ def test_a_build_can_override_the_tool_call_bound_it_is_given(tmp_path, monkeypa class _Config: harbor_requirement = "harbor[modal]==0.20.0" agent_env = {"OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS": "900000"} - optimizer_harbor_args: list[str] = [] + optimizer_harbor_args = ["--max-retries", "3"] extra_harbor_args: list[str] = [] name = "vero/stub-benchmark" @@ -522,6 +527,42 @@ def _record(command, env=None): override = "OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS=900000" assert command.index(default) < command.index(override) + # Same rule for the trial retry: vero's value first, the build's last, so + # click's last-value-wins leaves the build with 3 attempts rather than ours. + assert command.count("--max-retries") == 2 + assert command.index("3") > command.index(str(harbor_cli.HARNESS_TRIAL_RETRIES)) + + +def test_a_lost_connection_costs_a_trial_rather_than_the_whole_run(): + """A dropped stdio stream is transport noise; harbor must be told to retry. + + Modal reconnects a dropped stream on its own, but the budget is per stream + and never replenished: `stream_stdio_max_retries` is 10 for the life of the + stream, with about 10s of backoff in total. Harbor reads the whole agent + phase through one stream, so a long optimizer run exhausts that budget + either by accumulating drops or in one outage, and the next drop is fatal. + Two runs on 2026-08-01 died that way and recorded `Trials 0 | Exceptions 1`, + because harbor's own max_retries defaults to 0. + + Timing bounds do not help here. Both runs sat through repeated 242s + silences and then died during shorter ones (104s and 188s), which is what + ruled out the idle-reap explanation this fix originally shipped with. + """ + from vero.harbor.cli import HARNESS_TRIAL_RETRIES, _trial_retry_args + + arguments = _trial_retry_args() + assert arguments[:2] == ["--max-retries", str(HARNESS_TRIAL_RETRIES)] + + # Restricted to transport failures. Harbor's default retries *every* + # exception not explicitly excluded, so an unrestricted retry would spend a + # second full optimizer run re-deriving a deterministic crash. + assert arguments[2::2] == ["--retry-include", "--retry-include"] + assert arguments[3::2] == ["ConnectionError", "StreamTerminatedError"] + + # One attempt, not a loop: a retry restarts the optimizer from zero, so each + # one costs a full run's wall clock and tokens. + assert HARNESS_TRIAL_RETRIES == 1 + def test_litellm_harnesses_get_the_gateway_url_under_the_name_they_read(tmp_path): """litellm reads _API_BASE; the provider SDKs read _BASE_URL.