diff --git a/harness-engineering-bench/CONFIGURATION.md b/harness-engineering-bench/CONFIGURATION.md index 8d77ecfe..1f12ed2a 100644 --- a/harness-engineering-bench/CONFIGURATION.md +++ b/harness-engineering-bench/CONFIGURATION.md @@ -184,7 +184,8 @@ benchmark can be checked against the others at a glance. | declared `[verifier] timeout_sec` | 300 | 300 | 900 | 300 | 300 | 360–12000 ✦ | n/a (registry dataset) | | declared `build_timeout_sec` | 300 | 600 | 600 | 600 | 7200 | 600 | n/a (registry dataset) | | verifier_timeout_seconds ‖ | 14400 | 54000 | 176400 | 158400 | 75600 | 64800 | 28800 | -| BASH_MAX_TIMEOUT_MS (tool) ¤ | 3600 s | 10800 s | 39600 s | 32400 s | 14400 s | 28800 s | n/a | +| optimizer tool-call cap ¤ | 300 s | 300 s | 300 s | 300 s | 300 s | 300 s | 300 s | +| outer-trial retries ¤ | 1 | 1 | 1 | 1 | 1 | 1 | 1 | | harness_user | harness | harness | null ‡ | null ‡ | null ‡ | harness | harness | | task_services_use_upstream | false | false | true (rubric judge) | true (user-sim + grader) | true (answer judge) | false (own tests) | false | | task-specific extras | — | `--no-force-build` (prebuilt corpus image) | `keepalive` --ek (ENTRYPOINT images) | `TAU2_*` model pins | pinned 2.2 GB BM25 index | per-task declared timeouts ✦ | registry dataset; `expose_case_resources: false`; sampled variant ◈ | @@ -305,14 +306,33 @@ build's own comment): 3.3–5.1× the worst measured cost of 1.33M/case-run for for `max_concurrency: 24` and for `n_attempts: 3` on the held-out target, so raising a benchmark to a 3× finalize needs no timeout change. - **Case budgets** are 4× the partition size, i.e. four full passes. -- **Optimizer `agent_env`** (now on all five): inner evals take 15–30 min, but - Claude Code caps a single Bash call at `BASH_MAX_TIMEOUT_MS` (default - 600000=10min), which forces the agent into `--detach` + background-poll + - end-turn — and in headless `--print` mode, ending the turn ends the run. Set - `BASH_MAX_TIMEOUT_MS`/`BASH_DEFAULT_TIMEOUT_MS` above a worst-case full - validation eval so the agent can block on one in a single call. +- **Optimizer `agent_env`** (now on all five): inner evals take 15–30 min, and no + benchmark sets a Bash timeout any more. Both ways of sizing one were wrong. The + 10-minute default forces the agent into `--detach` + background-poll + + end-turn, and in headless `--print` mode ending the turn ends the run; run #2 + died that way. Raising it above a full validation eval, which is what these + configs did until 2026-08-01, is not a fix but a different problem: one tool + call then sits silent for hours, and harbor reads the optimizer through a + single long-lived stdout stream, so nothing distinguishes a working optimizer + from a wedged one for that whole window. Both are now handled below the + config: `evals run` returns inside its own bound with a `job_id` to wait on + again, so no evaluation needs a long call, and vero caps the tool call itself + (`HARNESS_TOOL_TIMEOUT_SECONDS`, `vero/harbor/cli.py`). + + The cap does **not** keep that stream alive, and an earlier version of this + note claimed it did. Modal's budget for silently reconnecting a dropped stream + is per stream and never replenished (10 for its whole life), so a long run + exhausts it and the next drop kills the trial regardless of how quiet the + stream was; two runs on 2026-08-01 survived repeated 242 s silences and then + died during 104 s and 188 s ones. What covers that is `--max-retries 1`, also + set by vero (`HARNESS_TRIAL_RETRIES`). **Do not set + `BASH_MAX_TIMEOUT_MS`/`BASH_DEFAULT_TIMEOUT_MS` (or opencode's + `OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS`) in a build**: a build's + `agent_env` is forwarded after vero's, harbor keeps the last value for a key, + and the run would look configured while the cap did nothing. + `test_no_benchmark_re_raises_the_tool_call_bound` enforces this. `ENABLE_BACKGROUND_TASKS`/`FORCE_AUTO_BACKGROUND_TASKS=0` are defence in depth - only — **they gate *automatic* backgrounding and do not remove the Bash tool's + only: **they gate *automatic* backgrounding and do not remove the Bash tool's `run_in_background` parameter**, which the model can still choose, and run #2 did. Only the optimizer instruction actually forbids it. - **`infrastructure_max_attempts: 3`** applies only to trusted finalization @@ -455,13 +475,21 @@ newly clipping ~1% of cases whose p99 sat at 608); that is the benchmark's intent, and suggestively the gaia agent's own `MAX_TURNS` cap lands right at ~608 s, i.e. it was written against the declared 600. -¤ Bash tool cap for the optimizer, set above that benchmark's widest single -blocking eval — a full validation pass, `ceil(val_cases / 24) × case_timeout` -worst case. `BASH_DEFAULT_TIMEOUT_MS` is set equal to it so an eval invoked -without an explicit `timeout` still blocks rather than being truncated: a -truncated eval is what pushed run #2's optimizer into backgrounding and ended the -run. The agent additionally wraps its own calls (`timeout 1750`, `timeout 3000` -observed), so this is a ceiling, not the expected duration. +¤ Cap on a single optimizer tool call, uniform across benchmarks because it is +set by vero, not by the build (`HARNESS_TOOL_TIMEOUT_SECONDS`, +`vero/harbor/cli.py`). It is a bound on *silence*, not on evaluation length: an +evaluation runs in the sidecar for as long as it needs, while `evals run` +returns inside a 240 s bound with a `job_id` to wait on again. Until 2026-08-01 +each benchmark instead raised `BASH_MAX_TIMEOUT_MS`/`BASH_DEFAULT_TIMEOUT_MS` +above its widest single blocking eval so one evaluation fit in one call; see the +`agent_env` bullet above for why that was replaced. + +The retry row is vero-set for the same reason (`HARNESS_TRIAL_RETRIES`) and is +narrowed to `StreamTerminatedError`/`ConnectionError`: a lost Modal stdio stream +costs a trial rather than the run, while a deterministic crash still fails once. +It is an *outer*-trial retry, unrelated to `n_attempts` (§) on the held-out +target. Each attempt restarts the optimizer from zero, so raising it multiplies +wall clock and tokens. ¶ `evaluation` and `finalization` each get this cap independently — they are separate scopes with separate tokens and separate ledgers, so the numbers do not diff --git a/harness-engineering-bench/browsecomp-plus/baseline/build.yaml b/harness-engineering-bench/browsecomp-plus/baseline/build.yaml index e419c6b3..c0c4bd36 100644 --- a/harness-engineering-bench/browsecomp-plus/baseline/build.yaml +++ b/harness-engineering-bench/browsecomp-plus/baseline/build.yaml @@ -143,18 +143,24 @@ instruct_exhaust_budget: true task_services_use_upstream: true harness_user: null # Optimizer-agent env (forwarded to the harbor claude-code agent as --ae KEY=VALUE). -# Claude Code's Bash tool caps a single call at BASH_MAX_TIMEOUT_MS (default -# 600000=10min), well under one inner eval, which pushed the officeqa optimizer -# into --detach + background-poll + end-turn -- and a headless --print run is -# never re-woken, so the search died there. Raise the cap so a whole eval fits in -# one blocking call. The background-task vars are defence in depth only: they gate -# *automatic* backgrounding and do NOT remove the Bash tool's run_in_background -# parameter, which the model can still choose. The instruction forbids that. +# This block used to raise BASH_MAX_TIMEOUT_MS/BASH_DEFAULT_TIMEOUT_MS to hours so +# a whole eval fit in one blocking Bash call: the 10-minute default had pushed the +# officeqa optimizer into --detach + background-poll + end-turn, and a headless +# --print run is never re-woken, so the search died there. A real trade, and it +# bought the opposite failure. Harbor reads the optimizer through one long-lived +# stdout stream, and a harness flushes a command's output only when the command +# returns, so an hours-long call is an hours-long silence and the idle stream gets +# reaped: a swe-atlas-qna cell died that way at 71 minutes on 2026-07-31, 9m57s +# into one wait, discarding a candidate already scored 0.1224. +# Neither cap belongs here now. `evals run` returns inside its own bound carrying a +# job_id to wait on again, so the optimizer is never forced to detach and end its +# turn, and vero sets the tool-call cap itself (HARNESS_TOOL_TIMEOUT_SECONDS in +# vero/harbor/cli.py). Setting either variable here would silently switch that off: +# harbor keeps the last value for a key and this block is applied after vero's. +# The background-task vars are defence in depth only: they gate *automatic* +# backgrounding and do NOT remove the Bash tool's run_in_background parameter, +# which the model can still choose. The instruction forbids that. agent_env: - # Above this benchmark's widest single eval: a full validation pass is - # ceil(66/24) x 3600 = 10800s worst case. - BASH_MAX_TIMEOUT_MS: "14400000" - BASH_DEFAULT_TIMEOUT_MS: "14400000" # same as max: an un-timed eval must still block ENABLE_BACKGROUND_TASKS: "0" FORCE_AUTO_BACKGROUND_TASKS: "0" # Harnesses installed with `uv tool install` (mini-swe-agent, swe-agent) diff --git a/harness-engineering-bench/gaia/baseline/build.yaml b/harness-engineering-bench/gaia/baseline/build.yaml index 551aab25..6a3dffce 100644 --- a/harness-engineering-bench/gaia/baseline/build.yaml +++ b/harness-engineering-bench/gaia/baseline/build.yaml @@ -112,18 +112,24 @@ secrets: # candidate harness runs as an unprivileged uid, unable to read held-out state. harness_user: harness # Optimizer-agent env (forwarded to the harbor claude-code agent as --ae KEY=VALUE). -# Claude Code's Bash tool caps a single call at BASH_MAX_TIMEOUT_MS (default -# 600000=10min), well under one inner eval, which pushed the officeqa optimizer -# into --detach + background-poll + end-turn -- and a headless --print run is -# never re-woken, so the search died there. Raise the cap so a whole eval fits in -# one blocking call. The background-task vars are defence in depth only: they gate -# *automatic* backgrounding and do NOT remove the Bash tool's run_in_background -# parameter, which the model can still choose. The instruction forbids that. +# This block used to raise BASH_MAX_TIMEOUT_MS/BASH_DEFAULT_TIMEOUT_MS to hours so +# a whole eval fit in one blocking Bash call: the 10-minute default had pushed the +# officeqa optimizer into --detach + background-poll + end-turn, and a headless +# --print run is never re-woken, so the search died there. A real trade, and it +# bought the opposite failure. Harbor reads the optimizer through one long-lived +# stdout stream, and a harness flushes a command's output only when the command +# returns, so an hours-long call is an hours-long silence and the idle stream gets +# reaped: a swe-atlas-qna cell died that way at 71 minutes on 2026-07-31, 9m57s +# into one wait, discarding a candidate already scored 0.1224. +# Neither cap belongs here now. `evals run` returns inside its own bound carrying a +# job_id to wait on again, so the optimizer is never forced to detach and end its +# turn, and vero sets the tool-call cap itself (HARNESS_TOOL_TIMEOUT_SECONDS in +# vero/harbor/cli.py). Setting either variable here would silently switch that off: +# harbor keeps the last value for a key and this block is applied after vero's. +# The background-task vars are defence in depth only: they gate *automatic* +# backgrounding and do NOT remove the Bash tool's run_in_background parameter, +# which the model can still choose. The instruction forbids that. agent_env: - # Above this benchmark's widest single eval: a full validation pass is - # ceil(66/24) x 600 = 1800s worst case. - BASH_MAX_TIMEOUT_MS: "3600000" - BASH_DEFAULT_TIMEOUT_MS: "3600000" # same as max: an un-timed eval must still block ENABLE_BACKGROUND_TASKS: "0" FORCE_AUTO_BACKGROUND_TASKS: "0" # Harnesses installed with `uv tool install` (mini-swe-agent, swe-agent) diff --git a/harness-engineering-bench/officeqa/baseline/build.yaml b/harness-engineering-bench/officeqa/baseline/build.yaml index 6702a552..30b5302c 100644 --- a/harness-engineering-bench/officeqa/baseline/build.yaml +++ b/harness-engineering-bench/officeqa/baseline/build.yaml @@ -117,19 +117,26 @@ secrets: harness_user: harness # Optimizer-agent env (forwarded to the harbor claude-code agent as --ae KEY=VALUE). -# Inner evals take 15-30 min; Claude Code's Bash tool caps a single call at -# BASH_MAX_TIMEOUT_MS (default 600000=10min), which forced the agent into -# --detach + background-poll + end-turn (it then ended its turn waiting for a -# notification that never re-wakes a headless --print run). Raise the cap so the -# agent can block on a whole eval in one Bash call. +# Inner evals take 15-30 min, and this block used to raise BASH_MAX_TIMEOUT_MS / +# BASH_DEFAULT_TIMEOUT_MS to hours so one of them fit in a single blocking Bash +# call: the 10-minute default forced the agent into --detach + background-poll + +# end-turn, and a headless --print run is never re-woken, so the search died +# there. A real trade, and it bought the opposite failure. Harbor reads the +# optimizer through one long-lived stdout stream, and a harness flushes a +# command's output only when the command returns, so an hours-long call is an +# hours-long silence and the idle stream gets reaped: a swe-atlas-qna cell died +# that way at 71 minutes on 2026-07-31, 9m57s into one wait, discarding a +# candidate already scored 0.1224. +# Neither cap belongs here now. `evals run` returns inside its own bound carrying +# a job_id to wait on again, so the optimizer is never forced to detach and end +# its turn, and vero sets the tool-call cap itself (HARNESS_TOOL_TIMEOUT_SECONDS +# in vero/harbor/cli.py). Setting either variable here would silently switch that +# off: harbor keeps the last value for a key and this block is applied after +# vero's. # The background-task vars are kept for defence in depth but do NOT actually # disable the Bash tool's run_in_background parameter -- the model can still # choose it, and run #2 did. The instruction is what forbids it. agent_env: - # Above this benchmark's widest single eval: a full validation pass is - # ceil(98/24) x 1800 = 9000s worst case. - BASH_MAX_TIMEOUT_MS: "10800000" - BASH_DEFAULT_TIMEOUT_MS: "10800000" # same as max: an un-timed eval must still block ENABLE_BACKGROUND_TASKS: "0" # gates auto-backgrounding only (see above) FORCE_AUTO_BACKGROUND_TASKS: "0" # Harnesses installed with `uv tool install` (mini-swe-agent, swe-agent) diff --git a/harness-engineering-bench/swe-atlas-qna/baseline/build.gpt54mini.yaml b/harness-engineering-bench/swe-atlas-qna/baseline/build.gpt54mini.yaml index 6052e9ce..d04bdadc 100644 --- a/harness-engineering-bench/swe-atlas-qna/baseline/build.gpt54mini.yaml +++ b/harness-engineering-bench/swe-atlas-qna/baseline/build.gpt54mini.yaml @@ -202,18 +202,24 @@ task_services_use_upstream: true # audit of every candidate version. Proper fix: per-role egress isolation. harness_user: null # Optimizer-agent env (forwarded to the harbor claude-code agent as --ae KEY=VALUE). -# Claude Code's Bash tool caps a single call at BASH_MAX_TIMEOUT_MS (default -# 600000=10min), well under one inner eval, which pushed the officeqa optimizer -# into --detach + background-poll + end-turn -- and a headless --print run is -# never re-woken, so the search died there. Raise the cap so a whole eval fits in -# one blocking call. The background-task vars are defence in depth only: they gate -# *automatic* backgrounding and do NOT remove the Bash tool's run_in_background -# parameter, which the model can still choose. The instruction forbids that. +# This block used to raise BASH_MAX_TIMEOUT_MS/BASH_DEFAULT_TIMEOUT_MS to hours so +# a whole eval fit in one blocking Bash call: the 10-minute default had pushed the +# officeqa optimizer into --detach + background-poll + end-turn, and a headless +# --print run is never re-woken, so the search died there. A real trade, and it +# bought the opposite failure. Harbor reads the optimizer through one long-lived +# stdout stream, and a harness flushes a command's output only when the command +# returns, so an hours-long call is an hours-long silence and the idle stream gets +# reaped: a swe-atlas-qna cell died that way at 71 minutes on 2026-07-31, 9m57s +# into one wait, discarding a candidate already scored 0.1224. +# Neither cap belongs here now. `evals run` returns inside its own bound carrying a +# job_id to wait on again, so the optimizer is never forced to detach and end its +# turn, and vero sets the tool-call cap itself (HARNESS_TOOL_TIMEOUT_SECONDS in +# vero/harbor/cli.py). Setting either variable here would silently switch that off: +# harbor keeps the last value for a key and this block is applied after vero's. +# The background-task vars are defence in depth only: they gate *automatic* +# backgrounding and do NOT remove the Bash tool's run_in_background parameter, +# which the model can still choose. The instruction forbids that. agent_env: - # Above this benchmark's widest single eval: a full validation pass is - # ceil(49/24) x 10800 = 32400s worst case. - BASH_MAX_TIMEOUT_MS: "39600000" - BASH_DEFAULT_TIMEOUT_MS: "39600000" # same as max: an un-timed eval must still block ENABLE_BACKGROUND_TASKS: "0" FORCE_AUTO_BACKGROUND_TASKS: "0" # Harnesses installed with `uv tool install` (mini-swe-agent, swe-agent) diff --git a/harness-engineering-bench/swe-atlas-qna/baseline/build.yaml b/harness-engineering-bench/swe-atlas-qna/baseline/build.yaml index 52e66d45..c4acd7d4 100644 --- a/harness-engineering-bench/swe-atlas-qna/baseline/build.yaml +++ b/harness-engineering-bench/swe-atlas-qna/baseline/build.yaml @@ -191,18 +191,24 @@ task_services_use_upstream: true # audit of every candidate version. Proper fix: per-role egress isolation. harness_user: null # Optimizer-agent env (forwarded to the harbor claude-code agent as --ae KEY=VALUE). -# Claude Code's Bash tool caps a single call at BASH_MAX_TIMEOUT_MS (default -# 600000=10min), well under one inner eval, which pushed the officeqa optimizer -# into --detach + background-poll + end-turn -- and a headless --print run is -# never re-woken, so the search died there. Raise the cap so a whole eval fits in -# one blocking call. The background-task vars are defence in depth only: they gate -# *automatic* backgrounding and do NOT remove the Bash tool's run_in_background -# parameter, which the model can still choose. The instruction forbids that. +# This block used to raise BASH_MAX_TIMEOUT_MS/BASH_DEFAULT_TIMEOUT_MS to hours so +# a whole eval fit in one blocking Bash call: the 10-minute default had pushed the +# officeqa optimizer into --detach + background-poll + end-turn, and a headless +# --print run is never re-woken, so the search died there. A real trade, and it +# bought the opposite failure. Harbor reads the optimizer through one long-lived +# stdout stream, and a harness flushes a command's output only when the command +# returns, so an hours-long call is an hours-long silence and the idle stream gets +# reaped: a swe-atlas-qna cell died that way at 71 minutes on 2026-07-31, 9m57s +# into one wait, discarding a candidate already scored 0.1224. +# Neither cap belongs here now. `evals run` returns inside its own bound carrying a +# job_id to wait on again, so the optimizer is never forced to detach and end its +# turn, and vero sets the tool-call cap itself (HARNESS_TOOL_TIMEOUT_SECONDS in +# vero/harbor/cli.py). Setting either variable here would silently switch that off: +# harbor keeps the last value for a key and this block is applied after vero's. +# The background-task vars are defence in depth only: they gate *automatic* +# backgrounding and do NOT remove the Bash tool's run_in_background parameter, +# which the model can still choose. The instruction forbids that. agent_env: - # Above this benchmark's widest single eval: a full validation pass is - # ceil(49/24) x 10800 = 32400s worst case. - BASH_MAX_TIMEOUT_MS: "39600000" - BASH_DEFAULT_TIMEOUT_MS: "39600000" # same as max: an un-timed eval must still block ENABLE_BACKGROUND_TASKS: "0" FORCE_AUTO_BACKGROUND_TASKS: "0" # Harnesses installed with `uv tool install` (mini-swe-agent, swe-agent) diff --git a/harness-engineering-bench/tau3/baseline/build.yaml b/harness-engineering-bench/tau3/baseline/build.yaml index 1d2a0985..2663b457 100644 --- a/harness-engineering-bench/tau3/baseline/build.yaml +++ b/harness-engineering-bench/tau3/baseline/build.yaml @@ -138,18 +138,24 @@ task_services_use_upstream: true # Re-enable once task-service credentials are delivered off the harness env. harness_user: null # Optimizer-agent env (forwarded to the harbor claude-code agent as --ae KEY=VALUE). -# Claude Code's Bash tool caps a single call at BASH_MAX_TIMEOUT_MS (default -# 600000=10min), well under one inner eval, which pushed the officeqa optimizer -# into --detach + background-poll + end-turn -- and a headless --print run is -# never re-woken, so the search died there. Raise the cap so a whole eval fits in -# one blocking call. The background-task vars are defence in depth only: they gate -# *automatic* backgrounding and do NOT remove the Bash tool's run_in_background -# parameter, which the model can still choose. The instruction forbids that. +# This block used to raise BASH_MAX_TIMEOUT_MS/BASH_DEFAULT_TIMEOUT_MS to hours so +# a whole eval fit in one blocking Bash call: the 10-minute default had pushed the +# officeqa optimizer into --detach + background-poll + end-turn, and a headless +# --print run is never re-woken, so the search died there. A real trade, and it +# bought the opposite failure. Harbor reads the optimizer through one long-lived +# stdout stream, and a harness flushes a command's output only when the command +# returns, so an hours-long call is an hours-long silence and the idle stream gets +# reaped: a swe-atlas-qna cell died that way at 71 minutes on 2026-07-31, 9m57s +# into one wait, discarding a candidate already scored 0.1224. +# Neither cap belongs here now. `evals run` returns inside its own bound carrying a +# job_id to wait on again, so the optimizer is never forced to detach and end its +# turn, and vero sets the tool-call cap itself (HARNESS_TOOL_TIMEOUT_SECONDS in +# vero/harbor/cli.py). Setting either variable here would silently switch that off: +# harbor keeps the last value for a key and this block is applied after vero's. +# The background-task vars are defence in depth only: they gate *automatic* +# backgrounding and do NOT remove the Bash tool's run_in_background parameter, +# which the model can still choose. The instruction forbids that. agent_env: - # Above this benchmark's widest single eval: a full validation pass is - # ceil(150/24) x 3600 = 25200s worst case. - BASH_MAX_TIMEOUT_MS: "32400000" - BASH_DEFAULT_TIMEOUT_MS: "32400000" # same as max: an un-timed eval must still block ENABLE_BACKGROUND_TASKS: "0" FORCE_AUTO_BACKGROUND_TASKS: "0" # Harnesses installed with `uv tool install` (mini-swe-agent, swe-agent) diff --git a/harness-engineering-bench/terminal-bench/README.md b/harness-engineering-bench/terminal-bench/README.md index 49948bff..c7f51a80 100644 --- a/harness-engineering-bench/terminal-bench/README.md +++ b/harness-engineering-bench/terminal-bench/README.md @@ -123,7 +123,7 @@ Derived from the split and the declared budgets, at `max_concurrency: 24`: | widest single search eval | 24,000s | 36 validation ÷ 24 = 2 waves × slowest validation task (12,000s) | | `timeout_seconds` | 43,200 | above worst-case finalize | | `verifier_timeout_seconds` | 64,800 | finalize + a `rescore_top_k: 3` validation pass | -| optimizer `BASH_MAX_TIMEOUT_MS` | 28,800,000 | above the widest single eval, so one evaluation fits in a single blocking foreground call | +| optimizer tool-call cap | 300 s | set by vero (`HARNESS_TOOL_TIMEOUT_SECONDS`), not here; bounds how long one call may sit silent, while `evals run` returns inside it with a `job_id` to wait on again | ## The seed agent diff --git a/harness-engineering-bench/terminal-bench/baseline/build.yaml b/harness-engineering-bench/terminal-bench/baseline/build.yaml index 4cc5fa79..99b52643 100644 --- a/harness-engineering-bench/terminal-bench/baseline/build.yaml +++ b/harness-engineering-bench/terminal-bench/baseline/build.yaml @@ -125,13 +125,19 @@ secrets: - WANDB_BASE_URL harness_user: harness +# This block used to raise BASH_MAX_TIMEOUT_MS/BASH_DEFAULT_TIMEOUT_MS to 28800000 +# so the optimizer could block on one evaluation in a single foreground call, a +# headless run that backgrounds a long call never being re-woken. That is still +# true, and it bought the opposite failure: harbor reads the optimizer through one +# long-lived stdout stream, a harness flushes a command's output only when the +# command returns, and the idle stream gets reaped. A swe-atlas-qna cell died that +# way at 71 minutes on 2026-07-31, 9m57s into one wait. +# Neither cap belongs here now. `evals run` returns inside its own bound carrying a +# job_id to wait on again, so a foreground call never has to run for hours, and +# vero sets the tool-call cap itself (HARNESS_TOOL_TIMEOUT_SECONDS in +# vero/harbor/cli.py). Setting either variable here would silently switch that off: +# harbor keeps the last value for a key and this block is applied after vero's. agent_env: - # Above this benchmark's widest single search eval: a full validation pass is - # ceil(36/24) = 2 waves x 12,000s = 24,000s worst case. The optimizer must be - # able to block on one evaluation in a single foreground call -- a headless run - # that backgrounds a long call is never re-woken and the search dies there. - BASH_MAX_TIMEOUT_MS: "28800000" - BASH_DEFAULT_TIMEOUT_MS: "28800000" ENABLE_BACKGROUND_TASKS: "0" FORCE_AUTO_BACKGROUND_TASKS: "0" UV_TOOL_BIN_DIR: "/home/agent/.local/bin" diff --git a/vero/docs/guide.md b/vero/docs/guide.md index 1cdf6969..cbe22d8f 100644 --- a/vero/docs/guide.md +++ b/vero/docs/guide.md @@ -96,11 +96,15 @@ build.yaml --output task` compiles without running, for inspection. > Harbor constructs the agent; a raw `harbor run` would let the agent adapter > read the upstream key from its own host process first. -Inside the container the agent evaluates candidates with `evals run ---detach`, then `evals status JOB` / `evals result JOB` / `evals status` (via `VERO_EVAL_URL`). -Detached evaluations are **durable jobs** — the candidate version is captured -before the command returns, so ending the agent process can't lose or race a -running measurement. +Inside the container the agent evaluates candidates with `evals run` (via +`VERO_EVAL_URL`), which waits for the result and, if the evaluation outlives +that wait's bound, hands back a `job_id` for `evals wait JOB`; `--detach` +returns the id immediately instead, for evaluating concurrently. Either way the +evaluation is a **durable job**: the candidate version is captured before the +command returns, so ending the agent process can't lose or race a running +measurement. The bound matters because vero also caps how long one of the +optimizer's tool calls may run (`HARNESS_TOOL_TIMEOUT_SECONDS`), so an `evals` +call has to return on its own terms rather than be killed mid-evaluation. ### How the boundaries hold diff --git a/vero/docs/harbor-architecture.md b/vero/docs/harbor-architecture.md index 0bb3ea69..0387b4df 100644 --- a/vero/docs/harbor-architecture.md +++ b/vero/docs/harbor-architecture.md @@ -84,6 +84,24 @@ propagates past `await self._run_verifier()`. Two outer trials died on `StreamTerminatedError` at 71 minutes left nothing, discarding a candidate that had already scored 0.1224 on 49 validation cases. +That last one is transport, not the optimizer. Harbor reads the whole agent +phase through a single Modal stdio stream, and Modal's budget for silently +reconnecting that stream is per *stream* rather than per drop: +`stream_stdio_max_retries` is 10 for its entire life and is never replenished, +with roughly 10s of backoff in total. A long optimizer run exhausts it either by +accumulating drops or in one outage, and the next drop ends the trial. Harbor +already classes `StreamTerminatedError` as retryable but defaults `max_retries` +to 0, so vero passes `--max-retries 1` with `--retry-include` narrowed to +`StreamTerminatedError` and `ConnectionError` (`HARNESS_TRIAL_RETRIES`, +`vero/harbor/cli.py`). A retry restarts the optimizer from zero, which is why +it is one attempt and why it is scoped to transport failures. + +Timing bounds do not help with this, and it is worth recording why, because the +first fix assumed they would. If an idle stream were being reaped, a shorter +maximum silence would prevent the reap. Two runs on 2026-08-01 falsified that: +both sat through repeated 242s silences and then died during shorter ones, at +104s and 188s. No single idle threshold is both above 242 and below 104. + What still runs on that path is **artifact collection**: Harbor calls `_collect_artifacts` from `Trial._recover_outputs` too, and in the failed run it succeeded from the same sandbox moments after the stream died. So the compiled diff --git a/vero/src/vero/evals_cli.py b/vero/src/vero/evals_cli.py index 006e7ca1..0894b9c8 100644 --- a/vero/src/vero/evals_cli.py +++ b/vero/src/vero/evals_cli.py @@ -26,6 +26,25 @@ CONTEXT_DIRECTORY = ".evals" _CELL_WIDTH = 48 +# How long a blocking `evals run` / `evals wait` waits before returning what it +# has so far. +# +# Not ergonomics: the optimizer's whole process is read through one long-lived +# stdout stream, and an agent harness only flushes a command's output when that +# command *returns*. A call that sits silent long enough for the network path to +# reap that stream kills the trial, and the outer trial is not retried. Observed +# 2026-07-31: a cell died at 71 minutes, 9m57s into a single silent wait, +# discarding a candidate that had already scored 0.1224 on 49 validation cases. +# Returning on a bound is what keeps the stream alive, and each return costs one +# line of output. +# +# Deliberately below HARNESS_TOOL_TIMEOUT_SECONDS (`vero/harbor/cli.py`), the cap +# vero sets on the optimizer's harness: these commands must return on their own +# terms, handing back a job id that can be waited on again, rather than be killed +# by the harness and hand the model a "retry with a larger timeout" message. +WAIT_TIMEOUT_SECONDS = 240.0 +WAIT_POLL_INTERVAL_SECONDS = 15.0 + # -------------------------------------------------------------------------- # Context discovery and shared helpers @@ -301,10 +320,11 @@ def evals() -> None: resources), `candidates/` (prior program versions), and `plan.json` (what you may evaluate, and remaining budget). - Typical loop: `evals plan` -> edit + commit -> `evals run` (blocks and - returns the result) -> `evals diff BASELINE CANDIDATE` -> - `evals cases ID --sort score` -> `evals trace ID CASE`. Add `--detach` only - to run several evaluations at once, then poll `evals status JOB`. + Typical loop: `evals plan` -> edit + commit -> `evals run` (waits for the + result, or returns a `job_id` to `evals wait` on) -> `evals diff BASELINE + CANDIDATE` -> `evals cases ID --sort score` -> `evals trace ID CASE`. Add + `--detach` only to run several evaluations at once, then poll + `evals status JOB`. """ @@ -384,41 +404,55 @@ def status_command(job_id): @click.argument("job_id") @click.option( "--poll-interval", - default=15.0, + default=WAIT_POLL_INTERVAL_SECONDS, show_default=True, type=click.FloatRange(min=1), help="Seconds between status polls.", ) @click.option( "--timeout", + default=WAIT_TIMEOUT_SECONDS, + show_default=True, type=click.FloatRange(min=0), - help="Optional max seconds to wait. On expiry, print the current " - "(still-running) status and exit 0 so you can simply call `evals wait` " - "again. Default: wait until the job finishes.", + help="Max seconds to wait. On expiry, print the current (still-running) " + "status and exit 0, so calling `evals wait` again resumes the wait. The " + "default is bounded on purpose; raising it risks the run (see the docstring).", ) def wait_command(job_id, poll_interval, timeout): - """Block until a detached job finishes, then print its result. + """Wait for a detached job, then print its result. The blocking companion to `evals run --detach`: one call that waits, so you - never hand-roll a poll loop. Idempotent — safe to call again if it returns - while the job is still running (only happens when --timeout is set). + never hand-roll a poll loop. Idempotent, so it is always safe to call again + when it returns while the job is still running. + + The wait is *bounded* by default rather than open-ended. A call that returns + nothing for tens of minutes can get the whole run killed (see + WAIT_TIMEOUT_SECONDS), and re-entering costs one line of output, so the + bound is the default rather than something to opt into. + + A job that ended in `failed` or `cancelled` exits non-zero naming the reason, + the same as `evals run`. This is the resumption of that command's own wait, + so it has to report a dead evaluation the same way; printing the record and + exiting 0 would let a shell caller read a failure as a finished measurement. """ request = _sidecar_request() terminal = {"complete", "failed", "cancelled"} - deadline = None if timeout is None else time.monotonic() + timeout + deadline = time.monotonic() + timeout while True: job = request("GET", f"/eval/jobs/{job_id}") status = job.get("status") if isinstance(job, dict) else None if status in terminal: break - if deadline is not None and time.monotonic() >= deadline: + if time.monotonic() >= deadline: click.echo(json.dumps(_enrich_job(job), indent=2)) return time.sleep(poll_interval) - if status == "complete": - click.echo(json.dumps(request("GET", f"/eval/jobs/{job_id}/result"), indent=2)) - else: - click.echo(json.dumps(_enrich_job(job), indent=2)) + if status != "complete": + raise click.ClickException( + f"evaluation job {job_id} {status}: " + f"{job.get('error') or 'no reason recorded'}" + ) + click.echo(json.dumps(request("GET", f"/eval/jobs/{job_id}/result"), indent=2)) @evals.command("list") diff --git a/vero/src/vero/harbor/build/compiler.py b/vero/src/vero/harbor/build/compiler.py index 9f2cefce..59297a08 100644 --- a/vero/src/vero/harbor/build/compiler.py +++ b/vero/src/vero/harbor/build/compiler.py @@ -15,6 +15,7 @@ from importlib.metadata import version as distribution_version from pathlib import Path, PurePosixPath +from vero.evals_cli import WAIT_TIMEOUT_SECONDS from vero.evaluation import ( EvaluationBudget, EvaluationLimits, @@ -799,6 +800,9 @@ def compile_harbor_task( ], "exhaust_budget": config.instruct_exhaust_budget, "disclose_budget": config.disclose_budget, + # Read from the CLI constant rather than restated here, so the number the + # instruction quotes cannot drift from the one `evals run` enforces. + "wait_timeout_seconds": WAIT_TIMEOUT_SECONDS, "build_timeout": config.build_timeout_seconds, "verifier_timeout": ( config.verifier_timeout_seconds or max(1, int(config.timeout_seconds)) diff --git a/vero/src/vero/harbor/build/templates/instruction.md.j2 b/vero/src/vero/harbor/build/templates/instruction.md.j2 index f6a9f678..e4c2b914 100644 --- a/vero/src/vero/harbor/build/templates/instruction.md.j2 +++ b/vero/src/vero/harbor/build/templates/instruction.md.j2 @@ -25,9 +25,11 @@ hidden final evaluation. The trusted evaluation sidecar owns the cases, scoring, --partition {{ selection_partition }} ``` - An evaluation can take many minutes — that is expected; let the call block - and read the result it returns. Iterate cheaply on a subset first - (`--start 0 --stop N`, or repeated `--case-id ID`). + An evaluation can take many minutes. `evals run` waits for it and prints the + result. If it is still running after {{ wait_timeout_seconds | int }}s the + command returns the job record instead, carrying a `job_id`; the evaluation + keeps running and `evals wait JOB_ID` resumes the wait. Iterate cheaply on a + subset first (`--start 0 --stop N`, or repeated `--case-id ID`). {% if seed_supported %} Pass `--seed N` to reproduce a noisy comparison exactly. {% else %} @@ -46,18 +48,21 @@ hidden final evaluation. The trusted evaluation sidecar owns the cases, scoring, truncated — look it up. Add `--detach` **only** to run several evaluations at once: it returns a - `job_id` immediately instead of blocking. Then `evals wait JOB_ID` blocks - until that job finishes and prints its result (or poll `evals status JOB_ID`, - which also shows elapsed time). + `job_id` immediately without waiting. `evals wait JOB_ID` then waits for one + on the same bound, and `evals status JOB_ID` reports its state and elapsed + time without waiting at all. A job is finished when its status is `complete`, + `failed` or `cancelled`; any other status means it is still running, so wait + on it again. `evals wait` prints the result if the job completed and fails + with the reason if it did not, so you never have to read a status to tell. **Run every `evals` call in the foreground.** You are a single-shot headless run: nothing exists to deliver a notification or wake you later. If you put a long call in a background task, schedule a wake-up, or say you will "report back when it finishes", the run simply ends there and whatever you have not - submitted is lost. A foreground call that blocks for half an hour is working - correctly — let it block. To wait on two jobs, `evals wait` the first, then - the second. Do not claim an improvement before its comparison baseline has - been scored on the same cases. + submitted is lost. A call that waits for minutes and hands back a + still-running job is working correctly: wait on it again. To wait on two + jobs, wait out the first, then the second. Do not claim an improvement before + its comparison baseline has been scored on the same cases. 4. Use `evals status` to inspect evaluation jobs and allowed evaluation sets{% if disclose_budget %}, and to see remaining budgets{% endif %}. diff --git a/vero/src/vero/harbor/cli.py b/vero/src/vero/harbor/cli.py index 90562c56..20a2b4cb 100644 --- a/vero/src/vero/harbor/cli.py +++ b/vero/src/vero/harbor/cli.py @@ -11,12 +11,18 @@ import subprocess import sys import tempfile +import time import urllib.error import urllib.request from pathlib import Path import click +from vero.evals_cli import ( + WAIT_POLL_INTERVAL_SECONDS, + WAIT_TIMEOUT_SECONDS, + _enrich_job, +) from vero.evaluation import ( CaseIds, CaseRange, @@ -231,6 +237,81 @@ def _load_env_file(path: Path) -> dict[str, str]: # the gateway token cap are the intended limits. OPENCODE_STEP_LIMIT = 1000 +# The ceiling vero puts on a single optimizer tool call, in seconds. +# +# The optimizer's stdout reaches harbor as one long-lived stream, and an agent +# harness only flushes output when a tool call *returns*, so "how long may one +# tool call run" is the same question as "how long may that stream go silent". +# A cell that goes quiet for tens of minutes reports nothing while it is quiet: +# no progress, no partial trajectory, and no way to tell a working optimizer +# from a wedged one. This bounds that blind window. +# +# It is *not* what keeps the stream alive. Two runs on 2026-08-01 each sat +# through repeated 242s silences and then died anyway during shorter ones (104s +# and 188s), which rules out an idle-reap threshold: no single threshold is both +# above 242 and below 104. See HARNESS_TRIAL_RETRIES for what actually drops the +# stream and what survives it. +# +# Configured on the harness rather than requested in the instruction, because +# the instruction is advisory and this is not. Telling the optimizer to wait in +# bounded steps leaves it free to ignore the advice, and free to reconstruct the +# loop wrongly. Sits above WAIT_TIMEOUT_SECONDS so the evals CLI always returns +# first on its own terms; the cap is the backstop for everything else the +# optimizer runs. +HARNESS_TOOL_TIMEOUT_SECONDS = 300 + +# How many times harbor may restart a failed outer trial, and on what. +# +# Modal reconnects a dropped stdio stream transparently, but the budget for +# those reconnects is per *stream*, not per drop: `stream_stdio_max_retries` is +# 10, is decremented for the life of the stream, and is never replenished (only +# the backoff delay resets on a successful chunk, see +# modal/_utils/task_command_router_client.py). Harbor reads the whole agent +# phase through one stream, so an hour-long optimizer run gets 10 reconnects +# total and the 11th drop is fatal whenever it lands. The backoff sums to about +# 10s, so one outage longer than that also exhausts the budget in a single +# burst. Neither is reachable from vero: the limits are constructor keywords on +# a client vero does not build. +# +# What *is* reachable is harbor's own retry. It already treats +# StreamTerminatedError as retryable (the exception is absent from +# RetryConfig.exclude_exceptions), but max_retries defaults to 0, so the two +# 2026-08-01 runs recorded `Trials 0 | Exceptions 1` rather than trying again. +# +# One retry, not more: a retry discards the trial directory and restarts the +# optimizer from zero, so each attempt costs a full run's wall clock and tokens. +# This is mitigation, not a cure -- if the drops accumulate with stream age then +# a second attempt re-rolls the same dice -- but it converts the common case of +# a transient blip from "no result" into "a result, late". +HARNESS_TRIAL_RETRIES = 1 + +# Restricted to transport failures a fresh attempt could plausibly fix. Harbor's +# default is to retry *every* exception not explicitly excluded, which would +# spend a second full optimizer run on a deterministic crash. ConnectionError is +# included because Modal raises it from the same reconnect path when the give-up +# is a timeout or a socket error rather than a terminated stream. +_RETRYABLE_TRIAL_EXCEPTIONS: tuple[str, ...] = ( + "ConnectionError", + "StreamTerminatedError", +) + +# How each harness spells "bound one tool call". Only knobs verified in the +# harness's own source or docs are listed: a harness missing here keeps its own +# default rather than being sent a variable it silently ignores. +_TOOL_TIMEOUT_ENVIRONMENT: dict[str, tuple[str, ...]] = { + # opencode reads a *default* only. `packages/opencode/src/tool/shell.ts` + # resolves `flags.bashDefaultTimeoutMs ?? 2 * 60 * 1000` and then + # `params.timeout ?? defaultTimeoutMs` with no clamp, so a model that names + # its own timeout still escapes the bound. It covers the case that actually + # killed the run -- the instruction said to let the call block, so the model + # never named one -- and it lowers the default quoted in the tool + # description the model reads. + "opencode": ("OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS",), + # claude-code takes both, and its MAX is a true ceiling: a per-call timeout + # from inside the conversation cannot raise it. + "claude-code": ("BASH_DEFAULT_TIMEOUT_MS", "BASH_MAX_TIMEOUT_MS"), +} + # Harnesses that drive the model through litellm rather than a provider SDK. # litellm reads the base URL as _API_BASE; the SDKs read # _BASE_URL. vero sets the SDK names, so a litellm-based harness sees no @@ -375,6 +456,45 @@ def _opencode_gateway_args(agent: str, model: str | None, task: Path) -> list[st return ["--ak", f"opencode_config={json.dumps(payload, separators=(',', ':'))}"] +def _agent_tool_timeout_args(agent: str) -> list[str]: + """Bound one optimizer tool call, so no single call can idle the stream. + + The mechanism the previous fix reached for was the instruction: it told the + optimizer to wait in bounded steps instead of one open-ended block. That is + the right shape and the wrong layer. A prompt cannot enforce anything, the + recipe it shipped had to be corrected twice in review, and a model that + reconstructs the loop from memory reintroduces the failure. The harnesses + already expose the bound as a setting; set it. + + Returns `--ae NAME=VALUE` pairs, which harbor merges into the scoped exec env + wrapping the agent's run phase. Values are milliseconds, the unit every + harness here uses. + """ + + milliseconds = int(HARNESS_TOOL_TIMEOUT_SECONDS * 1000) + arguments: list[str] = [] + for name in _TOOL_TIMEOUT_ENVIRONMENT.get(agent, ()): + arguments.extend(["--ae", f"{name}={milliseconds}"]) + return arguments + + +def _trial_retry_args() -> list[str]: + """Let harbor restart an outer trial that lost its connection to the sandbox. + + `--retry-include` *replaces* harbor's include set rather than adding to it, + so naming these two is what narrows retries to transport failures. A build or + caller passing its own `--retry-include` later on the command line widens the + set (typer collects repeats), which is the intended direction: a benchmark + may know of another exception worth a second attempt, and none of them should + have to re-state these. + """ + + arguments = ["--max-retries", str(HARNESS_TRIAL_RETRIES)] + for name in _RETRYABLE_TRIAL_EXCEPTIONS: + arguments.extend(["--retry-include", name]) + return arguments + + def _outer_app_name_args( environment: str, config_name: str, extra: tuple[str, ...] ) -> list[str]: @@ -806,6 +926,10 @@ def run_command(config_path, agent, model, environment, params, env_file, extra) ] if model is not None: command.extend(["-m", model]) + # Ahead of the build's own agent env, so a build can raise or drop the + # cap by naming the same variable: harbor keeps the last value for a key, + # and a vero default must not silently outrank an explicit choice. + command.extend(_agent_tool_timeout_args(agent)) # Forward the build's declared agent env to the optimizer agent's shell. # Harbor's `--ae KEY=VALUE` populates the agent's extra_env, which harbor # injects into the agent's setup/install exec (scoped_exec_env). Sorted @@ -816,6 +940,9 @@ def run_command(config_path, agent, model, environment, params, env_file, extra) command.extend(_opencode_gateway_args(agent, model, task)) command.extend(_litellm_base_url_args(agent, task)) command.extend(_kimi_gateway_args(agent, task)) + # Ahead of both, so a build or a caller can raise, lower or disable the + # retry: a later `--max-retries` wins on click's last-value rule. + command.extend(_trial_retry_args()) # Build-declared outer-trial flags first, so a command-line arg can still # override them (harbor's `--ek` takes the last value for a key). command.extend(config.optimizer_harbor_args) @@ -887,6 +1014,43 @@ def inference_gateway_command(config_path, host, port): serve_inference_gateway(config_path=config_path, host=host, port=port) +def _await_evaluation_job(job: dict, timeout: float = WAIT_TIMEOUT_SECONDS) -> dict: + """Wait out a started evaluation job, bounded, and return what to print. + + A blocking `POST /eval` is one HTTP call that can take half an hour and + prints nothing until it returns, which is precisely the silence that killed + an optimization on 2026-07-31. The sidecar drives both entry points through + the same tracked job (`Sidecar._execute_tracked_job`), so starting a job and + polling it is the same evaluation, the same budget and the same + `SidecarEvaluationResult` -- only interruptible. + + Returns the evaluation result once the job completes, or the job record when + the bound expires first, in which case the caller re-enters with + `evals wait JOB_ID` and the evaluation keeps running in the sidecar + regardless. A failed or cancelled job raises, so a bounded run still exits + non-zero carrying the sidecar's own reason, exactly as the blocking call did. + """ + + job_id = job.get("job_id") if isinstance(job, dict) else None + if not job_id: + return job + terminal = {"complete", "failed", "cancelled"} + deadline = time.monotonic() + timeout + while True: + status = job.get("status") + if status in terminal: + break + if time.monotonic() >= deadline: + return _enrich_job(job) + time.sleep(WAIT_POLL_INTERVAL_SECONDS) + job = _request("GET", f"/eval/jobs/{job_id}") + if status == "complete": + return _request("GET", f"/eval/jobs/{job_id}/result") + raise click.ClickException( + f"evaluation job {job_id} {status}: {job.get('error') or 'no reason recorded'}" + ) + + @harbor.command("eval") @click.option( "--backend", "backend_id", required=True, @@ -1018,15 +1182,12 @@ def evaluate_command( limits=EvaluationLimits(**limit_values) if limit_values else None, seed=seed, ) + payload = body.model_dump(mode="json") + if detach: + click.echo(json.dumps(_request("POST", "/eval/jobs", payload=payload), indent=2)) + return click.echo( - json.dumps( - _request( - "POST", - "/eval/jobs" if detach else "/eval", - payload=body.model_dump(mode="json"), - ), - indent=2, - ) + json.dumps(_await_evaluation_job(_request("POST", "/eval/jobs", payload=payload)), indent=2) ) diff --git a/vero/src/vero/skills/evals/SKILL.md b/vero/src/vero/skills/evals/SKILL.md index 3381c081..50fe1c55 100644 --- a/vero/src/vero/skills/evals/SKILL.md +++ b/vero/src/vero/skills/evals/SKILL.md @@ -40,20 +40,27 @@ you truncated — it is on disk. ## Blocking vs detached -By default `evals run` **blocks** until scoring finishes and returns the result -— one call, nothing to track. Evaluations can take many minutes; that is -expected, so let it block. This is what you want almost always. +By default `evals run` **waits** for scoring and returns the result: one call, +nothing to track. This is what you want almost always. Evaluations can take many +minutes, and a wait is *bounded*: if the evaluation is still running when the +bound expires, the command returns the job record (with its `job_id`) instead of +the result and exits 0. Nothing is lost when that happens, the evaluation keeps +running in the sidecar, and `evals wait JOB_ID` resumes waiting for it. Waiting +again is always safe. Use `--detach` **only** to run several evaluations concurrently: it returns a -`job_id` immediately instead of blocking. Then `evals wait JOB_ID` blocks until -that job finishes and prints its result; or poll `evals status JOB_ID`, which -now also reports `elapsed_seconds` (and `requested_cases` for a subset) so you -can see it is progressing. To wait on two jobs, wait the first, then the second. +`job_id` immediately without waiting. Then `evals wait JOB_ID` waits for one on +the same bound and prints its result; or poll `evals status JOB_ID`, which also +reports `elapsed_seconds` (and `requested_cases` for a subset) so you can see it +is progressing. A job is finished when its status is `complete`, `failed` or +`cancelled`; `evals wait` prints the result if it completed and fails with the +reason if it did not. To wait on two jobs, wait the first, then the second. Run every `evals` call in the **foreground**. If you are a headless single-shot run, nothing can wake you: putting a long call in a background task, scheduling a wake-up, or promising to "report back when it finishes" ends the run right -there. A call that blocks for half an hour is working correctly. +there. A call that waits for minutes and hands back a still-running job is +working correctly; wait on it again. ## Run options worth knowing (`evals run --help` for all) diff --git a/vero/tests/test_v05_benchmark_configs.py b/vero/tests/test_v05_benchmark_configs.py index 55e6dbaa..5e9f68b8 100644 --- a/vero/tests/test_v05_benchmark_configs.py +++ b/vero/tests/test_v05_benchmark_configs.py @@ -123,6 +123,34 @@ def test_upstream_rerouting_benchmarks_keep_their_agent_on_the_gateway(benchmark ) +@pytest.mark.parametrize("benchmark", BENCHMARKS) +def test_no_benchmark_re_raises_the_tool_call_bound(benchmark): + """A build's agent_env is applied after vero's, so these keys switch it off. + + Every one of these configs used to raise BASH_MAX_TIMEOUT_MS and + BASH_DEFAULT_TIMEOUT_MS to hours, so one evaluation fit in a single blocking + Bash call. That was a fix for a real failure (a headless run that detaches + and ends its turn is never re-woken) and it bought the opposite one: the + hours-long call is an hours-long silence on harbor's stdout stream, the idle + stream gets reaped, and the trial dies without a retry. + + Setting them here again would be invisible: harbor keeps the last value for + a key and the build's agent_env is forwarded after vero's cap, so the run + would look configured while the cap did nothing. Both failures are now + handled below the config, by `evals run` returning inside its own bound and + by HARNESS_TOOL_TIMEOUT_SECONDS, so a benchmark has no reason to touch them. + """ + from vero.harbor.cli import _TOOL_TIMEOUT_ENVIRONMENT + + governed = {name for names in _TOOL_TIMEOUT_ENVIRONMENT.values() for name in names} + declared = governed.intersection(_config(benchmark).agent_env) + + assert not declared, ( + f"{benchmark} sets {sorted(declared)} in agent_env, which overrides vero's " + f"tool-call cap; remove it, or raise HARNESS_TOOL_TIMEOUT_SECONDS instead" + ) + + @pytest.mark.parametrize("benchmark", BENCHMARKS) def test_target_model_is_the_only_model_the_evaluation_scope_allows(benchmark): """The measurement substrate is fixed: one target model, allow-listed. diff --git a/vero/tests/test_v05_cli.py b/vero/tests/test_v05_cli.py index 335a52a0..998095f5 100644 --- a/vero/tests/test_v05_cli.py +++ b/vero/tests/test_v05_cli.py @@ -416,6 +416,154 @@ def test_opencode_gets_a_step_limit_that_does_not_truncate_the_search(tmp_path): assert _opencode_gateway_args("claude-code", "claude-sonnet-5", tmp_path) == [] +def test_the_harness_is_configured_to_bound_one_tool_call(): + """How long a run may report nothing is a harness setting, not a request. + + The optimizer's stdout reaches harbor as one long-lived stream and a harness + flushes a command's output only when that command returns, so the length of + one tool call *is* the length of the stream's silence, and a cell that goes + quiet for tens of minutes cannot be told from a wedged one. + + This does not keep the stream alive; see the trial-retry test for what does. + + The first fix asked the optimizer, in the instruction, to wait in bounded + steps. A prompt enforces nothing: the recipe it shipped had to be corrected + twice in review, and a model reconstructing the loop from memory + reintroduces the failure. Both harnesses expose the bound as a setting. + """ + from vero.evals_cli import WAIT_TIMEOUT_SECONDS + from vero.harbor.cli import HARNESS_TOOL_TIMEOUT_SECONDS, _agent_tool_timeout_args + + milliseconds = str(HARNESS_TOOL_TIMEOUT_SECONDS * 1000) + + # opencode exposes a default only: packages/opencode/src/tool/shell.ts reads + # `flags.bashDefaultTimeoutMs ?? 2 * 60 * 1000`, then `params.timeout ?? + # defaultTimeoutMs` with no clamp. It covers the case that actually killed + # the run, where the instruction said to let the call block so the model + # never named a timeout of its own. + assert _agent_tool_timeout_args("opencode") == [ + "--ae", + f"OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS={milliseconds}", + ] + + # claude-code takes a true ceiling as well: a per-call timeout chosen inside + # the conversation cannot raise BASH_MAX_TIMEOUT_MS. + claude = _agent_tool_timeout_args("claude-code") + assert claude[0::2] == ["--ae", "--ae"] + assert claude[1::2] == [ + f"BASH_DEFAULT_TIMEOUT_MS={milliseconds}", + f"BASH_MAX_TIMEOUT_MS={milliseconds}", + ] + + # A harness whose knob is not verified is left at its own default rather + # than handed a variable it silently ignores. + assert _agent_tool_timeout_args("codex") == [] + assert _agent_tool_timeout_args("mini-swe-agent") == [] + + # The evals CLI has to return on its own terms first. If the harness killed + # the call instead, the model would be handed opencode's "retry with a + # larger timeout" message, take the advice, and restore the silence. + assert HARNESS_TOOL_TIMEOUT_SECONDS > WAIT_TIMEOUT_SECONDS + + +def test_a_build_can_override_the_tool_call_bound_it_is_given(tmp_path, monkeypatch): + """vero's cap is a default, not a seizure of the variable. + + Harbor keeps the last `--ae` value for a key, so ordering decides who wins. + A build that legitimately needs a longer single call has to be able to say + so through `agent_env`, the same way it declares any other agent variable. + The same ordering rule has to hold for every vero default on this command + line, so the trial retry is checked here too: emitting it after the build's + own flags is exactly what made the first version of the tool-call cap a + no-op on every benchmark that set BASH_MAX_TIMEOUT_MS. + """ + from vero.harbor import build as harbor_build + from vero.harbor import cli as harbor_cli + + config_path = tmp_path / "build.yaml" + config_path.write_text("name: org/task\n", encoding="utf-8") + + class _Config: + harbor_requirement = "harbor[modal]==0.20.0" + agent_env = {"OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS": "900000"} + optimizer_harbor_args = ["--max-retries", "3"] + extra_harbor_args: list[str] = [] + name = "vero/stub-benchmark" + + monkeypatch.setattr(harbor_build, "load_harbor_build_config", lambda *a, **k: _Config()) + monkeypatch.setattr(harbor_build, "compile_harbor_task", lambda config, output: output) + monkeypatch.setattr(harbor_cli.shutil, "which", lambda name: "/usr/bin/uvx") + monkeypatch.setattr(harbor_cli, "_compiled_run_environment", lambda task, overrides: {}) + + recorded: list[list[str]] = [] + + def _record(command, env=None): + recorded.append(command) + return subprocess.CompletedProcess(command, 0) + + monkeypatch.setattr(harbor_cli.subprocess, "run", _record) + + result = CliRunner().invoke( + main, + [ + "harbor", + "run", + "--config", + str(config_path), + "--agent", + "opencode", + "--model", + "anthropic/claude-sonnet-5", + "--yes", + ], + ) + + assert result.exit_code == 0, result.output + command = recorded[0] + default = ( + "OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS=" + f"{harbor_cli.HARNESS_TOOL_TIMEOUT_SECONDS * 1000}" + ) + override = "OPENCODE_EXPERIMENTAL_BASH_DEFAULT_TIMEOUT_MS=900000" + assert command.index(default) < command.index(override) + + # Same rule for the trial retry: vero's value first, the build's last, so + # click's last-value-wins leaves the build with 3 attempts rather than ours. + assert command.count("--max-retries") == 2 + assert command.index("3") > command.index(str(harbor_cli.HARNESS_TRIAL_RETRIES)) + + +def test_a_lost_connection_costs_a_trial_rather_than_the_whole_run(): + """A dropped stdio stream is transport noise; harbor must be told to retry. + + Modal reconnects a dropped stream on its own, but the budget is per stream + and never replenished: `stream_stdio_max_retries` is 10 for the life of the + stream, with about 10s of backoff in total. Harbor reads the whole agent + phase through one stream, so a long optimizer run exhausts that budget + either by accumulating drops or in one outage, and the next drop is fatal. + Two runs on 2026-08-01 died that way and recorded `Trials 0 | Exceptions 1`, + because harbor's own max_retries defaults to 0. + + Timing bounds do not help here. Both runs sat through repeated 242s + silences and then died during shorter ones (104s and 188s), which is what + ruled out the idle-reap explanation this fix originally shipped with. + """ + from vero.harbor.cli import HARNESS_TRIAL_RETRIES, _trial_retry_args + + arguments = _trial_retry_args() + assert arguments[:2] == ["--max-retries", str(HARNESS_TRIAL_RETRIES)] + + # Restricted to transport failures. Harbor's default retries *every* + # exception not explicitly excluded, so an unrestricted retry would spend a + # second full optimizer run re-deriving a deterministic crash. + assert arguments[2::2] == ["--retry-include", "--retry-include"] + assert arguments[3::2] == ["ConnectionError", "StreamTerminatedError"] + + # One attempt, not a loop: a retry restarts the optimizer from zero, so each + # one costs a full run's wall clock and tokens. + assert HARNESS_TRIAL_RETRIES == 1 + + def test_litellm_harnesses_get_the_gateway_url_under_the_name_they_read(tmp_path): """litellm reads _API_BASE; the provider SDKs read _BASE_URL. diff --git a/vero/tests/test_v05_evals_cli.py b/vero/tests/test_v05_evals_cli.py index 090844e2..34d0ec4a 100644 --- a/vero/tests/test_v05_evals_cli.py +++ b/vero/tests/test_v05_evals_cli.py @@ -362,3 +362,25 @@ def test_wait_timeout_returns_still_running_and_enriched(monkeypatch): payload = json.loads(result.output) assert payload["status"] == "running" assert payload["requested_cases"] == 3 + + +def test_wait_on_a_dead_job_exits_non_zero_like_the_run_it_resumes(monkeypatch): + """`evals run` raises on a failed job, so its resumption has to as well. + + Now that a plain `evals run` returns a job_id whenever an evaluation + outlives its bound, `evals wait` is where most failures are actually + observed. Printing `status: failed` and exiting 0 there would let a shell + caller, and an optimizer reading only the exit code, treat a dead + evaluation as a finished measurement. + """ + import vero.harbor.cli as harbor_cli + + def fake_request(method, path, **kw): + assert not path.endswith("/result"), "a dead job has no result to fetch" + return {"job_id": "j", "status": "failed", "error": "backend refused the partition"} + + monkeypatch.setattr(harbor_cli, "_request", fake_request) + result = CliRunner().invoke(evals, ["wait", "j"]) + + assert result.exit_code != 0 + assert "backend refused the partition" in result.output diff --git a/vero/tests/test_v05_harbor_http.py b/vero/tests/test_v05_harbor_http.py index 6c9e14dc..0bc4b3a5 100644 --- a/vero/tests/test_v05_harbor_http.py +++ b/vero/tests/test_v05_harbor_http.py @@ -530,7 +530,11 @@ def fake_request(method, path, *, payload=None, headers=None): assert result.exit_code == 0, result.output assert captured["method"] == "POST" - assert captured["path"] == "/eval" + # Even a plain (non-detached) run starts a job and waits on it, rather than + # holding one open-ended `POST /eval` that prints nothing until it returns. + # Same evaluation either way: the sidecar drives both entry points through + # `_execute_tracked_job`. + assert captured["path"] == "/eval/jobs" assert captured["payload"]["evaluation_set"]["selection"] == { "kind": "ids", "ids": ["a", "b"], @@ -571,6 +575,80 @@ def fake_request(method, path, *, payload=None, headers=None): assert requests[2][:2] == ("GET", "/eval/jobs/job-1/result") +def test_a_plain_run_hands_back_a_waitable_job_when_its_bound_expires(monkeypatch): + """The wait ends before the harness would kill it, and loses nothing. + + The optimizer's harness caps one tool call, so `evals run` has to return on + its own terms while the evaluation is still running: it prints the job + record, enriched with elapsed time so the next call can see progress, and + exits 0. `evals wait JOB_ID` resumes. Were the harness to time the call out + instead, the model would be told to retry with a larger timeout and would + walk straight back into the silence that killed the trial. + """ + from vero.harbor.cli import _await_evaluation_job + + requests = [] + + def fake_request(method, path, *, payload=None, headers=None): + requests.append((method, path)) + raise AssertionError("an expired bound must not poll again") + + monkeypatch.setattr("vero.harbor.cli._request", fake_request) + running = { + "job_id": "job-1", + "status": "running", + "created_at": "2026-07-31T00:00:00+00:00", + } + + awaited = _await_evaluation_job(running, timeout=0) + + assert awaited["job_id"] == "job-1" + assert awaited["status"] == "running" + assert awaited["elapsed_seconds"] > 0 + assert requests == [] + + +def test_a_finished_job_is_awaited_into_its_result(monkeypatch): + """A job that is already terminal costs no extra wait and no poll interval.""" + from vero.harbor.cli import _await_evaluation_job + + requests = [] + + def fake_request(method, path, *, payload=None, headers=None): + requests.append((method, path)) + return {"metrics": {"score": 0.1224}} + + monkeypatch.setattr("vero.harbor.cli._request", fake_request) + + awaited = _await_evaluation_job({"job_id": "job-1", "status": "complete"}) + + assert awaited == {"metrics": {"score": 0.1224}} + assert requests == [("GET", "/eval/jobs/job-1/result")] + + +def test_a_failed_job_still_fails_the_command_with_the_sidecar_reason(monkeypatch): + """Detaching internally must not turn a failed evaluation into a success. + + The blocking `POST /eval` path raised through to a non-zero exit; the job + path records the reason on the job record instead, so waiting has to read it + back off the record and raise. + """ + from vero.harbor.cli import _await_evaluation_job + + def fake_request(method, path, *, payload=None, headers=None): + raise AssertionError("a failed job has no result to fetch") + + monkeypatch.setattr("vero.harbor.cli._request", fake_request) + + with pytest.raises(click.ClickException) as failure: + _await_evaluation_job( + {"job_id": "job-1", "status": "failed", "error": "backend refused the partition"} + ) + + assert "job-1" in str(failure.value) + assert "backend refused the partition" in str(failure.value) + + def test_harbor_finalize_cli_writes_only_rewards(tmp_path, monkeypatch): token_file = write_admin_token(tmp_path / "token", "admin-secret") output = tmp_path / "logs/reward.json"