Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions harness-opt-bench/CONFIGURATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,8 @@ repeat their meaning.
- `harbor_requirement` names the Harbor package the evaluation service installs.
It carries the extra for the evaluation environment and can be overridden with
`--param harbor_requirement=`.
- `vero_requirement` names the published VeRO release the compiled task installs.
It must equal the version doing the compiling.
- `secrets` lists the environment variable names the run needs for its execution
and telemetry services. `vero harbor run` refuses to launch while any is
unset. Edit the list and the env file together.
Expand Down
1 change: 1 addition & 0 deletions harness-opt-bench/browsecomp-plus/baseline/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ task_source: ../tasks
task_manifest: ../partitions/manifest.json
agent_import_path: browsecomp_plus_agent.agent:BrowseCompPlusAgent
harbor_requirement: ${harbor_requirement:-harbor[modal]==0.20.0}
vero_requirement: scaleapi-vero==0.6.0

partition_files:
development: ../partitions/development.json
Expand Down
1 change: 1 addition & 0 deletions harness-opt-bench/gaia/baseline/build.shell.e2e.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ task_source: gaia/gaia@sha256:bbc356f476e0b70ba77da11a9be7d6345918d1e4a2daade0d6
task_manifest: ../partitions/manifest.json
agent_import_path: gaia_agent.agent:GaiaAgent
harbor_requirement: ${harbor_requirement:-harbor[modal]==0.20.0}
vero_requirement: scaleapi-vero==0.6.0

# E2E SMOKE VARIANT of build.shell.yaml. Same seed, same target, same gateway
# scopes; eight cases in total, tiny budgets, and a 25-minute agent clock so the
Expand Down
1 change: 1 addition & 0 deletions harness-opt-bench/gaia/baseline/build.shell.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ task_source: gaia/gaia@sha256:bbc356f476e0b70ba77da11a9be7d6345918d1e4a2daade0d6
task_manifest: ../partitions/manifest.json
agent_import_path: gaia_agent.agent:GaiaAgent
harbor_requirement: ${harbor_requirement:-harbor[modal]==0.20.0}
vero_requirement: scaleapi-vero==0.6.0

partition_files:
development: ../partitions/development.json
Expand Down
1 change: 1 addition & 0 deletions harness-opt-bench/gaia/baseline/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ task_source: gaia/gaia@sha256:bbc356f476e0b70ba77da11a9be7d6345918d1e4a2daade0d6
task_manifest: ../partitions/manifest.json
agent_import_path: gaia_agent.agent:GaiaAgent
harbor_requirement: ${harbor_requirement:-harbor[modal]==0.20.0}
vero_requirement: scaleapi-vero==0.6.0

partition_files:
development: ../partitions/development.json
Expand Down
1 change: 1 addition & 0 deletions harness-opt-bench/officeqa/baseline/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ task_source: ../tasks
task_manifest: ../partitions/manifest.json
agent_import_path: officeqa_agent.agent:OfficeQaAgent
harbor_requirement: ${harbor_requirement:-harbor[modal]==0.20.0}
vero_requirement: scaleapi-vero==0.6.0

partition_files:
development: ../partitions/development.json
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ task_source: terminal-bench/terminal-bench-2-1@sha256:7d7bdc1cbedad549fc1140404b
task_manifest: ../partitions/manifest.json
agent_import_path: terminal_bench_agent.agent:TerminalBenchAgent
harbor_requirement: ${harbor_requirement:-harbor[modal]==0.20.0}
vero_requirement: scaleapi-vero==0.6.0

partition_files:
development: ../partitions/development.json
Expand Down
1 change: 1 addition & 0 deletions harness-opt-bench/terminal-bench/baseline/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ task_source: terminal-bench/terminal-bench-2-1@sha256:7d7bdc1cbedad549fc1140404b
task_manifest: ../partitions/manifest.json
agent_import_path: terminal_bench_agent.agent:TerminalBenchAgent
harbor_requirement: ${harbor_requirement:-harbor[modal]==0.20.0}
vero_requirement: scaleapi-vero==0.6.0

partition_files:
development: ../partitions/development.json
Expand Down
Loading