Resolve Harbor ${VAR} env templates from the runtime environment

[ayushnangia]

Harbor interop: resolve ${VAR} / ${VAR:-default} env templates

Problem

Harbor's task schema supports environment variable templates in every env
map — env = { OPENAI_API_KEY = "${OPENAI_API_KEY}" } — resolved from the
host environment when a trial starts (harbor/utils/env.py:resolve_env_vars,
applied at sandbox creation and again just before the verifier runs). This is
how real Harbor datasets inject LLM-judge credentials: among the adapters
shipped in harbor 0.20.0, hle, clbench, tau3-bench, theagentcompany,
strongreject, seal0, ineqmath, crmarena, scienceagentbench and
others all template their verifier or environment env this way.

The HUD adapter passes these values through verbatim: adapt.py copies the
raw dicts into the baked manifest, and env.py applies them with
os.environ.update(...). An agent or verifier then sees the literal string
${OPENAI_API_KEY}, and an LLM-judge verifier fails auth (typically scoring
0.0) instead of grading — silently, per task, with no signal that the task
config asked for a host credential.

What resolution has to respect

Two HUD invariants rule out the obvious fixes:

• Adapt-time resolution would bake secrets. The manifest is part of the
  content-hashed image payload, and task rows persist to tasks.json.
  Harbor has templatize_sensitive_env for exactly this concern — templates
  are the safe form to persist. (A regression test now pins this: host
  values must never appear in the adapt output tree.)
• Compose-file interpolation would break the platform path. Generated
  compose.json files must stay hermetic (ComposeConfig.from_file rejects
  $ interpolation for remote adaptation), so the templates can't be pushed
  into the Compose document for docker compose to resolve.

The channel that already exists for launch-time host values is the runtime
provider: ModalRuntime(env_vars=...) stages values into the Compose
override, which lives only in a temp directory for the duration of up.

Change

1. DockerRuntime gains env_vars with the same meaning as
   ModalRuntime's existing parameter: applied to the Compose override for
   Compose environments (ComposeProject.stage already supported this) and
   as --env arguments for image environments. Without this there is no way
   to hand a local Compose run a host value at all (run_args is rejected
   for Compose).

2. env.py ports Harbor's resolution semantics and applies them to the
   environment, agent, and verifier env maps at startup, before
   anything consumes them. Matching resolve_env_vars: only values that are
   exactly ${VAR} or ${VAR:-default} resolve (embedded templates stay
   literal); ${VAR:-} yields an empty string; a required variable with no
   default and no value fails loudly with the variable's name. The source is
   the control process's environment — populated by the runtime's
   env_vars — so resolution is equivalent to Harbor's host-side pass
   without any secret entering a persisted artifact.

One deliberate divergence: Harbor resolves the verifier env just before the
test runs, so a missing judge credential surfaces only after the agent has
done all its work. Here all three maps resolve at environment startup, so a
missing variable aborts before a rollout spends anything.

Usage matches Harbor's harbor run on the same dataset:

taskset = harbor.adapt(dataset)
await taskset.run(
    agent,
    runtime=DockerRuntime(env_vars={"OPENAI_API_KEY": os.environ["OPENAI_API_KEY"]}),
)

Tests

• test_docker_runtime_passes_env_vars_to_docker_run /
  test_docker_runtime_stages_env_vars_into_the_compose_override: the new
  parameter reaches both Docker paths (scripted docker CLI, no daemon).
• test_env_templates_are_persisted_verbatim_not_resolved: adapt output
  never contains a host value that a template references, only the template.
• test_env_templates_resolve_from_runtime_env_vars (integration): a task
  whose task.toml templates [environment.env] and [verifier.env] runs
  end-to-end; the verifier asserts the resolved value, the default, the
  empty default, and that an embedded Bearer ${...} stays literal; the
  agent-side workspace sees the resolved values too. Fails on current main
  (verifier sees literal templates, reward 0.0).
• test_missing_env_template_aborts_startup (integration): a required
  template with no value aborts the adapted artifact's main service with an
  error naming the variable. The test runs the generated Compose service in
  the foreground because providers yield as soon as the published port
  exists — readiness is client-owned — so an early abort is only observable
  from the artifact itself.

uv run pytest -q, -m integration (Docker), ruff format/check, and
ty check all pass.

---

Note

Medium Risk
Changes how credentials reach Harbor verifiers at runtime; behavior is well-tested but misconfigured env_vars could still break grading or leak expectations about which host vars are required.

Overview
Harbor tasks often declare env values like ${OPENAI_API_KEY} that must be filled from the host at trial start. Before this change, those strings stayed literal in the adapted image, so LLM-judge verifiers could fail auth with no clear signal.

DockerRuntime now accepts env_vars, aligned with ModalRuntime: values go into the Compose override for Compose runs and as docker run --env for image-only runs, so host secrets never need to be baked into content-hashed manifests.

In harbor/env.py, startup applies Harbor-style resolution for environment, agent, and verifier env maps: only values that are exactly ${VAR} or ${VAR:-default} resolve from the control process env (populated by the provider); embedded templates stay literal; missing required variables raise an error naming the variable. Adapt output still stores templates verbatim—host values are not persisted in the image tree.

Tests cover Docker wiring, adapt-time non-leakage of secrets, end-to-end template resolution via DockerRuntime(env_vars=...), and startup failure when a required template has no value.

^{Reviewed by Cursor Bugbot for commit 285aa24. Bugbot is set up for automated code reviews on this repo. Configure here.}