Summary
sandbox.python.env_packages (and the Node equivalent, sandbox.node.env_packages) are only ever installed by the host-side Sandbox.provision() path (_install_packages / _install_node_packages in src/coder_eval/sandbox.py), which backs driver: tempdir.
src/coder_eval/isolation/docker_runner.py has no equivalent step at all — no pip/uv pip install, no npm install, nothing. So any task with driver: docker plus sandbox.python.env_packages (or sandbox.node.env_packages) silently never gets those packages installed inside the container. Neither the docs (docs/TASK_DEFINITION_GUIDE.md) nor the field description on PythonEnvConfig/NodeEnvConfig (src/coder_eval/models/sandbox.py) mention this as a driver: docker limitation, so this reads as a real functionality gap rather than a documented restriction.
How it surfaced
The Harbor E2E template_sources scenario (tests/harbor_e2e/fixtures/template_sources.yaml, driver: docker + sandbox.python.env_packages: [pytest]) failed in CI (#187) with:
Command: python -m pytest tests/
Exit code: 1 (expected: 0)
Stderr:
/usr/local/bin/python: No module named pytest
The agent's own recorded trajectory shows it also tried python -m pytest tests/test_main.py -v and then fell back to manually verifying its implementation with a bare python -c check — consistent with pytest never having been installed in its own sandbox either.
Suggested fix
Teach DockerRunner to provision sandbox.python.env_packages / sandbox.node.env_packages inside the container (an exec step analogous to Sandbox._install_packages/_install_node_packages) before the agent starts, or explicitly document the gap if it's intentionally out of scope for driver: docker today.
Scope note
Confirmed via code inspection that this predates #186 (c9b96a80) — it is not a regression from that commit, just the first thing to exercise this path in CI. Filed separately per triage of https://github.com/UiPath/coder_eval/actions/runs/35242305732 rather than folding into PR #187.
Summary
sandbox.python.env_packages(and the Node equivalent,sandbox.node.env_packages) are only ever installed by the host-sideSandbox.provision()path (_install_packages/_install_node_packagesinsrc/coder_eval/sandbox.py), which backsdriver: tempdir.src/coder_eval/isolation/docker_runner.pyhas no equivalent step at all — nopip/uv pip install, nonpm install, nothing. So any task withdriver: dockerplussandbox.python.env_packages(orsandbox.node.env_packages) silently never gets those packages installed inside the container. Neither the docs (docs/TASK_DEFINITION_GUIDE.md) nor the field description onPythonEnvConfig/NodeEnvConfig(src/coder_eval/models/sandbox.py) mention this as adriver: dockerlimitation, so this reads as a real functionality gap rather than a documented restriction.How it surfaced
The Harbor E2E
template_sourcesscenario (tests/harbor_e2e/fixtures/template_sources.yaml,driver: docker+sandbox.python.env_packages: [pytest]) failed in CI (#187) with:The agent's own recorded trajectory shows it also tried
python -m pytest tests/test_main.py -vand then fell back to manually verifying its implementation with a barepython -ccheck — consistent with pytest never having been installed in its own sandbox either.Suggested fix
Teach
DockerRunnerto provisionsandbox.python.env_packages/sandbox.node.env_packagesinside the container (anexecstep analogous toSandbox._install_packages/_install_node_packages) before the agent starts, or explicitly document the gap if it's intentionally out of scope fordriver: dockertoday.Scope note
Confirmed via code inspection that this predates #186 (
c9b96a80) — it is not a regression from that commit, just the first thing to exercise this path in CI. Filed separately per triage of https://github.com/UiPath/coder_eval/actions/runs/35242305732 rather than folding into PR #187.