feat: give the host→container boundary a contract and fold aggregate into report --rebuild

[uipreliga]

Gives the host→container boundary of driver: docker a typed contract that both sides must echo, and folds coder-eval aggregate into coder-eval report --rebuild. Rebased onto a2498770 (#177, #176 and #180 merged), so it follows the prose rules those PRs put in force on src/ and tests/.

Breaking changes

• A skewed or unlabeled driver: docker image is refused before the container starts. The host reads the image's org.coder-eval.version label. If the label is missing, or differs from the installed coder-eval, the run fails. This applies to dockerfile_path images too. Set ALLOW_IMAGE_SKEW=1 (not an empty string) to run a deliberately different image; a missing label is never excused.
• An image older than the host→container contract is refused. The container must echo the contract it parsed. A result with no echo, or a different one, is moved to task.json.unhonored, and a synthetic ERROR task.json takes its place. This replaces task.json.graded and task.json.rerun.
• coder-eval aggregate is removed. Use coder-eval report <run_dir> --rebuild.
• aggregate -o/--output <dir> has no replacement. --rebuild writes in place; gather task directories into the target directory first.

Downstream: UiPath/skills#3318 sets ALLOW_IMAGE_SKEW on the one workflow that tests an ad-hoc image tag against the pinned wheel. It is safe to merge before this release.

What changes

1. ContainerContext contract. context.json is a strict Pydantic model: every field is required, unknown keys are refused, and grade, regrade and replicate_index are strictly typed. A host/image mismatch fails with exit 2 and names the field, instead of falling back to a default.
2. Contract echo. The container writes the contract it parsed into environment_info["container_contract"], after the regrade seed so a prior row cannot overwrite it. One host check, _assert_contract_echoed, replaces _assert_grade_honored, _assert_regrade_honored and _warn_on_version_mismatch. The image-version preflight is now the single reader of the label and refuses instead of warning.
3. Driver rewrite moved host-side. The host stages the task with driver: tempdir and sends the authored sandbox in the contract, so task.json still records driver: docker and the container needs no CE051 exemption.
4. report --rebuild. A printer-free orchestration/run_summary_rebuild.py backs it. It refuses --format, --output, a task directory, a directory inside another run, and a symlinked run.json/run.md.
5. evaluate refreshes run.json. coder-eval evaluate <task_run_dir> rebuilds the owning run's run.json itself. It is best-effort (never changes the exit code), accepts only a coder-eval run summary on the walk up, and skips when --run-dir sits inside the run. write_run_summary now writes atomically.
6. run/execute option sharing: dropped. A factory module was built and gave byte-identical --help, but review found it saved no lines, so cli/ is unchanged. A test now pins every shared flag as identical on both commands.

Rationale lives in .claude/notes/isolation.md (§ The container contract, § The image version preflight, § The contract echo) and .claude/notes/orchestration.md (§ The host-side driver rewrite). Code carries the contract and a Rationale: pointer.

Rebase onto main

14 conflicts across 7 of the 8 commits, all resolved to main's shape plus this branch's facts:

• Notes. Main reflowed .claude/notes/*.md to 88 columns. Both sides were reflowed with the same wrap (checked: it reproduces main's file byte for byte from the old one), then merged, so the resulting diff is content only.
• Slimmed docstrings (CE021, CE051, CE052, one test). Main's shorter form is kept. Where this branch changed a fact in it (the driver rewrite is now in DockerRunner._stage_inputs, and the image preflight refuses), the fact is updated in the docstring and in .claude/notes/lint-rules.md.
• reports_stats.py was split by refactor(reports): generate the pricing mirror and split the reports layer #176. The container_contract exclusion moved to reports/helpers.py::ENV_TABLE_EXCLUDE.
• prose_budget._TYPER_COMMANDS keys are repo-relative since feat(lint): gate tests/ on the prose rules, and cap the comment run as well as the file total #180. The aggregate entry is removed from the new form, and the exemption-list test compares repo-relative paths.

One follow-up commit applies main's prose rules to this branch's own additions: no history wording in docstrings, and module-level imports in the new CLI helpers and tests/test_container_context.py.

Worth a human's eye

• ALLOW_IMAGE_SKEW is a Settings field, not a task field. Image freshness is a property of the operator's machine, not of the evaluation.
• The echo is not a security boundary. The run dir is bind-mounted writable into the agent's container, so an agent can forge container_contract as easily as the verdict beside it. It proves the harness code honored the contract.
• Seen during the live check, not caused by this PR:
  • With --type codex and no --model, the task's default claude-sonnet-4-6 went to the Azure Codex endpoint. The turn ended with 0 messages, 0 tokens and error: null, and the row was recorded as FAILURE. The Codex adapter should record this as an error.
  • Without --run-dir, evaluate writes grade.log and grade.docker.log to its own new runs/<timestamp>/, not beside the graded row.

Verification

After the rebase (ce5d761e): make verify — ruff format, ruff check, pyright 0 errors, 671 custom lint, prose gate 0 essays, 5,899 passed, 2 skipped, 92.74% coverage.

Before the rebase (4d79e933), not repeated after it:

• Every phase had a spec-compliance and a quality review; a final whole-change review found and fixed three Medium issues (stale echo in prior.json, a refused row vanishing from run.json, the refused record lost on the grading path).
• run --help and execute --help byte-identical to the pre-change baseline.
• A real driver: docker task against a freshly built make docker-image image (0.12.1, label matches the host). tasks/agents/claude_hello_world_docker.yaml with --type codex --model gpt-5.4: SUCCESS, 1.0, 3/3. The echo holds all 10 contract keys with grade: true, regrade: false, and task.json records driver: docker.
• Detached grade of that row: coder-eval evaluate <row> --allow-recorded-commands graded in a container of the task's image, SUCCESS → SUCCESS, 3/3, exit 0. The echo shows regrade: true, started_at is unchanged (the agent did not run again), task.execute.json was kept, run.json was refreshed, and no graded_on_host stamp was set.

🤖 Generated with Claude Code

[github-actions]

Claude finished @uipreliga's task in 1m 37s —— View job

---

Code Review

Todo List

• Read .github/code_review.md for review guidelines
• Read CLAUDE.md for project conventions
• Review the full diff (git diff origin/main...HEAD)
• Analyze key architectural changes
• Check cross-file consistency
• Identify what's missing
• Provide structured feedback

Starting code review...

[akshaylive]

One thin I'm unsure about it it's behavior on custom docker images. Skills repo uses a custom docker image based on the coder-eval docker image. In this scenario, do we force consumers to set the env var? If not this PR LGTM.

[uipreliga]

Good question — no, consumers don't need to set anything.

A custom image built FROM coder-eval-agent:<ver> inherits the org.coder-eval.version label from the base (Docker propagates Config.Labels), so the preflight passes with no LABEL line of its own. The skills image is exactly this shape: tests/docker/Dockerfile is FROM ${CODER_EVAL_IMAGE}, which run-coder-eval.yml sets to ghcr.io/uipath/coder-eval-agent:${CE_IMAGE_TAG}, and that tag defaults to the pinned wheel version from tests/.coder-eval-version. Image and host agree, so the check is silent. The overlay adds only npm/dotnet/SDK layers; it never reinstalls coder-eval. The one skills path where the two differ on purpose is a coder_eval_image_tag override, and UiPath/skills#3318 sets the escape hatch there.

A consumer only hits the refusal when their base is pinned to an older coder-eval than the host wheel (or a stale latest) — which is the skew the check exists for. The fix is to rebuild the derived image against a matching base, not the escape hatch; the error text names both the rebuild command and the variable.

Two things ALLOW_IMAGE_SKEW deliberately does not cover:

• A missing label. An image that doesn't descend from the framework image — for example one built with the runtime kit — must apply LABEL org.coder-eval.version itself.
• The contract echo. An overlay that reinstalls coder-eval keeps the inherited label but runs different code, so the result is refused.

One nit for anyone wiring it up: the value must be 1 or 0, never an empty string — ALLOW_IMAGE_SKEW= fails the boolean parse at CLI startup.

All of this is in docs/DOCKER_ISOLATION.md § Using a pre-built custom image, plus the Troubleshooting custom images table.

[akshaylive]

Clean PR. At some point we need to change the documentation structure. Ship it!