feat: give the host→container boundary a contract and fold aggregate into report --rebuild

.claude/harness-candidates.md

@@ -992,3 +992,20 @@ re-derive from scratch.
   such pointer exists today. Deferred because restricting the target is a design
   decision (a `docs/` guide heading is a plausible SSOT target) rather than a
   mechanical guard. Caught in: tests prose slimming, final review (gpt-5.6-sol).
+
+## From container-contract-and-command-surface (2026-09-15)
+
+- [ ] **A field-name assertion against CLI output can pass vacuously through `tmp_path`.**
+  `assert "variant_id" in result.output` matched the echoed `context.json` path, because
+  pytest names each `tmp_path` after the test (`test_a_non_string_variant_id_i…`). Nothing
+  guards it today; the fix was to assert the pydantic `loc` line (`"\nvariant_id\n"`).
+  Deferred because telling a vacuous substring from a real one needs to know the test's
+  own name and what the command echoes — a convention for reviewers, not an AST pattern.
+  Caught in: Phase 1 quality review.
+- [ ] **A path interpolated into Rich markup without `escape()`.** A run directory name is
+  untrusted, and `[/y]` in it raised `rich.errors.MarkupError` (not an `OSError`) inside
+  `evaluate`'s best-effort refresh, after the verdict printed. Nothing guards it; the fix
+  escaped every new console line. Deferred because the rule needs type information (which
+  f-string placeholders are `Path`s) that an AST-only rule does not have, and the existing
+  CLI has many pre-existing unescaped lines a literal rule would flag at once. Caught in:
+  Phase 5 quality review.

.claude/notes/lint-rules.md

@@ -70,7 +70,7 @@ boundary is mechanically detectable, so the rule guards it.
 crosses the container boundary, and the per-task record every dashboard and timeline reads.
 A bare `EvaluationResult.model_validate_json(text)` turns a present-but-malformed file
 into an uncaught exception that crashes the run. Two causes produce such a file: schema
-skew between a stale `:latest` image and the host (the docker version checks only warn),
+skew between a stale `:latest` image and the host that the image preflight did not catch,
 and a truncated or torn write. The incident was at `docker_runner.py`: the parse re-bucketed
 the task to a non-persisted in-memory ERROR with no per-task report. The fix degrades:
 catch `ValueError` and persist a synthetic ERROR record (`batch.py::_load_completed_result`,
@@ -517,8 +517,8 @@ host.
 The motivating bug: `regrade.grading_sandbox_config` rewrote the driver unconditionally
 on BOTH new grading entry points, which also neutralized the `driver: docker` refusal in
 `Sandbox.adopt` — a guard added in the same change specifically to catch this. The
-legitimate suppressions are the in-container rewrite in `run_task_internal_command` and
-the opt-in host-grading branch, which refuses by default and stamps `graded_on_host` on
+legitimate suppressions are the host-side staging rewrite in `docker_runner._stage_inputs`
+and the opt-in host-grading branch, which refuses by default and stamps `graded_on_host` on
 the row.
 
 ## CE052

.claude/notes/orchestration.md

@@ -42,20 +42,20 @@
   non-zero on a crash). The switch is `BatchRunConfig.grade` → `Orchestrator(grade=...)`
   → the **four** grading call sites (single-shot, evaluate-only, the simulation dialog
   check, and post-failure diagnostics); it crosses the docker boundary in `context.json`
-  (defaulting to `True` in-container, so a host predating `execute` keeps grading). It
-  is **deliberately not a task-config field** — no 5-layer merge, no `-D` path — because
-  a task YAML must never declare itself ungraded; only the invoking command decides.
-  `run` and `execute` share one body (`run_command.run_pipeline`) and differ solely in
-  that flag, so there is no third code path. Three things are refused rather than
-  degraded: `--junit-xml` (a report of verdicts, and there are none — though
-  `reports/junit.py` still emits `<skipped>` for an ungraded row it encounters),
-  `--allow-host-grading` (it decides how an ungraded row is GRADED, and `execute` grades
-  nothing), and simulation tasks (their turn-continuation logic reads criteria results,
-  so an ungraded dialog would silently change its own stopping behavior). `stop_early:`
-  blocks are inert under `execute` for the same reason the kill switch exists: the full
-  trajectory is the deliverable. Motivating consumer: an external harness (Harbor /
-  Terminal-Bench 2.0) that builds its own container, calls coder-eval as the agent, and
-  grades with its own tests.
+  (a required contract field with no default, so a host and image that disagree about it
+  fail at parse time). It is **deliberately not a task-config field** — no 5-layer
+  merge, no `-D` path — because a task YAML must never declare itself ungraded; only the
+  invoking command decides. `run` and `execute` share one body
+  (`run_command.run_pipeline`) and differ solely in that flag, so there is no third code
+  path. Three things are refused rather than degraded: `--junit-xml` (a report of
+  verdicts, and there are none — though `reports/junit.py` still emits `<skipped>` for
+  an ungraded row it encounters), `--allow-host-grading` (it decides how an ungraded row
+  is GRADED, and `execute` grades nothing), and simulation tasks (their
+  turn-continuation logic reads criteria results, so an ungraded dialog would silently
+  change its own stopping behavior). `stop_early:` blocks are inert under `execute` for
+  the same reason the kill switch exists: the full trajectory is the deliverable.
+  Motivating consumer: an external harness (Harbor / Terminal-Bench 2.0) that builds its
+  own container, calls coder-eval as the agent, and grades with its own tests.
 
 ### The terminal-status chain
 
@@ -176,8 +176,8 @@ on both the success and the raised path, without catching the `typer.Exit` decid
 Per-suite rollups are skipped entirely under `execute`: a rollup aggregates per-criterion
 results and there are none, so running it would gate a suite on an empty aggregate and
 report a threshold failure for a run that was never measured. The ungraded bucket is named
-explicitly in the aggregate line for the same reason — `coder-eval aggregate <run>` is the
-step right after `coder-eval execute`, so an ungraded run is the FIRST thing it renders, and
+explicitly in the rebuild line for the same reason — `coder-eval report <run> --rebuild` is
+the step right after `coder-eval execute`, so an ungraded run is the FIRST thing it renders, and
 without the term it reads "Aggregated 12 task(s) (0 ok / 0 fail / 0 err)": four numbers that
 no longer sum to `tasks_run`, with nothing on screen to say where the rest went. The
 end-of-run summary likewise reports what happened instead of "0/N succeeded", which for a
@@ -478,9 +478,9 @@ rather than errors live in the run-limits validator for the same post-merge visi
 `task_config.resolved` and `source_file` describe the task as AUTHORED, which is NOT
 always what this process runs.
 
-`run_task_internal_command` rewrites `driver: docker` → `tempdir` before building the
-in-container orchestrator, because it is already inside the container the driver asked
-for. Recording that rewrite made the run's own record deny it ever used docker — and a
+The host stages a `driver: docker` task for its container with `driver: tempdir`, because
+the container is the isolation the driver asked for. Recording that execution copy made
+the run's own record deny it ever used docker — and a
 later `evaluate <run_dir>` reads the driver back out of the record, so the host-grading
 refusal never fired and the `graded_on_host` stamp was never applied. A container task's
 criteria ran against the host filesystem silently, which is the exact outcome that gate
@@ -494,17 +494,29 @@ around it: the docker dispatch guard saw a non-None `Path` and let it through, a
 task-dir mount then silently mounted nothing, so every `$TASK_DIR` criterion resolved
 against the wrong tree and scored a verdict nobody could explain.
 
-### The in-container driver rewrite
-
-CE051 forbids rewriting `sandbox.driver`, and this is its single exemption: the process is
-already inside the container the docker driver asked for, so the isolation the driver names
-is present rather than bypassed, and a nested docker would be both wrong and impossible (no
-docker CLI in the image). The rewrite goes through `model_validate` rather than
-`model_copy(update=...)`, matching its sibling in `regrade.grading_sandbox_config`: `update`
-skips BOTH pydantic and pyright, so a typo produces a `SandboxConfig` violating its own
-`Literal` and only surfaces far downstream. Two driver-rewrite sites landing in one change
-with two different levels of type safety is how the weaker one becomes the pattern people
-copy.
+### The host-side driver rewrite
+
+CE051 forbids rewriting `sandbox.driver` silently, and `DockerRunner._stage_inputs` is one of
+its two exemptions: the host resolves the driver for the container it is itself about to
+start, so the isolation the driver names is present rather than bypassed, and a nested docker
+inside the image would be both wrong and impossible (no docker CLI in it). The rewrite happens
+where both values are in hand — the staged `task.yaml` carries the execution copy and
+`ContainerContext.authored_sandbox` carries the block as authored, which the container
+records. Doing it inside the container instead took a rewrite plus a "captured BEFORE the
+rewrite" local in the consumer, and a lint exemption for code on the far side of the boundary.
+
+The rewrite goes through `model_validate` rather than `model_copy(update=...)`, matching its
+sibling in `regrade.grading_sandbox_config`: `update` skips BOTH pydantic and pyright, so a
+typo produces a `SandboxConfig` violating its own `Literal` and only surfaces far downstream.
+Two driver-rewrite sites with two different levels of type safety is how the weaker one
+becomes the pattern people copy.
+
+The two sites are deliberately NOT collapsed into a `SandboxConfig.as_tempdir()` helper.
+CE051 exempts `models/sandbox.py` outright ("the model's own construction"), so moving the
+rewrite there would take both call sites out of the rule's view and turn a guarded operation
+into an unguarded one-liner any future caller could reach. Each site carries a different
+reason in its `noqa`, and that reason text is the operator-visible control the rule exists to
+force.
 
 ## Three routes, resolved separately
 

.claude/shared/run-layout.md

@@ -13,7 +13,7 @@ runs/<run_id>/<variant_id>/<task_id>/<NN>/{task.json, task.log, artifacts/}
 - `task.json` — the persisted per-replicate result (the consumer contract; carries the large `iterations` array — still accepted under its former name `turns` when reading, but not what current runs write).
 - `task.json.malformed` — present only on the docker degrade path: when an existing `task.json` fails to parse (schema skew from a stale `:latest` image, or a truncated/torn write), the docker runner moves the unparseable original aside to this sidecar and writes a synthetic `final_status=ERROR` `task.json` in its place. Diagnostic-only; `rglob("task.json")` consumers do not match it.
 - `task.execute.json` — present only after a DETACHED grade (`coder-eval evaluate <run_dir>` or `coder-eval run --resume` over a `NOT_GRADED` row). The pre-grade snapshot of `task.json`, written once and never overwritten by a later grade, so "this run was executed separately from grading" stays auditable. Diagnostic-only; `rglob("task.json")` consumers do not match it.
-- `task.json.graded` — present only after `coder-eval execute --driver docker` refused a container's verdict: the runtime image predated `execute` and graded anyway, so the runner quarantines the graded record here rather than leaving it readable as `task.json`, where a later `--resume` / `aggregate` would fold in exactly the row it declined to publish. Diagnostic-only; `rglob("task.json")` consumers do not match it.
+- `task.json.unhonored` — present only after the docker runner refused a container's result because it did not echo the contract the host staged (`environment_info.container_contract` absent or different — for example an image that predates `execute` and graded anyway, or one that re-ran the agent instead of grading). The runner quarantines the record here rather than leaving it readable as `task.json`, where a later `--resume` or run-level rebuild would fold in exactly the row it refused, and writes a synthetic `final_status=ERROR` `task.json` in its place so the row stays visible. On a detached grade the refused record is folded back beside the graded row. Diagnostic-only; `rglob("task.json")` consumers do not match it.
 - `grade.log` — present only after a DETACHED grade over this directory (`coder-eval run --resume`). The grading pass's own log. It is a separate file because the log handler truncates whatever file it opens, so writing to `task.log` would destroy the agent trajectory log the run already paid for.
 - `task.log` — the human-readable task log; `artifacts/` — files the agent produced.
 

docker/Dockerfile.runtime

@@ -14,7 +14,7 @@
 #   /usr/local/bin/coder_eval_entrypoint.sh      entrypoint (host pins --entrypoint here)
 #   LABEL org.coder-eval.version=<ver>           for `docker inspect` / parity
 #
-# NOTE: the host's _assert_runtime_image gate inspects the *injected task image*,
+# NOTE: the host's _preflight_image_contract gate inspects the *injected task image*,
 # not this kit image — labels don't survive `COPY --from`. The converter re-stamps
 # org.coder-eval.version on the injected image (that's the label the host asserts).
 # This kit carries it too purely for `docker inspect`/parity.

docs/DOCKER_ISOLATION.md

@@ -105,8 +105,9 @@ coder-eval run tasks/byod_smoke_test.yaml
 | Symptom | Cause and fix |
 | --- | --- |
 | `docker: Error response from daemon: pull access denied` | The image isn't built locally and isn't pullable. Check `docker images`, then rebuild it. Docker treats an unknown local tag as a remote reference, which is why the error mentions a pull. |
-| `Image <your-image> coder_eval <a> != host <b>` | The custom image carries an `org.coder-eval.version` label inherited from a stale framework base. Rebuild the base with `make docker-image`, then rebuild your derived image with `docker build --no-cache`. |
-| `Image <your-image> has no org.coder-eval.version label` | The image doesn't descend from `coder-eval-agent` (or predates the label). Rebase it on the framework image, or use the runtime kit. |
+| `Image <your-image> runs coder_eval <a> but the host runs <b>` | The run is refused before the container starts: the image carries an `org.coder-eval.version` label from a different coder-eval than the one installed on the host, usually inherited from a stale framework base. Rebuild the base with `make docker-image`, then rebuild your derived image with `docker build --no-cache`. To run a deliberately different image anyway (for example an unreleased build under test), set `ALLOW_IMAGE_SKEW=1` in the environment or `.env`; the mismatch then only warns, and the run gives up the reproducibility guarantee. |
+| `Image <your-image> is not a coder-eval runtime image (missing the org.coder-eval.version label)` | The image doesn't descend from `coder-eval-agent` (or predates the label). Rebase it on the framework image, or use the runtime kit. `ALLOW_IMAGE_SKEW` does not bypass this. |
+| `The container returned a result with no container_contract echo` or `The container did not honor the contract it was sent` | The code inside the image does not match the host's, even if its label agrees — for example an overlay image that reinstalled coder-eval, or a stale image rebuilt under the same tag. The result is refused: the record is moved to `task.json.unhonored` and a synthetic ERROR `task.json` takes its place. Rebuild or pull a matching image. `ALLOW_IMAGE_SKEW` does not bypass this. |
 
 ## Building the image from a task Dockerfile
 
@@ -183,9 +184,10 @@ Behavior:
   `coder-eval-task-<task_id>:built`, so repeat runs of the same task reuse
   Docker's layer cache. Edit the Dockerfile and the next run rebuilds the
   changed layers only.
-- **Version-label check skipped** — the `org.coder-eval.version` preflight only
-  applies to the framework image; task-built images don't carry it and won't
-  warn.
+- **Version-checked too** — the built image inherits `org.coder-eval.version`
+  from its `FROM coder-eval-agent` base, so the same preflight applies: a missing
+  label, or a version that differs from the host's, refuses the run (see
+  [Troubleshooting custom images](#troubleshooting-custom-images)).
 
 A build failure aborts the task with a `DockerBuildError` (a `DockerRunError`
 subclass) carrying `docker build`'s output. Because the build runs before the

docs/REPORT_SCHEMA.md

@@ -11,7 +11,7 @@ Coder Eval writes machine-readable JSON alongside every markdown/HTML report. Th
 page is the field-level reference for consumers (dashboards, CI parsers, evalboard
 forks). For the on-disk directory tree see
 [User Guide → Output Structure](USER_GUIDE.md#output-structure); for how to
-re-generate these files see [`coder-eval report` / `aggregate`](USER_GUIDE.md#cli-commands).
+re-generate these files see [`coder-eval report`](USER_GUIDE.md#cli-commands) (`--rebuild` for `run.json`).
 
 All JSON is Pydantic `model_dump_json` output — keys are the model field names
 verbatim (no aliases, except `iterations` also accepts the legacy key `turns` on
@@ -21,7 +21,7 @@ read). Times are ISO-8601.
 
 | File | Model | When |
 | --- | --- | --- |
-| `run.json` / `run.md` | `RunSummary` | Every run (and rebuildable via `coder-eval aggregate`) |
+| `run.json` / `run.md` | `RunSummary` | Every run; refreshed by `coder-eval evaluate <run_dir>` and rebuildable via `coder-eval report <run_dir> --rebuild` |
 | `<variant>/<task_id>/<NN>/task.json` | `EvaluationResult` | One per replicate |
 | `<variant>/<task_id>/<NN>/task.execute.json` | `EvaluationResult` | Pre-grade snapshot, written once by a detached grade (`evaluate <run_dir>` / `run --resume`). Deliberately **not** matched by `rglob("task.json")`, so it never enters an aggregation. |
 | `<variant>/<suite_id>/suite.json` / `.md` | `SuiteRollup` | Dataset-backed suites only |
@@ -361,6 +361,6 @@ respectively), checked after each completed agent turn — see
 ## See also
 
 - [User Guide → Output Structure](USER_GUIDE.md#output-structure) and the
-  [`aggregate`](USER_GUIDE.md#cli-commands) command
+  [`report --rebuild`](USER_GUIDE.md#cli-commands) command
 - [A/B Experiments → Reading the Report](AB_EXPERIMENTS.md#reading-the-report)
 - [Task Definition Guide](TASK_DEFINITION_GUIDE.md)