feat(e2e): batch cases across Harbor tasks

[PsiACE]

Summary

No linked issue or RFC. Batch already-selected compatible v1 E2E tasks into one run-local Harbor task so they share a trial and container while retaining independent reset, scope, evidence, and evaluation. Single-task selection stays independent; collect-all and fail-fast are supported. No public API changes.

Validation

SQLite and OceanBase acceptance; Python 3.11–3.14; quality, docs, and license checks.

AI usage

Codex (GPT-5) was used for implementation and verification.

[PsiACE]

@knqiufan I’ve updated the existing LoCoMo tasks to share a Harbor runtime. Could you help review this?

[knqiufan]

@PsiACE Thanks for the ping — I reviewed the shared Harbor batch path.

The happy path looks solid: compatible LoCoMo tasks share one trial/container, scopes stay per-task via invocation_scopes + scoped_exec_env, evidence is split by step prefix, and selecting a single LoCoMo still uses the independent path. CI matches that.

I checked out a few failure-path issues:

1. collect-all does not continue after an agent exception.
   PowerContextBubAcpAgent.run() touches the failure marker and re-raises. The shared test.sh then exits 1 without writing reward.txt. On Harbor 0.16, exception_info + missing verifier_result aborts remaining steps regardless of min_reward. So collect-all and fail-fast behave the same on this path, which contradicts the CLI help.
   Suggestion: if the marker exists, write reward 0 so the verifier returns. Then collect-all can continue (no min_reward) and fail-fast can stop via min_reward = 1.0. I guess add a regression that actually raises from the agent under both policies.

2. Batch collection status is now coupled to Harbor rewards.
   test_memory_acceptance_does_not_require_the_harbor_task_to_pass documents that a Harbor reward of 1/3 is diagnostic only; Memory evaluation can still pass when status is completed.
   run_task only fails collection on trial-level exception_type. _step_failed() treats any step exception_info or reward < 1 as status=failed, which then fails collection_completed.
   Current LoCoMo success writes reward 1, so this is not a happy-path locomo flake — it is a contract change in the batch runner. It will show up on step-level exceptions, any diagnostic reward < 1, and immediately if we adopt “write reward 0” from (1). I guess keep collection_completed = “this case finished and evidence was collected”; use native rewards only for reporting and fail-fast.

3. One bad capture line can drop the entire batch.
   collect_task_artifacts() sits outside run_task_group’s try/except, and CaptureRecord.model_validate_json() is not isolated per source. A truncated/invalid powercontext-capture.jsonl raises out of the group, so no task gets replay/evaluation. The single-task path absorbs this into that task’s errors. That breaks “evidence remains independent”. I guess collect per source, attach parse errors to that task, and still write artifacts for the others.

4. Re-running against the same evidence directory fails.
   prepare_runtime_dataset() raises FileExistsError if harbor-runtime-dataset/batch-* already exists. Compose/run.sh reuse POWERCONTEXT_E2E_OUTPUT and do not clean it. CI is green because the workspace is fresh. Local make harness-compose-acceptance a second time will fail before Harbor starts. I guess replace the directory or namespace it by run_id.

A few follow-ups, not merge blockers if the four above are handled:

• skipped tasks still go through full Memory assertions, so the aggregate report looks like Memory quality failures rather than “shared trial stopped early”.
• _source_harbor_observation averages only steps that have rewards, so a failed task can still get task_outcome=passed.
• The schemas are still powercontext.e2e-task/v1 and powercontext.e2e-evidence/v1 with extra=forbid, but manifests now have batch, every replay now emits scope_id (including single-task), batch replay adds source_task_checksum, and status gained skipped. Old extra=forbid consumers reject the new files. That sits poorly next to “No public API changes”. Either bump the schema with a migration story, or keep v1 compatible and add a compatibility test.