Improve generic Harbor and MiniSWE evaluation resilience

[neubig]

Summary

• provide the OpenAI-compatible proxy aliases expected by the MiniSWE Harbor agent while retaining the existing generic LLM variables
• add an optional bounded-history LiteLLM model that preserves setup and complete assistant/tool turns
• treat a primary verifier reward as authoritative when Harbor records an agent timeout or non-zero exit
• refresh converted output from raw Harbor results during evaluation so rescoring uses the current converter

Root cause

Long MiniSWE runs retained unbounded message history until requests exceeded model context limits. Separately, completed verifier results could be classified as incomplete after an agent error, and evaluation-only reruns continued to aggregate stale converted JSONL instead of the authoritative Harbor trial files.

Impact

The changes are benchmark- and provider-agnostic: context trimming is opt-in and configurable, scoring follows Harbor's primary verifier reward, and rescoring consistently uses raw results when available.

Validation

Live evidence was refreshed on 2026-07-19 UTC from the existing PR branch codex/deepswe-pier-eval-20260716 after re-fetching PR #759. Environment: uv 0.11.24, Python 3.12.13, OpenHands SDK submodule 43376f1, Harbor CLI 0.20.0 installed in the local .venv, Docker daemon 29.6.1. Current remediation commit: 652d92dcd1186cb719aaadb222a31c93d7cd2c1c (empty evidence-only commit; no source files needed changes).

Commands and observed results:

make build

Result: submodule initialized at 43376f1868ffd702746080714a59c16d3f69ec12, uv sync --dev completed, and pre-commit hooks installed.

uv pip install harbor

Result: installed Harbor CLI 0.20.0 into .venv for the live run.

PYTHONPATH=/tmp/pr759-live OPENHANDS_SUPPRESS_BANNER=1 PYTHONUNBUFFERED=1 \
  /workspace/project/benchmarks/.venv/bin/harbor-infer \
  /tmp/pr759-live/llm-dummy.json \
  --harbor-target /tmp/pr759-live/fixture \
  --harbor-target-type path \
  --harbor-agent nop \
  --harbor-executable /workspace/project/benchmarks/.venv/bin/harbor \
  --benchmark-slug pr759-harbor-live \
  --output-dir /tmp/pr759-live/output-plugin \
  --num-workers 1 \
  --n-limit 1 \
  --note 20260719-live-nop-verifier-plugin \
  --harbor-arg=--yes \
  --harbor-arg="--verifier verifier_plugin:RewardOneVerifier"

Result: actual repository harbor-infer entrypoint invoked Harbor on a real local Harbor task fixture using Harbor nop agent and a deterministic local verifier plugin returning VerifierResult(rewards={"reward": 1.0}), so no external LLM/API spend occurred. Harbor output reported 1/1 Mean: 1.000, Trials 1, Exceptions 0, Reward 1.0 Count 1. The converter logged Processed trial fixture: reward=1.0 and Wrote 1 successful + 0 failed entries to /tmp/pr759-live/output-plugin/pr759-harbor-live/dummy/noop-model_sdk_43376f1_maxiter_100_N_20260719-live-nop-verifier-plugin/output.jsonl. Cost report showed $0.000000.

OPENHANDS_SUPPRESS_BANNER=1 PYTHONUNBUFFERED=1 \
  /workspace/project/benchmarks/.venv/bin/harbor-eval \
  /tmp/pr759-live/output-plugin/pr759-harbor-live/dummy/noop-model_sdk_43376f1_maxiter_100_N_20260719-live-nop-verifier-plugin/output.jsonl \
  --output-file /tmp/pr759-live/plugin-report.json

Result: actual repository harbor-eval entrypoint refreshed output.jsonl from raw harbor_output, regenerated the report, and logged Resolved 1/1 completed instances. /tmp/pr759-live/plugin-report.json contained total_instances=1, completed_instances=1, resolved_instances=1, incomplete_instances=0, error_instances=0, and aggregate cost/tokens all zero.

uv run pytest tests/test_harbor_eval_infer.py tests/test_harbor_run_infer.py \
  tests/test_harbor.py tests/test_terminalbench.py tests/test_miniswe_context_history.py

Result: 68 passed, 2 warnings in 0.20s.

Limitations: this was the smallest safe Harbor-compatible live fixture, not a full MiniSWE or Terminal-Bench run. It deliberately used Harbor nop plus a local deterministic verifier to avoid external LLM spend. A preliminary shell-script verifier fixture that wrote /logs/verifier/reward.txt through Harbor 0.20.0 in this DinD environment ended with RewardFileNotFoundError; the successful evidence therefore uses a custom real Harbor verifier plugin. Docker required enabling the current sudo-capable user to access /var/run/docker.sock in this sandbox.

This PR was prepared by an AI agent on behalf of the user. This validation update was added by an AI agent (OpenHands) on behalf of the user.

Closes #761