Reference-harness grid: scoring convention, installed-harness support, producer fixes

[varunursekar]

Four changes the reference-harness grid needed. Everything here is code and config;
the run artefacts live in the gitignored runs/ and are not included.

1. Score a trial the harness killed as zero, not as absent

trial_rewards dropped every rewardless trial, so a harness that crashed was scored
only on the trials that survived. Now it follows runs/recompute.py — the convention
that produced the pinned baselines: a harness kill scores 0, a platform kill is
dropped, and an unrecognised exception is a hard error instead of a quiet omission.

This changes published numbers. It recovers 73 zeroes across the grid; the largest
single move is swe-atlas × mini-swe-agent, 0.0656 → 0.0533, where 28 of 150 trials were
being ignored.

2. Let rescore run an installed harness in place of the seed

--agent swaps the seed program for a harness a practitioner could install and changes
nothing else — same dataset ref, partition, rounds, aggregation and pinned target model
— so the harness is the only variable and the number stays comparable to
baseline_reward. Four supporting flags, each added because a specific harness needed
it: --setup-timeout-multiplier (opencode's nvm → node 22 → npm -g install overruns
harbor's default), --agent-env (goose needs the proxy pointed at explicitly),
--with-requirement (harbor-native harnesses import their framework in the
orchestrator, which harbor does not declare), and --harbor-requirement.

3. Pin gpt-5.x producers to one upstream deployment

An unqualified model group load-balances across deployments, and Responses-API
encrypted reasoning is decryptable only by the deployment that produced it, so every
turn after the first fails invalid_encrypted_content. Measured on swe-atlas: 8 of 8
replays failed against the bare group, 0 of 5 on azure_ai. It killed both gpt-5.6-sol
cells at 2m58s under opencode as well as codex, so it is not the codex-only fault
terminal-bench's instance suggested.

Aliases apply after the allow-list check, so allowed_models still governs what an
optimizer may ask for and a cell's label still means what it says.

4. Give the producer the credential surfaces goose and openhands-sdk read

openhands-sdk reads its own LLM_* pair and goose reads OPENAI_HOST +
OPENAI_BASE_PATH, which it joins — so the host must not already carry /v1 or the
request lands on /v1/v1/chat/completions and the proxy 403s. The producer token is
minted per run, so neither can come from build.yaml agent_env.

Verification

• tests/test_v05_cli.py: 21 passed
• all three modified build.yamls parse, with model_aliases landing under
  inference_gateway.producer
• rescore_candidate.py compiles; the commit split was verified byte-identical to the
  working version

🤖 Generated with Claude Code

Greptile Summary

This PR updates reference-harness rescoring and producer configuration.

• Classifies rewardless harness failures as zero and drops recognized infrastructure failures.
• Adds installed-agent, environment, dependency, Harbor-version, and setup-timeout options to the rescoring script.
• Pins GPT-5.x producer aliases to specific upstream deployments.
• Exposes additional producer credentials and endpoint variables for OpenHands SDK and Goose.

Confidence Score: 4/5

The PR is not yet safe to merge because qualified infrastructure exceptions can still abort otherwise valid rescoring runs.

The current classifier compares persisted exception types directly against bare names, while existing Harbor result fixtures use qualified values such as openai.RateLimitError; those failures therefore reach the unclassified-exception exit instead of being omitted from scoring.

Files Needing Attention: harness-engineering-bench/scripts/rescore_candidate.py

Important Files Changed

Filename	Overview
harness-engineering-bench/scripts/rescore_candidate.py	Adds installed-harness execution and explicit rewardless-trial classification; the previously reported qualified-exception mismatch remains.
vero/src/vero/harbor/cli.py	Adds producer credential and endpoint environment surfaces used by OpenHands SDK and Goose.
harness-engineering-bench/gaia/baseline/build.shell.yaml	Pins GPT-5.6 producer aliases to Azure AI deployments.
harness-engineering-bench/officeqa/baseline/build.yaml	Pins GPT-5.1 through GPT-5.5 producer aliases to Azure AI deployments.
harness-engineering-bench/swe-atlas-qna/baseline/build.yaml	Pins GPT-5.6 producer aliases to Azure AI deployments.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Load benchmark build] --> B{Seed or installed agent?}
    B -->|Seed| C[Copy seed workspace]
    B -->|Installed| D[Configure Harbor agent]
    C --> E[Run fixed rounds and tasks]
    D --> E
    E --> F[Read trial results]
    F --> G{Reward present?}
    G -->|Yes| H[Include reward]
    G -->|No| I{Failure classification}
    I -->|Harness| J[Include zero]
    I -->|Platform| K[Drop trial]
    I -->|Unrecognized| L[Abort rescore]
    H --> M[Aggregate score]
    J --> M
    K --> M

Loading

_{Reviews (2): Last reviewed commit: "Classify the five failure types a rescor..." | Re-trigger Greptile}