fix(cli): point integration_test users at the platform flow - #585
fix(cli): point integration_test users at the platform flow#585ayushnangia wants to merge 5 commits into
Conversation
The exported task.toml carries a top-level 'version' key, but the adapter parses 'schema_version' — so the stamped version was invisible to validation and a task declaring any unknown schema_version adapted silently with wrong semantics. Define HARBOR_SCHEMA_VERSION, fail adaptation loudly for unsupported values, and stamp the field in export so round-tripped tasks validate against the same contract.
The 01-coding-template README documents 'hud eval <env-name> integration_test' as the authoring check (pre-stages Task.validation, runs graders, expects Reward 1.0) — but the local CLI rejected the name with a bare 'Invalid agent: integration_test', leaving a user who followed the README with no path forward. Recognize the name and give the precise remediation: it is the platform's authoring agent, run against a deployed env, not available with --runtime local.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ca13eb0. Configure here.
Implements the 01-coding-template's documented authoring flow locally: 'hud eval <taskset> integration_test' now pre-stages the golden solution by replaying every Task.validation tool call through the task's own MCP capabilities, ends the trace, and lets the environment's scenario graders run. The CLI enforces the authoring gate afterwards: every run must grade Reward 1.0, otherwise the eval exits non-zero naming the failing reward/raw grade. Replaces the hud-evals#585 error path with the real implementation. No LLM calls; timeout_seconds bounds the staging pass.
Implements the 01-coding-template's documented authoring flow locally: now pre-stages the golden solution by replaying every Task.validation tool call through the task's own MCP capabilities, ends the trace, and lets the environment's scenario graders run. The CLI enforces the authoring gate afterwards: every run must grade Reward 1.0, otherwise the eval exits non-zero naming the failing reward/raw grade. Replaces the hud-evals#585 error path with the real implementation. No LLM calls; timeout_seconds bounds the staging pass.
73ddba5 to
d9e3c5a
Compare
|
Update: the real thing now exists locally. #586 implements |
|
Thanks Bugbot — the guidance is intentionally the interim behavior: this PR only adds the helpful error, and #586 (same author) implements the real local agent, which makes the name parse and replaces this error path. Cross-comment is on #586's description; happy to close this one in #586's favor if reviewers prefer. |

Summary
A user following the official 01-coding-template README's authoring loop runs:
Against a local source (or before deploying), the CLI rejects the agent with a bare
Invalid agent: integration_test. Must be one of: claude, openai, gemini, openai_compatible— leaving the documented flow with no explanation thatintegration_testis the platform's authoring agent (it pre-stagesTask.validation, runs the graders, and expects Reward 1.0).Changes
hud/cli/eval.py: theagent_typevalidator recognizesintegration_testand raises a precise, actionable error — deploy the environment and run it against the platform (hud eval <env-name> integration_test --task-ids <slug> -y); not available with--runtime local. Unknown agents keep the existing generic message.Validation
hud/cli/tests/test_eval_config.py(test_parse_agent_type_points_integration_test_to_the_platform).pytest hud/cli/tests/test_eval_config.py→ 38 passed; full local suite → 136 passed.Scope note
Deliberately an error-message fix, not a local implementation of the integration_test agent — that belongs on the platform side and would be a feature PR.
Note
Low Risk
Changes are user-facing error text and Harbor schema guards on adapt/export; no auth, runtime execution, or agent behavior changes beyond clearer validation.
Overview
CLI: Choosing agent
integration_testno longer surfaces a generic “invalid agent” message.EvalConfigraises a specific error explaining thatintegration_testis the platform authoring agent (pre-stages goldenTask.validation, expects Reward 1.0) and must be run against a deployed environment on the platform—not with--runtime local.Harbor integration: Introduces
HARBOR_SCHEMA_VERSION = "1.0".harbor.adaptrejectstask.tomlfiles that declare a differentschema_version; missingschema_versionstill adapts for legacy exports.exportwritesschema_versioninto generatedtask.tomlfor round-trip consistency. Contract tests cover unknown, supported, and absent schema versions.Reviewed by Cursor Bugbot for commit 612ccb7. Bugbot is set up for automated code reviews on this repo. Configure here.