Skip to content

fix(cli): point integration_test users at the platform flow - #585

Open
ayushnangia wants to merge 5 commits into
hud-evals:mainfrom
ayushnangia:fix/eval-integration-test-hint
Open

fix(cli): point integration_test users at the platform flow#585
ayushnangia wants to merge 5 commits into
hud-evals:mainfrom
ayushnangia:fix/eval-integration-test-hint

Conversation

@ayushnangia

@ayushnangia ayushnangia commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

Summary

A user following the official 01-coding-template README's authoring loop runs:

hud eval <env-name> integration_test --task-ids my-task -y

Against a local source (or before deploying), the CLI rejects the agent with a bare Invalid agent: integration_test. Must be one of: claude, openai, gemini, openai_compatible — leaving the documented flow with no explanation that integration_test is the platform's authoring agent (it pre-stages Task.validation, runs the graders, and expects Reward 1.0).

Changes

  • hud/cli/eval.py: the agent_type validator recognizes integration_test and raises a precise, actionable error — deploy the environment and run it against the platform (hud eval <env-name> integration_test --task-ids <slug> -y); not available with --runtime local. Unknown agents keep the existing generic message.

Validation

  • New test in hud/cli/tests/test_eval_config.py (test_parse_agent_type_points_integration_test_to_the_platform).
  • pytest hud/cli/tests/test_eval_config.py38 passed; full local suite → 136 passed.

Scope note

Deliberately an error-message fix, not a local implementation of the integration_test agent — that belongs on the platform side and would be a feature PR.


Note

Low Risk
Changes are user-facing error text and Harbor schema guards on adapt/export; no auth, runtime execution, or agent behavior changes beyond clearer validation.

Overview
CLI: Choosing agent integration_test no longer surfaces a generic “invalid agent” message. EvalConfig raises a specific error explaining that integration_test is the platform authoring agent (pre-stages golden Task.validation, expects Reward 1.0) and must be run against a deployed environment on the platform—not with --runtime local.

Harbor integration: Introduces HARBOR_SCHEMA_VERSION = "1.0". harbor.adapt rejects task.toml files that declare a different schema_version; missing schema_version still adapts for legacy exports. export writes schema_version into generated task.toml for round-trip consistency. Contract tests cover unknown, supported, and absent schema versions.

Reviewed by Cursor Bugbot for commit 612ccb7. Bugbot is set up for automated code reviews on this repo. Configure here.

The exported task.toml carries a top-level 'version' key, but the
adapter parses 'schema_version' — so the stamped version was invisible
to validation and a task declaring any unknown schema_version adapted
silently with wrong semantics. Define HARBOR_SCHEMA_VERSION, fail
adaptation loudly for unsupported values, and stamp the field in export
so round-tripped tasks validate against the same contract.
The 01-coding-template README documents 'hud eval <env-name>
integration_test' as the authoring check (pre-stages Task.validation,
runs graders, expects Reward 1.0) — but the local CLI rejected the name
with a bare 'Invalid agent: integration_test', leaving a user who
followed the README with no path forward. Recognize the name and give
the precise remediation: it is the platform's authoring agent, run
against a deployed env, not available with --runtime local.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit ca13eb0. Configure here.

Comment thread hud/cli/eval.py Outdated
ayushnangia added a commit to ayushnangia/hud-python that referenced this pull request Aug 15, 2026
Implements the 01-coding-template's documented authoring flow locally:
'hud eval <taskset> integration_test' now pre-stages the golden solution
by replaying every Task.validation tool call through the task's own MCP
capabilities, ends the trace, and lets the environment's scenario
graders run. The CLI enforces the authoring gate afterwards: every run
must grade Reward 1.0, otherwise the eval exits non-zero naming the
failing reward/raw grade.

Replaces the hud-evals#585 error path with the real implementation. No LLM calls;
timeout_seconds bounds the staging pass.
ayushnangia added a commit to ayushnangia/hud-python that referenced this pull request Aug 15, 2026
Implements the 01-coding-template's documented authoring flow locally:
 now pre-stages the golden solution
by replaying every Task.validation tool call through the task's own MCP
capabilities, ends the trace, and lets the environment's scenario
graders run. The CLI enforces the authoring gate afterwards: every run
must grade Reward 1.0, otherwise the eval exits non-zero naming the
failing reward/raw grade.

Replaces the hud-evals#585 error path with the real implementation. No LLM calls;
timeout_seconds bounds the staging pass.
@ayushnangia
ayushnangia force-pushed the fix/eval-integration-test-hint branch from 73ddba5 to d9e3c5a Compare August 15, 2026 20:07
@ayushnangia

Copy link
Copy Markdown
Contributor Author

Update: the real thing now exists locally. #586 implements integration_test as a local agent — it replays Task.validation through the task's own MCP capabilities, lets the scenario graders run, and the CLI enforces the Reward-1.0 authoring gate (exits non-zero below 1.0, naming the failing raw grade). If #586 lands, this PR's error path is superseded: the name parses, and the only remaining refusal is for the gateway shortcut (integration_test makes no LLM calls). Happy to close this one in #586's favor at that point — or keep it if reviewers want the hint to remain for the platform-only flags.

@ayushnangia

Copy link
Copy Markdown
Contributor Author

Thanks Bugbot — the guidance is intentionally the interim behavior: this PR only adds the helpful error, and #586 (same author) implements the real local agent, which makes the name parse and replaces this error path. Cross-comment is on #586's description; happy to close this one in #586's favor if reviewers prefer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant