feat(agents): local integration_test authoring agent

[ayushnangia]

Summary

The 01-coding-template documents hud eval <env-name> integration_test as the shipping check for every task: pre-stage the golden solution, run the graders, require Reward 1.0. Until now that agent only existed on the platform — the local CLI rejected the name (previously patched with a helpful error in #585). This PR implements the real thing locally.

What it does

• AgentType.INTEGRATION_TEST + IntegrationTestConfig (no model, timeout_seconds bounds the staging pass).
• IntegrationTestAgent — an Agent that makes no LLM calls: it opens the task's own MCP capabilities, replays every Task.validation tool call through them (recording ToolSteps like any other agent), and ends the trace with an empty answer so the environment's scenario graders grade the staged workspace. String-arguments steps are rejected without executing, unknown/invalid steps are skipped with warnings, and a missing-capability call surfaces as an error result.
• Run.validation — rollout attaches task.validation to the live run so the agent can consume it.
• The authoring gate — after the rollout, the CLI enforces the documented contract: every run must grade Reward 1.0; anything lower (or a grading error) exits non-zero and names the failing reward/raw grade, so the authoring loop fails loudly instead of shipping a task whose golden doesn't pass its own hidden graders.
• gateway_provider raises for this type (it's not a gateway shortcut — no LLM).

Validation

• New tests: hud/agents/tests/test_integration_test.py (dispatch + ToolStep recording, empty-validation no-op, invalid-entry tolerance) and CLI tests (config accepts the name, the reward gate exits on <1.0 and on grading errors).
• Full local suite: 392 passed (hud/tests + capabilities + patches + agents + cli).
• ruff + ty clean.

Notes

• Supersedes the fix(cli): point integration_test users at the platform flow #585 error path (the name now parses; the error only remains in spirit for the gateway shortcut). fix(cli): point integration_test users at the platform flow #585 can be closed as superseded if this lands.
• No platform changes: sync payloads and the platform-side agent are untouched.
• Template contract: the 01-coding-template README's authoring loop is the spec this implements.

---

Note

Medium Risk
Changes the eval rollout path and adds a hard CLI exit on grading outcomes; no auth or payment logic, but authors rely on this gate for shipping tasks.

Overview
Implements the local hud eval … integration_test authoring check: stage the golden solution, run graders, and fail the CLI if reward is not 1.0.

IntegrationTestAgent (with AgentType.INTEGRATION_TEST and IntegrationTestConfig) opens the task’s MCP capabilities and replays each Task.validation step as ToolStep records—no LLM calls. Invalid or unknown validation entries are skipped with warnings; string arguments are not executed.

rollout now sets Run.validation from task.validation so the agent can consume golden steps. After eval, _enforce_integration_test_reward exits non-zero when any run grades below 1.0 or grading errors.

Config parsing accepts the integration_test agent name (including before AgentType enum resolution). gateway_provider correctly rejects this type as a gateway shortcut.

^{Reviewed by Cursor Bugbot for commit d13314d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit d13314d. Configure here.}

[cursor]

Agent skips SSH golden staging

High Severity

IntegrationTestAgent only opens MCP bindings and dispatches via call_tool, so it never drives the SSH workspace the coding template actually publishes. Golden bash/shell validation steps therefore do not run; graders see the unstaged baseline and the Reward 1.0 gate fails a shippable task.

Additional Locations (1)

• hud/agents/integration_test.py#L92-L124

 

^{Reviewed by Cursor Bugbot for commit d13314d. Configure here.}

[cursor]

Gateway factory accepts integration_test

Medium Severity

create_agent treats every AgentType value as a HUD gateway shortcut. Adding INTEGRATION_TEST makes create_agent("integration_test") construct a non-LLM agent and hide it behind a GatewayAgent cast, instead of raising the way gateway_provider already does.

Additional Locations (1)

• hud/types.py#L137-L140

 

^{Reviewed by Cursor Bugbot for commit d13314d. Configure here.}