Stream Claude CLI steps and inference trace IDs

[solvemproblr]

Summary

• stream Claude CLI stream-json output as it arrives over SSH
• record assistant and tool steps with their real execution timestamps
• propagate the active HUD trace ID into Claude Code gateway requests
• add a reconnect-aware SSH process API for streaming commands
• preserve result, usage, error, and raw-message trace metadata

Issue

ClaudeSDKAgent launches Claude Code as a separate process in the environment workspace. Two in-process assumptions therefore broke:

1. The agent awaited the entire remote CLI command before parsing stdout, so HUD received aggregate steps only after Claude exited.
2. HUD's normal HTTP client hook could not reach Claude Code's requests, so gateway inference logs were billed but had a null trace_id.

Original affected trace: https://hud.ai/trace/80c7c75a-db92-436b-a8c4-d0cc38745d79

Baseline streaming reproduction: https://hud.ai/trace/8163171d6ecb4e9fa6b34c179cd5d7e1

Solution

The agent now opens a streaming SSH process and incrementally translates Claude assistant, tool-use, tool-result, and terminal result messages into canonical HUD steps. Tool calls are paired with their results, process cleanup remains bounded on cancellation, and SSH reconnects only before process creation so commands are never replayed.

When the CLI is routed through the HUD gateway, the agent also exports the active trace context through Claude Code's ANTHROPIC_CUSTOM_HEADERS contract as Trace-Id: <id>. Direct Anthropic requests do not receive HUD-specific trace metadata.

Impact

Claude CLI rollouts now appear in HUD while they are running, with per-turn tool activity and file-change timing consistent with other SDK agents. Their gateway inference usage can also be attributed to the rollout trace.

Fixed streaming reproduction using the same deployed environment and task: https://hud.ai/trace/82e0cc87517d4c6495e7e074d2aeb0d2

Validation

• 21 passed in the focused Claude SDK agent and SSH client test files
• trace-linkage regression test passes
• Ruff passes on all changed files
• git diff --check passes
• deployed streaming comparison completed with reward 1.0; the fixed trace contains 12 turns and 29 timestamped spans across the live execution

---

Note

Medium Risk
Changes the Claude SDK execution and SSH streaming path and gateway header wiring, but behavior is covered by focused tests and preserves existing trace metadata on finish.

Overview
ClaudeSDKAgent no longer waits for the remote claude process to finish before HUD sees progress. It opens a streaming SSH session, parses stream-json lines as they arrive, and records assistant and tool steps with timestamps that reflect when each turn actually happened—not a single aggregate step at exit.

Assistant messages are mapped through a shared ClaudeAgent._message_to_agent_step helper so CLI streaming matches the in-process Messages API agent (usage, tool calls, citations). Tool results are paired with pending tool-use IDs; exit without a result message or unmatched tool calls is treated as an error. Cancellation closes the remote process with a bounded wait.

When the CLI uses the HUD gateway, the active rollout trace ID is passed via ANTHROPIC_CUSTOM_HEADERS (Trace-Id: …) so gateway inference logs can be attributed to the rollout trace.

SSHClient adds create_process with the same reconnect-before-open behavior as run, without replaying commands after a process has started. Tests cover streaming turn ordering, trace header forwarding, and SSH reconnect on stream open.

^{Reviewed by Cursor Bugbot for commit bd59c73. Bugbot is set up for automated code reviews on this repo. Configure here.}