feat(harness): add a Gemini CLI harness (tap, turn, and init templates) - #516
michaelxu2288 wants to merge 4 commits into
Conversation
| elif evt_type == "result": | ||
| done = _close_text() | ||
| if done is not None: | ||
| yield done | ||
| if on_result is not None: | ||
| await on_result(evt) |
There was a problem hiding this comment.
Terminal failures appear successful
A terminal Gemini result can have status: "error", but this branch closes the text stream and invokes the normal result callback without checking the status or error payload. UnifiedEmitter therefore treats authentication, turn-limit, and other terminal failures as successful completion, potentially returning missing or partial output instead of surfacing the error. Standalone error events are also logged without entering the canonical stream.
Knowledge Base Used:
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agentex/lib/adk/_modules/_gemini_cli_sync.py
Line: 262-267
Comment:
**Terminal failures appear successful**
A terminal Gemini `result` can have `status: "error"`, but this branch closes the text stream and invokes the normal result callback without checking the status or error payload. `UnifiedEmitter` therefore treats authentication, turn-limit, and other terminal failures as successful completion, potentially returning missing or partial output instead of surfacing the error. Standalone error events are also logged without entering the canonical stream.
**Knowledge Base Used:**
- [Agent framework integrations](https://app.greptile.com/scale-ai/-/custom-context/knowledge-base/scaleapi/scale-agentex-python/-/docs/agent-framework-integrations.md)
- [Harness delivery and metrics](https://app.greptile.com/scale-ai/-/custom-context/knowledge-base/scaleapi/scale-agentex-python/-/docs/harness-and-metrics.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Fixed in d9b1ceb. A result with status: "error" now emits a TextContent error ("Error: Gemini CLI turn failed: ") before on_result fires, the same shape the Codex tap uses for turn.failed. Standalone error events with severity "error" are delivered the same way; warnings are still only logged. Covered by tests at the tap level and through UnifiedEmitter.yield_turn.
bc51c52 to
761833e
Compare
Adds Gemini CLI as a framework harness alongside Claude Code and Codex: - convert_gemini_cli_to_agentex_events: maps the CLI's stream-json events (init, message deltas, tool_use, tool_result, error, result; schema per packages/core/src/output/types.ts in google-gemini/gemini-cli) onto the canonical StreamTaskMessage* stream. Assistant deltas open one text slot that closes on the next tool event, the result, or end of stream, so every Start has a Done; tool requests and results pair by tool_id. - GeminiCliTurn: HarnessTurn wrapper exposing session_id and model from the init event and normalising result.stats into TurnUsage. - Both exported from agentex.lib.adk. - agentex init templates sync-gemini-cli, default-gemini-cli and temporal-gemini-cli (registered in TemplateType, file map and menus), cloned from the Claude Code templates: prompt passed via -p with stdin closed (the CLI reads stdin to EOF in headless mode), optional GEMINI_MODEL, GEMINI_API_KEY credential, npm install -g @google/gemini-cli in the Dockerfile. Turns are independent prompts: the CLI's --resume takes latest/index, not a session id. - Tests: tap (text deltas, whole messages, tools, errors, callbacks, source close on cancel), turn (usage mapping, protocol), harness end to end through UnifiedEmitter with span derivation; template suite covers the three new templates. Offline tests only; a live smoke run needs a Gemini API key. Claude-Session: https://claude.ai/code/session_01HCVKnA7LeJZ44nxZz1uzF3
…silently A terminal `result` with `status: "error"` (auth failure, turn limit, fatal tool error) went straight to `on_result`, so UnifiedEmitter delivered the turn as an ordinary completion with partial or no output. Deliver it as a TextContent error message before firing `on_result`, the same shape the Codex tap uses for `turn.failed`. Standalone `error` events with severity "error" are delivered the same way; warnings are still only logged.
…ines - Read stdout with an 8 MiB line limit. A tool_result that echoes a file is one stream-json line, and asyncio's 64 KiB default made readline() raise and abort the turn. - After SIGTERM, wait at most 5 s for the CLI to exit, then SIGKILL, so a CLI that ignores SIGTERM cannot hang request or activity cancellation. - Match the sibling templates on current main: the optional private index step in both Dockerfiles, and make_workflow_logger in the Temporal workflow. - Fix "Gemini CLI CLI" wording. Verified with a fake `gemini` that prints a 100,000-character line and ignores SIGTERM: the line is read whole and aclose() returns after 5 s.
f853785 to
583856c
Compare
pyright 1.1.399 (the locked version `scripts/lint` runs) reported six errors in the Gemini CLI tests: attribute access on a content union after a list-comprehension filter, which does not narrow the indexed element, and aclose() on the tap's AsyncIterator return type. Assert the concrete content type before reading it, and cast the tap to AsyncGenerator where the test closes it. Test behaviour is unchanged.
What
Adds Gemini CLI as a framework harness, mirroring the Claude Code and Codex ones:
convert_gemini_cli_to_agentex_events(src/agentex/lib/adk/_modules/_gemini_cli_sync.py): maps the CLI'sstream-jsonevents (init,messagedeltas,tool_use,tool_result,error,result; schema frompackages/core/src/output/types.tsin google-gemini/gemini-cli) onto the canonicalStreamTaskMessage*stream. Assistant deltas open one text slot that closes on the next tool event, theresult, or end of stream, so everyStarthas aDone; tool requests and results pair bytool_id; afinallycloses the source iterator on cancellation like the other taps.GeminiCliTurn(_gemini_cli_turn.py):HarnessTurnwrapper exposingsession_idandmodelfrominitand normalisingresult.statsintoTurnUsage(tokens, cached, duration, tool calls; cost is not reported by the CLI).agentex.lib.adk.agentex inittemplatessync-gemini-cli,default-gemini-cli,temporal-gemini-cli, registered inTemplateType, the file map and the menus. Prompt passed via-pwith stdin closed (in headless mode the CLI reads stdin to EOF and appends it), optionalGEMINI_MODEL,GEMINI_API_KEYcredential,npm install -g @google/gemini-cliin the Dockerfile. Turns are independent prompts because the CLI's--resumetakeslatest/index rather than a session id; the Temporal template keeps the reportedsession_idfor observability only and says so.Companion docs page: scaleapi/scale-agentex#429.
Test
tests/lib/adk/test_gemini_cli_sync.py(12),tests/lib/adk/test_gemini_cli_turn.py(12),tests/lib/core/harness/test_harness_gemini_cli_sync.py(5): pass.tests/lib/cli/test_init_templates.py: 29 passed, including the three new templates; all rendered projects byte-compile.geminibinary (needs a Gemini API key). Happy to add a recorded live fixture if a maintainer can share one, or run it myself once I have a key.The PR is not ready to merge because a failed Gemini turn can still finish as a successful turn.
Fix with agent prompt
Summary
Adds Gemini CLI support to the Agentex harness and
agentex init, so developers can build Gemini CLI agents with sync, async, or Temporal delivery.Diagram
%%{init: {'theme': 'neutral'}}%% flowchart LR A[Gemini CLI output] --> B[GeminiCliTurn] B --> C[Task-message events] C --> D[Sync response] C --> E[Async task stream] C --> F[Temporal activity]Reviews (3) · Last reviewed commit: "test(harness): narrow content types in t..."