Skip to content

feat(harness): add a Gemini CLI harness (tap, turn, and init templates) - #516

Open
michaelxu2288 wants to merge 4 commits into
scaleapi:mainfrom
michaelxu2288:feat/gemini-cli-harness
Open

michaelxu2288 wants to merge 4 commits into
scaleapi:mainfrom
michaelxu2288:feat/gemini-cli-harness

Conversation

@michaelxu2288

@michaelxu2288 michaelxu2288 commented Sep 10, 2026 •

Copy link
Copy Markdown

What

Adds Gemini CLI as a framework harness, mirroring the Claude Code and Codex ones:

  • convert_gemini_cli_to_agentex_events (src/agentex/lib/adk/_modules/_gemini_cli_sync.py): maps the CLI's stream-json events (init, message deltas, tool_use, tool_result, error, result; schema from packages/core/src/output/types.ts in google-gemini/gemini-cli) onto the canonical StreamTaskMessage* stream. Assistant deltas open one text slot that closes on the next tool event, the result, or end of stream, so every Start has a Done; tool requests and results pair by tool_id; a finally closes the source iterator on cancellation like the other taps.
  • GeminiCliTurn (_gemini_cli_turn.py): HarnessTurn wrapper exposing session_id and model from init and normalising result.stats into TurnUsage (tokens, cached, duration, tool calls; cost is not reported by the CLI).
  • Both exported from agentex.lib.adk.
  • agentex init templates sync-gemini-cli, default-gemini-cli, temporal-gemini-cli, registered in TemplateType, the file map and the menus. Prompt passed via -p with stdin closed (in headless mode the CLI reads stdin to EOF and appends it), optional GEMINI_MODEL, GEMINI_API_KEY credential, npm install -g @google/gemini-cli in the Dockerfile. Turns are independent prompts because the CLI's --resume takes latest/index rather than a session id; the Temporal template keeps the reported session_id for observability only and says so.

Companion docs page: scaleapi/scale-agentex#429.

Test

  • tests/lib/adk/test_gemini_cli_sync.py (12), tests/lib/adk/test_gemini_cli_turn.py (12), tests/lib/core/harness/test_harness_gemini_cli_sync.py (5): pass.
  • tests/lib/cli/test_init_templates.py: 29 passed, including the three new templates; all rendered projects byte-compile.
  • ruff check and format clean on the new files.
  • Offline fixtures follow the CLI's published event schema; I have not run a live smoke test against the gemini binary (needs a Gemini API key). Happy to add a recorded live fixture if a maintainer can share one, or run it myself once I have a key.

RetriggerConfidence Score: 4/5

The PR is not ready to merge because a failed Gemini turn can still finish as a successful turn.

Fix All in CursorFindings

  1. P1 Terminal failures appear successful ▶
Fix with agent prompt
### Issue 1
src/agentex/lib/adk/_modules/_gemini_cli_sync.py:281-291
A terminal Gemini `result` can have `status: "error"`, but this branch closes the text stream and invokes the normal result callback without checking the status or error payload. `UnifiedEmitter` therefore treats authentication, turn-limit, and other terminal failures as successful completion, potentially returning missing or partial output instead of surfacing the error. Standalone error events are also logged without entering the canonical stream.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Adds Gemini CLI support to the Agentex harness and agentex init, so developers can build Gemini CLI agents with sync, async, or Temporal delivery.

  • Converts Gemini CLI events and usage stats into Agentex harness messages.
  • Adds starter projects for all three delivery styles, including CLI setup and credentials.
Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Gemini CLI output] --> B[GeminiCliTurn]
  B --> C[Task-message events]
  C --> D[Sync response]
  C --> E[Async task stream]
  C --> F[Temporal activity]
Loading

Reviews (3) · Last reviewed commit: "test(harness): narrow content types in t..."

Comment on lines +262 to +267
elif evt_type == "result":
done = _close_text()
if done is not None:
yield done
if on_result is not None:
await on_result(evt)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Terminal failures appear successful

A terminal Gemini result can have status: "error", but this branch closes the text stream and invokes the normal result callback without checking the status or error payload. UnifiedEmitter therefore treats authentication, turn-limit, and other terminal failures as successful completion, potentially returning missing or partial output instead of surfacing the error. Standalone error events are also logged without entering the canonical stream.

Knowledge Base Used:

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agentex/lib/adk/_modules/_gemini_cli_sync.py
Line: 262-267

Comment:
**Terminal failures appear successful**

A terminal Gemini `result` can have `status: "error"`, but this branch closes the text stream and invokes the normal result callback without checking the status or error payload. `UnifiedEmitter` therefore treats authentication, turn-limit, and other terminal failures as successful completion, potentially returning missing or partial output instead of surfacing the error. Standalone error events are also logged without entering the canonical stream.

**Knowledge Base Used:**
- [Agent framework integrations](https://app.greptile.com/scale-ai/-/custom-context/knowledge-base/scaleapi/scale-agentex-python/-/docs/agent-framework-integrations.md)
- [Harness delivery and metrics](https://app.greptile.com/scale-ai/-/custom-context/knowledge-base/scaleapi/scale-agentex-python/-/docs/harness-and-metrics.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in d9b1ceb. A result with status: "error" now emits a TextContent error ("Error: Gemini CLI turn failed: ") before on_result fires, the same shape the Codex tap uses for turn.failed. Standalone error events with severity "error" are delivered the same way; warnings are still only logged. Covered by tests at the tap level and through UnifiedEmitter.yield_turn.

Comment thread src/agentex/lib/cli/templates/default-gemini-cli/project/acp.py.j2 Outdated
Adds Gemini CLI as a framework harness alongside Claude Code and Codex:

- convert_gemini_cli_to_agentex_events: maps the CLI's stream-json events
  (init, message deltas, tool_use, tool_result, error, result; schema per
  packages/core/src/output/types.ts in google-gemini/gemini-cli) onto the
  canonical StreamTaskMessage* stream. Assistant deltas open one text slot
  that closes on the next tool event, the result, or end of stream, so every
  Start has a Done; tool requests and results pair by tool_id.
- GeminiCliTurn: HarnessTurn wrapper exposing session_id and model from the
  init event and normalising result.stats into TurnUsage.
- Both exported from agentex.lib.adk.
- agentex init templates sync-gemini-cli, default-gemini-cli and
  temporal-gemini-cli (registered in TemplateType, file map and menus),
  cloned from the Claude Code templates: prompt passed via -p with stdin
  closed (the CLI reads stdin to EOF in headless mode), optional GEMINI_MODEL,
  GEMINI_API_KEY credential, npm install -g @google/gemini-cli in the
  Dockerfile. Turns are independent prompts: the CLI's --resume takes
  latest/index, not a session id.
- Tests: tap (text deltas, whole messages, tools, errors, callbacks,
  source close on cancel), turn (usage mapping, protocol), harness end to
  end through UnifiedEmitter with span derivation; template suite covers the
  three new templates.

Offline tests only; a live smoke run needs a Gemini API key.

Claude-Session: https://claude.ai/code/session_01HCVKnA7LeJZ44nxZz1uzF3
…silently

A terminal `result` with `status: "error"` (auth failure, turn limit, fatal
tool error) went straight to `on_result`, so UnifiedEmitter delivered the
turn as an ordinary completion with partial or no output. Deliver it as a
TextContent error message before firing `on_result`, the same shape the
Codex tap uses for `turn.failed`. Standalone `error` events with severity
"error" are delivered the same way; warnings are still only logged.
…ines

- Read stdout with an 8 MiB line limit. A tool_result that echoes a file is
  one stream-json line, and asyncio's 64 KiB default made readline() raise
  and abort the turn.
- After SIGTERM, wait at most 5 s for the CLI to exit, then SIGKILL, so a
  CLI that ignores SIGTERM cannot hang request or activity cancellation.
- Match the sibling templates on current main: the optional private index
  step in both Dockerfiles, and make_workflow_logger in the Temporal workflow.
- Fix "Gemini CLI CLI" wording.

Verified with a fake `gemini` that prints a 100,000-character line and
ignores SIGTERM: the line is read whole and aclose() returns after 5 s.
@michaelxu2288
michaelxu2288 force-pushed the feat/gemini-cli-harness branch from f853785 to 583856c Compare October 1, 2026 09:40
@michaelxu2288
michaelxu2288 changed the base branch from next to main October 1, 2026 09:40
pyright 1.1.399 (the locked version `scripts/lint` runs) reported six
errors in the Gemini CLI tests: attribute access on a content union after
a list-comprehension filter, which does not narrow the indexed element,
and aclose() on the tap's AsyncIterator return type. Assert the concrete
content type before reading it, and cast the tap to AsyncGenerator where
the test closes it. Test behaviour is unchanged.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant