Skip to content

docs(examples): add runnable structured output example - #4418

Merged
VascoSch92 merged 3 commits into
OpenHands:mainfrom
luciobaiocchi:docs/structured-output-example
Aug 11, 2026
Merged

docs(examples): add runnable structured output example#4418
VascoSch92 merged 3 commits into
OpenHands:mainfrom
luciobaiocchi:docs/structured-output-example

Conversation

@luciobaiocchi

@luciobaiocchi luciobaiocchi commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

HUMAN:
Added example code for #4207

Referenced by the structured output guide in OpenHands/docs#695.


AGENT:

Why

#4207 landed response_schema but shipped without a runnable example — the one
written for it was deferred out of the PR to unblock the docs-example CI gate.
The docs guide for the feature (OpenHands/docs#695) needs it: the other SDK
guides all embed a synced examples/... block, and @VascoSch92 asked for the
same treatment there.

Summary

Adds examples/01_standalone_sdk/56_structured_output.py, restored from the
deferred version and updated for the merged API. It demonstrates both halves of
the feature on one run:

  • TerminalTool with a schema — every command must carry a purpose and
    an expected_outcome on top of the tool's own command field, showing that
    a schema can force per-call annotations on an existing tool with no
    subclassing.
  • FinishTool with a schema — the final answer comes back as a typed
    ProjectFacts object, read via parse_last_response().

It also shows reading a specific call with parse_response(event.action) while
walking the event stream.

Issue Number

Relates to #2566, follows up #4207. Docs counterpart: OpenHands/docs#695

How to Test

export LLM_MODEL="gemini/gemini-2.5-flash"   # or any model you have a key for
export LLM_API_KEY="..."
uv run python examples/01_standalone_sdk/56_structured_output.py

The run should print a rationale for each terminal command and a typed final
answer. ruff check / ruff format are clean and the file's imports and API
calls were verified against current main.

Video/Screenshots

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

The example is numbered 56_ (55 was the highest in use). The docs guide
already references this path; sync_code_blocks.py only warns when the file is
missing, so OpenHands/docs#695 is not blocked by merge order — once this lands,
the sync workflow fills in the code block.

Restores the example deferred out of OpenHands#4207, updated for the merged API.
Shows a response_schema on TerminalTool (forced per-command rationale)
and on FinishTool (typed final answer), and reads results back with
parse_response / parse_last_response.

Referenced by the structured output guide in OpenHands/docs#695.

Co-authored-by: Vasco Schiavo <115561717+VascoSch92@users.noreply.github.com>
@luciobaiocchi
luciobaiocchi marked this pull request as ready for review August 7, 2026 15:27
@all-hands-bot

Copy link
Copy Markdown
Collaborator

🚦 CI is currently failing on this PR's latest commit.

Please fix the failing checks before OpenHands reviews it - this is re-checked automatically once you push a new commit. (A maintainer can also request @all-hands-bot as a reviewer to have it reviewed regardless of CI status.)

This is an automated check - no AI was used to generate this comment.

@all-hands-bot

Copy link
Copy Markdown
Collaborator

🤖 OpenHands is reviewing this PR.

Head commit: b46f97ff2718953179b5c7856ed4f49e4c9f411b
View the conversation: https://oss-agent-canvas.ngrok.dev/conversations/596183d5-9b89-46d9-889b-7ae9242526e5

This comment was posted by an AI agent (OpenHands).

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was created by an AI agent (OpenHands) on behalf of the repository maintainers.

Summary

This PR adds examples/01_standalone_sdk/56_structured_output.py, a runnable example demonstrating the response_schema structured-output feature landed in #4207. The example shows both augmenting an existing tool (TerminalTool) with forced-annotation fields and binding a typed ProjectFacts model to FinishTool for the final answer.

Verification

I verified every API call in the example against the current main codebase and ran end-to-end checks:

  • register_tool("FinishTool", FinishTool) — Correct and necessary. FinishTool is in BUILT_IN_TOOL_CLASSES (used by include_default_tools) but is not registered in the tool resolver registry (_REG) by default, so it must be registered explicitly to resolve Tool(name="FinishTool", ...) via resolve_tool. This matches the pattern in tests/sdk/tool/test_response_schema.py.
  • Tool(name=TerminalTool.name, params={"response_schema": CommandRationale}) — Correct. CommandRationale fields (purpose, expected_outcome) do not collide with TerminalAction fields (command, is_input, timeout, reset) and are not in _RESERVED_RESPONSE_FIELDS. Verified resolution and parse_response() round-trip.
  • include_default_tools=["ThinkTool"] — Correct. This skips the auto-injected default FinishTool so the schema-bound one from tools= is used instead. Verified tools_map contains both finish (with schema) and think.
  • agent.tools_map["finish"] — Correct. The resolved tool's .name attribute is "finish" (from _camel_to_snake("FinishTool").removesuffix("_tool")), not "FinishTool".
  • finish_tool.parse_last_response(events) and terminal_tool.parse_response(event.action) — Both verified end-to-end with synthetic events. parse_last_response correctly walks reversed events, re-splits arguments, and returns a typed ProjectFacts.
  • conversation.conversation_stats.get_combined_metrics().accumulated_cost — Matches the pattern in other examples (e.g. 46_agent_settings.py).
  • EXAMPLE_COST: marker — Present, satisfying the example CI gate.
  • ruff check / ruff format — Clean.

Example CI Discovery

The file lives under examples/01_standalone_sdk/ (in _TARGET_DIRECTORIES), is not in _EXCLUDED_EXAMPLES, and will be discovered and run by tests/examples/test_examples.py.

Risk Assessment

Low risk. This is a documentation-only change (new example file, no modifications to SDK, tools, or tests). All API usage has been verified against the current codebase. No security concerns — the example reads credentials from environment variables and performs no privileged operations.

No Material Findings

No correctness, security, or maintainability issues were found. The example is well-structured, follows the flat script-like pattern used by other examples, and accurately documents the reserved-field constraints in its inline comment.

@VascoSch92

VascoSch92 commented Aug 10, 2026

Copy link
Copy Markdown
Member

@luciobaiocchi did you run the example? Could you post the output?

@luciobaiocchi

Copy link
Copy Markdown
Contributor Author

Yes I ran it with gemini-2.5-flash:

[Terminal commands with rationale]
  $ ls -F
    purpose:          List files in the current directory to understand the project structure.
    expected_outcome: A list of files and directories will be displayed.
  $ ls -F openhands-sdk/
    purpose:          List files in the openhands-sdk directory.
    expected_outcome: A list of files and directories within the openhands-sdk directory.
  $ ls -F openhands-sdk/openhands/
    purpose:          List files in the openhands-sdk/openhands directory.
    expected_outcome: A list of files and directories within the openhands-sdk/openhands directory.
  $ ls -F openhands-sdk/openhands/sdk/
    purpose:          List files in the openhands-sdk/openhands/sdk directory.
    expected_outcome: A list of files and directories within the openhands-sdk/openhands/sdk directory.

[Finish]
  description: The OpenHands Software Agent SDK is a set of Python and REST APIs for building agents that work with code, supporting various tasks from one-off actions to major refactors, and enabling both local and ephemeral workspace execution.
  - The OpenHands Software Agent SDK provides Python and REST APIs for building AI agents.
  - It can be used for tasks like building READMEs, updating dependencies, and performing major code refactors.
  - Agents can utilize either the local machine or ephemeral workspaces (like Docker or Kubernetes) via the Agent Server.

EXAMPLE_COST: 0.01450458

AGENT COMMENT:

Every terminal call carries the annotation fields, and the final answer comes back as a typed ProjectFacts.

One honest note from the run: mid-way the model also tried to put purpose / expected_outcome on file_editor, which has no schema attached, and validation rejected it (Extra inputs are not permitted) before the agent recovered on the next turn. Correct behaviour — but worth knowing that a schema on one tool can bleed into calls to other tools and cost a retry.

@VascoSch92
VascoSch92 enabled auto-merge (squash) August 10, 2026 20:39
@all-hands-bot

Copy link
Copy Markdown
Collaborator

🤖 OpenHands is reviewing this PR.

Head commit: c6ee47710990d5355e459f36f55bb8d7c5432401
View the conversation: https://oss-agent-canvas.ngrok.dev/conversations/a5cd6c4e-62b1-453a-8b6b-aa42a5ff878f

This comment was posted by an AI agent (OpenHands).

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was created by an AI agent (OpenHands) on behalf of the repository maintainers.

Summary

This PR adds examples/01_standalone_sdk/56_structured_output.py, a runnable example demonstrating the response_schema structured-output feature from #4207. The example demonstrates two patterns: (1) augmenting TerminalTool with forced per-call annotation fields (purpose, expected_outcome) via a CommandRationale schema, and (2) binding a typed ProjectFacts model to FinishTool for the final answer, recovered via parse_last_response().

The head SHA (c6ee477) is a merge of main into the branch since the last review (b46f97f); the example file itself is unchanged, but I re-verified all API usage against the merged codebase.

Verification

I verified every API call against the current workspace (head SHA c6ee477) and ran all pre-commit hooks:

  • register_tool("FinishTool", FinishTool) (line 48) — Correct and necessary. FinishTool is listed in BUILT_IN_TOOL_CLASSES (used by include_default_tools) but is not auto-registered in the tool resolver registry (_REG). Confirmed via list_registered_tools() returning empty before the call. The registration name "FinishTool" matches the Tool(name="FinishTool", ...) spec on line 66.
  • Tool(name=TerminalTool.name, params={"response_schema": CommandRationale}) (line 64) — Correct. CommandRationale fields (purpose, expected_outcome) do not collide with TerminalAction fields (command, is_input, timeout, reset) and are not in _RESERVED_RESPONSE_FIELDS (kind, security_risk, structured_output, summary).
  • include_default_tools=["ThinkTool"] (line 69) — Correct and necessary. This skips the auto-injected default FinishTool so the schema-bound one from tools= is the sole finish tool, avoiding a duplicate-name error. The comment on line 68 explains this.
  • agent.tools_map["finish"] (line 84) — Correct. The resolved FinishTool instance has .name == "finish" (from _camel_to_snake("FinishTool").removesuffix("_tool")), not "FinishTool".
  • agent.tools_map[TerminalTool.name] (line 83) — Correct. TerminalTool.name == "terminal".
  • terminal_tool.parse_response(event.action) (line 96) and finish_tool.parse_last_response(events) (line 103) — Both match the method signatures in tool.py (lines 558, 580). parse_last_response walks reversed events, re-splits arguments via _split_response_arguments, and returns a typed ProjectFacts | None.
  • conversation.conversation_stats.get_combined_metrics().accumulated_cost (line 111) — Matches the pattern used across 20+ existing examples.
  • EXAMPLE_COST: marker (line 112) — Present, satisfying the example CI gate.

Pre-commit hooks all pass: ruff check, ruff format, pycodestyle, pyright, check-import-rules, check-tool-registration.

Example CI Discovery

The file lives under examples/01_standalone_sdk/ (already in _TARGET_DIRECTORIES in tests/examples/test_examples.py), is not in _EXCLUDED_EXAMPLES, and will be auto-discovered via sorted(directory.glob("*.py")). No changes to the test file are needed.

Risk Assessment

Low risk. Documentation-only change (new example file, no modifications to SDK, tools, or tests). No security concerns — the example reads credentials from environment variables and performs no privileged operations. All API usage verified against the merged main codebase.

No Material Findings

No correctness, security, or maintainability issues were found. The example is well-structured, follows the top-level code-flow convention used by other linear walkthrough examples, accurately documents the reserved-field constraints in its inline comment, and correctly demonstrates both halves of the response_schema feature.

@VascoSch92
VascoSch92 merged commit 6c3b687 into OpenHands:main Aug 11, 2026
33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants