fix(agent): restore structured-output self-healing retry in fallback path by zhangliyuangit · Pull Request #2047 · agentscope-ai/agentscope-java

zhangliyuangit · 2026-07-08T06:39:28Z

Problem

On the tool-based (fallback) structured-output path, 2.0 relies entirely on the model volunteering the generate_response call. When the model answers with plain text instead — which happens routinely with longer contexts and busier toolkits — the raw text message is returned as-is: hasStructuredData() is false and getStructuredData() throws IllegalStateException: No structured output in message metadata.

This is a regression from 1.x, whose built-in StructuredOutputHook guaranteed the call: on a no-tool-call round it injected a reminder message and retried the reasoning step with tool_choice forced to the synthetic tool (up to 3 attempts).

The enforcement loop was dropped in the 2.0 rewrite while its supporting infrastructure survived unused:

StructuredOutputReminder still documents TOOL_CHOICE as "the default and recommended option" — but nothing consumes it;
MessageMetadataKeys.STRUCTURED_OUTPUT_REMINDER / STRUCTURED_OUTPUT_REMINDER_TYPE still exist;
compressStructuredOutputContext / isStructuredOutputRelated still clean reminder messages out of the final context — but nothing ever injected them.

The v2 docs (building-blocks/agent.md, Structured Output section) promise that tools and structured output compose transparently on either path, which the current implementation cannot guarantee.

Fixes #641 (model emitted <tool_call> pseudo-markup / fenced JSON as plain text → IllegalStateException)
Refs #1103

Fix

Restore the 1.x semantics inside the ReAct loop (ReActAgent):

Retry on silent rounds — when a fallback structured-output round (soTool != null) finishes with no tool call at all, append a TOOL_CHOICE reminder message (tagged with the existing reminder metadata, so context compression removes it as before) and re-enter reasoning(iter + 1, true), up to STRUCTURED_OUTPUT_MAX_RETRIES (3, same as 1.x).
Force tool_choice on the retry round — when the last input message is a TOOL_CHOICE reminder, merge ToolChoice.Specific("generate_response") into the round's GenerateOptions, enforcing the call at the API level rather than by prompt alone (port of 1.x handlePreReasoning).
Unchanged give-up behaviour — after exhausting retries the plain-text message is returned unchanged (parity with both current behaviour and 1.x).

Out of scope, unchanged by design:

the native response_format path (soTool == null there);
rounds that call business tools — those are normal ReAct steps and never trigger the reminder;
StructuredOutputReminder.PROMPT mode remains unwired (this PR implements the documented default, TOOL_CHOICE).

Tests

New ReActAgentStructuredOutputRetryTest (TDD — both regression tests fail on main):

plain-text round → reminder injected → retry round forces tool_choice → structured data present;
model never complies → gives up after exactly 1 + 3 model calls, returns text message without structured data;
voluntary generate_response on round 1 → single model call, no retry (guards existing behaviour).

Full agentscope-core suite: 2206 tests, 0 failures (9 pre-existing skips). Spotless clean.

🤖 Generated with Claude Code

…path The 1.x StructuredOutputHook guaranteed structured output on the tool-based path: when the model answered with plain text instead of calling generate_response, it injected a reminder message and retried the reasoning step with tool_choice forced to the synthetic tool (up to 3 attempts). This enforcement loop was dropped in the 2.0 rewrite while its supporting infrastructure survived unused: StructuredOutputReminder still documents TOOL_CHOICE as "the default and recommended option", MessageMetadataKeys.STRUCTURED_OUTPUT_REMINDER still exists, and compressStructuredOutputContext still cleans reminder messages up - but nothing ever injected them. As a result, doFallbackStructuredCall relies entirely on the model volunteering the generate_response call. When it answers in plain text the raw message is returned as-is, hasStructuredData() is false, and getStructuredData() throws IllegalStateException (see agentscope-ai#641, agentscope-ai#1103). Restore the 1.x semantics inside the ReAct loop: - When a fallback structured-output round finishes without any tool call, append a TOOL_CHOICE reminder message (tagged with the existing reminder metadata so context compression removes it) and re-enter reasoning, up to STRUCTURED_OUTPUT_MAX_RETRIES (3) times. - On the retry round (last input message is a TOOL_CHOICE reminder), force tool_choice to generate_response via GenerateOptions, so the constraint is enforced at the API level rather than by prompt alone. - After exhausting retries the plain-text message is returned unchanged, matching both the current give-up behaviour and 1.x. The native response_format path is unaffected (soTool == null there), and rounds that call business tools are normal ReAct steps and never trigger the reminder. Fixes agentscope-ai#641 Refs agentscope-ai#1103 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot

Pull request overview

This PR restores the 1.x “self-healing” behavior for fallback structured output in ReActAgent: when the model ends a structured-output call without invoking the synthetic generate_response tool, the agent injects a reminder and retries (up to 3 times), forcing tool_choice on the retry round to guarantee the tool call at the API level.

Changes:

Add a retry loop in the ReAct post-reasoning finish path to re-enter reasoning after injecting a structured-output reminder (bounded by STRUCTURED_OUTPUT_MAX_RETRIES).
Force GenerateOptions.toolChoice = Specific("generate_response") when the retry round is identified as a TOOL_CHOICE reminder round.
Add ReActAgentStructuredOutputRetryTest covering success-after-retry, give-up-after-max-retries, and no-retry-when-voluntary.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
agentscope-core/src/main/java/io/agentscope/core/ReActAgent.java	Implements structured-output fallback retries + forced `tool_choice`, plus helper reminder detection/building.
agentscope-core/src/test/java/io/agentscope/core/agent/ReActAgentStructuredOutputRetryTest.java	Regression tests verifying the retry loop and max-retry behavior.

zhangliyuangit · 2026-07-08T07:05:02Z

+                                        state.contextMutable()
+                                                .add(createStructuredOutputReminder());
+                                        return reasoning(iter + 1, true);


Intentional, keeping as-is. Retaining the plain-text attempt in the context is what lets the forced retry round restate the same content through the tool call — 1.x behaved identically (its compressMemory used the same isStructuredOutputRelated criteria and also kept the text attempt), and production ports of the 1.x hook rely on this echo for verbatim consistency between the text round and the tool round. Removing it would change 1.x parity semantics; happy to revisit if maintainers prefer.

zhangliyuangit · 2026-07-08T07:05:04Z

+                                                .add(createStructuredOutputReminder());
+                                        return reasoning(iter + 1, true);
+                                    }
                                    return Mono.justOrEmpty(eventMsg);


Fixed in e5d2b4a. The give-up path now strips reminder-tagged messages from the persisted context before returning, and the regression test asserts a follow-up turn's model input contains no leftover reminders. (Note this is stricter than 1.x, which also leaked reminders on give-up — but the cleanup is clearly right.)

zhangliyuangit · 2026-07-08T07:05:24Z

+            return Msg.builder()
+                    .name("system")
+                    .role(MsgRole.USER)
+                    .content(


Keeping as-is: this is a verbatim port of the 1.x StructuredOutputHook.createReminderMessage (name("system") + role(USER)), and teams migrating from 1.x diff old/new logs against each other — changing the name would break that parity for no functional gain. Provider formatters key off role, which is USER here as before. Can change in a follow-up if maintainers prefer consistency over parity.

zhangliyuangit · 2026-07-08T07:05:26Z

+            return metadata != null
+                    && StructuredOutputReminder.TOOL_CHOICE
+                            .toString()
+                            .equals(
+                                    metadata.get(
+                                            MessageMetadataKeys.STRUCTURED_OUTPUT_REMINDER_TYPE));


Fixed in e5d2b4a — isToolChoiceReminder now requires STRUCTURED_OUTPUT_REMINDER == true in addition to the TYPE key. (1.x checked only the TYPE key, but the extra guard is strictly safer.)

…der check - Give-up path now removes the injected structured-output reminder messages from the persisted context, so exhausted retries don't pollute subsequent turns (they were only cleaned on success). - isToolChoiceReminder now also requires STRUCTURED_OUTPUT_REMINDER == true in addition to the TYPE key, so unrelated metadata can't accidentally force tool_choice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

oss-maintainer · 2026-07-11T00:33:00Z

⚠️ Merge conflict detected

This PR has conflicts with the main branch and cannot be merged. Please rebase or merge main into your branch and resolve the conflicts:

git fetch origin
git checkout fix/structured-output-fallback-retry
git rebase origin/main
# resolve conflicts, then:
git push --force-with-lease

This is a one-time reminder. Feel free to @mention me for a re-review after conflicts are resolved.

Automated notification by github-manager-bot

zhangliyuangit requested review from a team and Copilot July 8, 2026 06:39

Copilot started reviewing on behalf of zhangliyuangit July 8, 2026 06:41 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(agent): restore structured-output self-healing retry in fallback path#2047

fix(agent): restore structured-output self-healing retry in fallback path#2047
zhangliyuangit wants to merge 2 commits into
agentscope-ai:mainfrom
zhangliyuangit:fix/structured-output-fallback-retry

zhangliyuangit commented Jul 8, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

zhangliyuangit Jul 8, 2026

Uh oh!

zhangliyuangit Jul 8, 2026

Uh oh!

zhangliyuangit Jul 8, 2026

Uh oh!

zhangliyuangit Jul 8, 2026

Uh oh!

oss-maintainer commented Jul 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

zhangliyuangit commented Jul 8, 2026

Problem

Fix

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

zhangliyuangit Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

zhangliyuangit Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

zhangliyuangit Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

zhangliyuangit Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

oss-maintainer commented Jul 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants