fix(agent): retry judge schema validation with prettified errors#37
Closed
caffeinum wants to merge 1 commit into
Closed
fix(agent): retry judge schema validation with prettified errors#37caffeinum wants to merge 1 commit into
caffeinum wants to merge 1 commit into
Conversation
…d throw on exhaust Replaces the prior lenientBool default-false approach which silently masked bu-2-0's tendency to emit undefined for `is_correct` and `verdict` boolean fields. Defaulting to false hid the model bug from operators and left the orchestrator unable to distinguish "model emitted false" from "model emitted nothing". Now: when SimpleJudgeSchema or JudgeSchema fails parse, we send the prettified zod errors back to the LLM and retry up to 2 times, matching PR webllm#34's pattern for action-emission retries. If retries exhaust, surface the failure on the run's final ActionResult so harbor's failure_reason picks it up — `_run_simple_judge` marks the run as failed with a `[Judge schema invalid: ...]` note; `_judge_trace` synthesizes a verdict=false judgement with the schema error in `failure_reason`. Adds JudgeSchemaInvalidError (src/exceptions.ts) for the internal throw. A first-attempt shape check (any judge-related key present) preserves the prior graceful-skip path when the LLM returns a non-judge JSON shape entirely (e.g. an agent-step JSON in mocked tests), so we don't regress component tests that wire one mock LLM for both agent and judge calls. This pairs strict zod with feedback-driven self-correction (per reference_zod_pydantic_parity.md) instead of papering over the model bug with a default. Adds test/agent-judge-schema-retry.test.ts covering: (1) bu-2-0 missing-is_correct retry-then-recover, (2) retries exhaust → run marked failed, (3) network errors stay swallowed, (4) JudgeSchema verdict-missing exhaustion path, (5) verdict self-correction on retry. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
9d788ca to
925339d
Compare
Contributor
Author
|
Closing — reverting the silent-skip-on-shape-mismatch logic. Will reopen with cleaner version. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
bu-2-0 occasionally omits required boolean fields (
is_correct,verdict) from judge structured outputs entirely. Python upstream silently defaults them tofalsevia pydantic's lax mode; zod hard-rejects and the structured-output parse throws — and operators never learn the model misbehaved.Observed on a real eval run (codesandbox.com getting-started, 2026-05-07):
The previous attempt in this branch defaulted missing booleans to
falsevia alenientBoolhelper. That hid the model bug from operators and made it impossible to distinguish "model said false" from "model said nothing."Approach
Replace the silent default with retry-with-feedback, matching the pattern from PR #34 for action-emission retries:
SimpleJudgeSchemaorJudgeSchemafails to parse, send the prettified zod errors back to the LLM and retry up to 2 times.ActionResultso harbor'sfailure_reasonpicks it up:_run_simple_judgemarks the run as failed with a[Judge schema invalid: ...]note._judge_tracesynthesizes averdict=falsejudgement with the schema error infailure_reason.JudgeSchemaInvalidError(src/exceptions.ts) for the internal throw.This pairs strict zod with feedback-driven self-correction instead of papering over the model bug with a default.
Tests
test/agent-judge-schema-retry.test.tscovers:is_correctretry-then-recoverJudgeSchemaverdict-missing exhaustion pathverdictself-correction on retryDiff scope
3 files, src + tests only:
src/agent/service.ts(retry loop + judge schemas)src/exceptions.ts(newJudgeSchemaInvalidError)test/agent-judge-schema-retry.test.tsSupersedes the earlier closed iteration of this branch (PR #36).