Skip to content

fix: validate structured output across providers#44

Merged
ebarti merged 1 commit into
mainfrom
agent/stack-04-provider-output-validation
Jul 10, 2026
Merged

fix: validate structured output across providers#44
ebarti merged 1 commit into
mainfrom
agent/stack-04-provider-output-validation

Conversation

@ebarti

@ebarti ebarti commented Jul 10, 2026

Copy link
Copy Markdown
Owner

What

  • validate Claude, Codex, and Antigravity results against output_schema locally
  • preserve valid textual JSON null
  • revalidate shallow-frozen schemas at dispatch
  • return failed results with JSON-path mismatch details
  • apply the same contract to FakeSDKRuntime
  • preserve usage, session, and tool-audit data on validation failure

Why

Vendor schema requests are not proof that a returned value conforms. Each adapter previously accepted any parseable JSON as success.

Root cause

Provider translators stopped after native output or JSON parsing and never applied the requested schema locally.

Checks

  • 293 passed, 12 skipped
  • ruff check src tests
  • mypy
  • uv lock --check
  • wheel and sdist build

Stack

@ebarti ebarti force-pushed the agent/stack-03-strict-structured-output branch from b30b75e to 809ae33 Compare July 10, 2026 22:42
Base automatically changed from agent/stack-03-strict-structured-output to main July 10, 2026 22:43
@ebarti ebarti force-pushed the agent/stack-04-provider-output-validation branch from e43a071 to 54a7f40 Compare July 10, 2026 22:43
@ebarti ebarti marked this pull request as ready for review July 10, 2026 22:43
@ebarti ebarti merged commit 5ef007f into main Jul 10, 2026
11 checks passed
@ebarti ebarti deleted the agent/stack-04-provider-output-validation branch July 10, 2026 22:44
ebarti added a commit that referenced this pull request Jul 10, 2026
## What

- reject blank task, session, MCP, tool, artifact, and result identities
- require positive execution hints and finite non-negative
budgets/counts
- reject ambiguous session start/resume state, duplicate MCP names, and
conflicting tool filters
- canonicalize sequence value fields while rejecting scalar strings
- add AgentResult.is_success
- represent unknown Usage and cost as None while preserving reported
zero
- preserve partial provider telemetry without inventing cache counts
- fix Claude resume result fallback when the SDK omits its session id

## Why

Several invalid or ambiguous states were constructible, and missing
telemetry was indistinguishable from a real zero.

## Root cause

The frozen dataclasses provided immutability but did not own their
cross-field invariants, while adapter coercion helpers normalized
missing and malformed counters to zero.

## Checks

- 321 passed, 12 skipped
- independent domain audit completed
- ruff check src tests
- mypy
- uv lock --check
- wheel and sdist build

## Compatibility

This intentionally rejects previously accepted invalid constructor
inputs and changes unknown Usage fields from zero to None.

## Stack

- Base: #44
- Next: #46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant