fix(backends): strict-ify OpenAI response schemas for OpenAI-compatible providers

[abhiojha8]

Pull Request

Issue

Fixes #1491

Description

OpenAI's structured outputs require additionalProperties: false on every object and reject $ref entries (schemas must be self-contained). Mellea only patched the top level of response_format schemas, and only when base_url was exactly api.openai.com. Any @generative call routed through an OpenAI-compatible proxy (e.g. OpenRouter → OpenAI model) failed with 400 invalid_json_schema — and because pydantic emits $defs/$ref for nested models, nested return types would fail on real api.openai.com too.

This PR removes the server-type gate and applies the fix universally for OpenAIBackend:

• New _make_response_schema_openai_strict() inlines $refs (reusing _recursively_inline_refs from mellea/backends/tools.py) and recursively sets additionalProperties: false on every object.
• Applied to both the chat response_format path and the raw completions path (structured_outputs/guided_json for vLLM-style backends), per maintainer guidance on @generative with OpenAI-compatible proxies (e.g. OpenRouter) fails with 400: response_format schema missing additionalProperties: false #1491.
• Removes the now-misleading "Mellea assumes you are NOT using the OpenAI platform…" info log (the assumption no longer holds).

Verified empirically (see #1491): top-level-only patching is insufficient; OpenAI-platform endpoints require inlined + fully-patched schemas.

Testing

• Tests added to the respective file if code was changed
• New code has 100% coverage if code was added
• Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

New tests in test/backends/test_openai_unit.py:

• helper unit tests: nested model ($ref inlined, all objects patched), list-of-model items, anyOf branches
• mocked-payload tests: chat response_format and raw guided_json / structured_outputs payloads reaching the provider are inlined + fully patched (covers the structured_outputs branch via _use_structured_output_for_raw)

Local verification:

• uv run pytest test/backends/test_openai_unit.py → 38 passed
• Broader backend suite (openai unit, pydantic tool params, discriminated unions, schema helpers, server_type, openai_ollama) → 134 passed / 13 skipped
• Live repro via OpenRouter → OpenAI (gpt-5.6-luna-pro) now returns typed output (previously 400); Google + Anthropic providers regression-checked, still working
• ruff format/ruff check, mypy, pre-commit (SPDX, ruff, mypy, codespell, DCO) all clean

Not run locally (no Ollama/vLLM on this machine): the full -m "not qualitative" suite and a real vLLM backend — happy to have CI and/or maintainers run those (as offered on #1491).

Attribution

• AI coding assistants used
  AI-assisted (Reasonix harness; commit includes Assisted-by: Reasonix trailer). The complete diff was reviewed and approved by the human author before submission.

---

Adding a new component, requirement, sampling strategy, or tool?

If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.

• Component
• Requirement
• Sampling Strategy
• Tool

[AngeloDanducci]

Will let @jakelorocco weigh in since he was involve in the original issue.

Thanks for the contribution! One small note from me.

[AngeloDanducci]

I think we should only set to false when it's not already a schema dict. Should still solve the original problem.

Suggested change

	def _patch_object(obj: dict[str, Any]) -> None:
	if obj.get("type") == "object":
	obj["additionalProperties"] = False
	props = obj.get("properties")
	if isinstance(props, dict):
	for prop_schema in props.values():
	if isinstance(prop_schema, dict):
	_patch_object(prop_schema)
	items = obj.get("items")
	if isinstance(items, dict):
	_patch_object(items)
	for key in ("anyOf", "oneOf", "allOf"):
	for branch in obj.get(key, []):
	if isinstance(branch, dict):
	_patch_object(branch)
	def _patch_object(obj: dict[str, Any]) -> None:
	add_props = obj.get("additionalProperties")
	if obj.get("type") == "object" and not isinstance(add_props, dict):
	# Only assert a closed object when the value schema isn't itself a
	# constraint (e.g. dict[str, Model] emits additionalProperties as a
	# schema — overwriting it with False would drop the value type).
	obj["additionalProperties"] = False
	elif isinstance(add_props, dict):
	_patch_object(add_props)
	props = obj.get("properties")
	if isinstance(props, dict):
	for prop_schema in props.values():
	if isinstance(prop_schema, dict):
	_patch_object(prop_schema)
	items = obj.get("items")
	if isinstance(items, dict):
	_patch_object(items)
	for key in ("anyOf", "oneOf", "allOf"):
	for branch in obj.get(key, []):
	if isinstance(branch, dict):
	_patch_object(branch)

[jakelorocco]

@abhiojha8, thanks for opening the PR. I tested the change against a vllm hosted model and saw no issues with our existing tests.

I think these changes look good, but I agree with @AngeloDanducci's comments and commented some tests we could add to ensure we don't hit regressions here.

Please let us know if you agree / disagree with the suggestions!

[jakelorocco]

We should add tests to capture the scenario @AngeloDanducci commented on:

class _ExtractUserMapResponse(BaseModel):
    """Wrapper with a dict-of-model field.

    Pydantic emits the value type as `additionalProperties: {schema}` (not
    `false`) on the map object, which is the case that must not be clobbered.
    """

    users: dict[str, _NestedProfile]


def test_make_response_schema_openai_strict_preserves_dict_value_schema():
    """A `dict[str, Model]` value-type schema is preserved, not overwritten.

    Regression guard for the `additionalProperties`-as-schema case: pydantic
    emits `additionalProperties: {<value schema>}` for `dict[str, Model]`
    fields. Overwriting that with `False` would silently drop the value type
    (turning "string keys -> Model" into "no extra properties allowed"), so
    the patcher must recurse into the value schema instead of clobbering it.
    """
    schema = _make_response_schema_openai_strict(
        _ExtractUserMapResponse.model_json_schema()
    )

    users = schema["properties"]["users"]
    assert users["type"] == "object"

    # The value-type schema must survive as a dict, NOT be replaced by False.
    value_schema = users["additionalProperties"]
    assert isinstance(value_schema, dict), (
        "dict[str, Model] value type was clobbered by additionalProperties=False"
    )

    # The nested model reachable through the map is inlined and closed.
    assert "$ref" not in value_schema
    assert value_schema["type"] == "object"
    assert value_schema["additionalProperties"] is False
    assert set(value_schema["properties"]) == {"name", "age"}


def test_make_response_schema_openai_strict_patches_freeform_dict_scalar_values():
    """A `dict[str, scalar]` value-type schema (e.g. int) is preserved too."""

    class _Counts(BaseModel):
        counts: dict[str, int]

    schema = _make_response_schema_openai_strict(_Counts.model_json_schema())

    counts = schema["properties"]["counts"]
    assert counts["type"] == "object"
    # Scalar value type stays intact instead of being turned into False.
    assert counts["additionalProperties"] == {"type": "integer"}

[nrfulton]

FYI @HendrikStrobelt. Can you try testing on this branch and see if it addresses your issue?

[abhiojha8]

Thanks for the review — and thanks for testing against a real vLLM backend, glad the existing tests passed there.

I agree with the additionalProperties-as-schema point: overwriting the value-type schema for dict[str, Model] with False would silently drop the value type (turning "string keys -> Model" into "no extra properties allowed"). I've pushed a follow-up commit (3ceb30c) that:

• applies exactly the suggested _patch_object change — only close an object when additionalProperties isn't itself a schema dict; recurse into it when it is
• adds the two regression tests from your comment: dict[str, Model] value schema preserved (nested model inlined + closed) and dict[str, int] scalar value type preserved

Local verification: test_openai_unit.py now 40 passed; ruff/mypy/pre-commit clean; new code fully covered. Happy to adjust if anything else comes up in review.