Problem
The server describes what a tool does in prose, but never declares the shape of the result it expects a caller to produce. When a caller returns the right science in the wrong envelope, nothing distinguishes that from getting the science wrong.
This is not hypothetical. In a 480-run evaluation of this server, 119 runs had to be re-scored by hand because the automated scorer could not tell format failures from scientific ones. Of seven disputed runs on the verification task, all seven failed only on JSON formatting — five of them were a single model emitting bare JSON instead of a fenced block. The underlying numbers were right.
Any consumer of this server hits the same wall: a strict parser rejects correct work, and a lenient one accepts sloppy work.
Suggested direction
Let an operation declare its expected result shape as data the caller can validate against, rather than as description:
- a machine-readable schema for the fields the caller is expected to return (names, types, which are required)
- ideally served alongside the result, so a caller does not have to have read the docstring
- optionally, a server-side validation helper so a caller can check its own output before committing to it
The point is to make "the answer is right but the wrapper is wrong" a detectable and separately reportable state.
Relationship to other issues
Problem
The server describes what a tool does in prose, but never declares the shape of the result it expects a caller to produce. When a caller returns the right science in the wrong envelope, nothing distinguishes that from getting the science wrong.
This is not hypothetical. In a 480-run evaluation of this server, 119 runs had to be re-scored by hand because the automated scorer could not tell format failures from scientific ones. Of seven disputed runs on the verification task, all seven failed only on JSON formatting — five of them were a single model emitting bare JSON instead of a fenced block. The underlying numbers were right.
Any consumer of this server hits the same wall: a strict parser rejects correct work, and a lenient one accepts sloppy work.
Suggested direction
Let an operation declare its expected result shape as data the caller can validate against, rather than as description:
The point is to make "the answer is right but the wrapper is wrong" a detectable and separately reportable state.
Relationship to other issues
run_analysisis a single tool with 38 parameters and 32 operations #89 are about shrinking what we send. A declared schema should be small and should not become another payload passenger; it likely belongs behind a separate call rather than in every reply.