Problem
We added status, warnings, advice, and provenance to analysis results expecting it to help models make better scientific decisions. It did not help on the task it was designed for. Measuring what the server actually sends explains why.
Summing serialized bytes per top-level key across 120 enriched analysis runs:
| Key |
Share of payload |
mcp_server_tools |
25.9% |
variables |
24.2% |
_provenance |
16.2% |
recommended_next_steps |
13.8% |
uxarray_capabilities |
8.1% |
recommendations |
3.5% |
scientific_status |
1.9% |
postconditions |
1.8% |
| everything else, including the computed numbers |
~4.6% |
Grouped: 74.1% is a catalog of other tools and capabilities, 16.2% is a provenance record, 7.5% is status and warnings, and 2.2% is the actual result.
On a vector-suitability task, the field that settles the scientific question — scientific_status — was 100 bytes inside a 2,172-byte reply, or 4.6%.
Effect
The enriched results roughly doubled that task's median token use, from ~14.0k to ~28.6k, and changed the outcome not at all: 10/20 correct with and without. Across all tasks, enrichment added a paired median of +2,078 tokens per run and increased demand in 111 of 120 matched cells, with no measurable improvement.
The conclusion is not that scientific evidence is useless. It is that a decisive signal at 2% of the payload, surrounded by a tool catalog, does not change behavior.
Where it comes from
src/uxarray_mcp/tools/capabilities.py builds mcp_server_tools, uxarray_capabilities, recommended_next_steps, and endpoint_profiles. These are useful once, during discovery. They are attached to results a model has already decided to request, and then re-sent on every subsequent turn because the conversation carries them forward.
Suggested direction
Separate discovery from results. A result should describe that result.
- Keep
get_capabilities as the discovery call and let it be verbose; it is called once.
- Remove
mcp_server_tools, uxarray_capabilities, recommended_next_steps, and endpoint_profiles from analysis results entirely.
- Consider putting
_provenance behind a handle. At 16% it is rarely what the model needs next, but it matters for the scientific record, so this needs thought rather than deletion.
- Keep
variables only when the operation is an inspection.
A rough target: computed numbers plus status should be the majority of a result payload, not 4%.
Problem
We added status, warnings, advice, and provenance to analysis results expecting it to help models make better scientific decisions. It did not help on the task it was designed for. Measuring what the server actually sends explains why.
Summing serialized bytes per top-level key across 120 enriched analysis runs:
mcp_server_toolsvariables_provenancerecommended_next_stepsuxarray_capabilitiesrecommendationsscientific_statuspostconditionsGrouped: 74.1% is a catalog of other tools and capabilities, 16.2% is a provenance record, 7.5% is status and warnings, and 2.2% is the actual result.
On a vector-suitability task, the field that settles the scientific question —
scientific_status— was 100 bytes inside a 2,172-byte reply, or 4.6%.Effect
The enriched results roughly doubled that task's median token use, from ~14.0k to ~28.6k, and changed the outcome not at all: 10/20 correct with and without. Across all tasks, enrichment added a paired median of +2,078 tokens per run and increased demand in 111 of 120 matched cells, with no measurable improvement.
The conclusion is not that scientific evidence is useless. It is that a decisive signal at 2% of the payload, surrounded by a tool catalog, does not change behavior.
Where it comes from
src/uxarray_mcp/tools/capabilities.pybuildsmcp_server_tools,uxarray_capabilities,recommended_next_steps, andendpoint_profiles. These are useful once, during discovery. They are attached to results a model has already decided to request, and then re-sent on every subsequent turn because the conversation carries them forward.Suggested direction
Separate discovery from results. A result should describe that result.
get_capabilitiesas the discovery call and let it be verbose; it is called once.mcp_server_tools,uxarray_capabilities,recommended_next_steps, andendpoint_profilesfrom analysis results entirely._provenancebehind a handle. At 16% it is rarely what the model needs next, but it matters for the scientific record, so this needs thought rather than deletion.variablesonly when the operation is an inspection.A rough target: computed numbers plus status should be the majority of a result payload, not 4%.