feat: Restruccturing Simulation eval and Scenario docs according to the latest code#630
feat: Restruccturing Simulation eval and Scenario docs according to the latest code#630sarthakFuture wants to merge 3 commits into
Conversation
…t, Eval, and Summary APIs - Added validation alignment plan for Column Management APIs, addressing security gaps and backend validation inconsistencies. - Created new MCP tools for Run Prompt APIs, enhancing validation and organization filtering. - Established shared validation utilities for Eval APIs, fixing organization filtering issues and ensuring consistent validation across layers. - Implemented organization filtering and bug fixes in Summary APIs to prevent cross-org data leaks and improve error handling. - Introduced centralized API base URL configuration for frontend integration.
There was a problem hiding this comment.
The PR title, desc and the diff don't seem to match..
Have added few comments and also getcallexecutions.mdx, updatetestcomponents.mdx, addemptyrowstodataset.mdx are 0-byte in the diff but the underlying endpoints (GET /simulate/run-tests/{id}/call-executions/, PATCH /simulate/run-tests/{id}/components/, POST /model-hub/develops/{id}/add_empty_rows/) still seem live on prod.
So, were these meant to be deleted, or left as empty placeholders?
Also - there seem to be merge conflicts with the other PR that was merged. Could you pls resolve them?
| {"name": "execution_ids", "in": "query", "required": false, "description": "JSON-encoded array of test execution UUIDs to compare.", "type": "string"} | ||
| ]} | ||
| responseExample={{"execution-uuid-1": {"evaluations": [{"name": "Tone Check", "average_score": 0.85}]}, "execution-uuid-2": {"evaluations": [{"name": "Tone Check", "average_score": 0.92}]}}} | ||
| responseExample={{ |
There was a problem hiding this comment.
prod wraps it in {status, result: {...}}.
Is this an intended new shape, or should it reflect prod?
| <ResponseField name="agent_definition" type="string"> | ||
| UUID of the associated agent definition. | ||
| </ResponseField> | ||
| <ResponseField name="agent_version" type="string"> |
There was a problem hiding this comment.
the newly added agent_version ResponseField is type="string", but for agent_definition source type prod returns it as an object (id, name, configuration_snapshot).
Also, while the response schema was expanded here, prompt_template, prompt_template_detail, prompt_version, prompt_version_detail, and the bare evals field weren't added. Are those intentionally omitted?
There was a problem hiding this comment.
fixed the agent version one and yes those fiedls are very specific usecase. can ignore them,
| </ParamField> | ||
| <ParamField name="404" type="Not Found"> | ||
| No test run found with the specified `run_test_id`. | ||
| ```json |
There was a problem hiding this comment.
Tested the newly added error blocks against Prod - with bogus inputs: a non-matching UUID returns HTTP 500 (not 404) with {"result": "Unable to fetch evaluation summary. Please try again"}, and 401 comes back as {"detail": "..."}.
sarthakFuture
changed the title
2 participants
…t, Eval, and Summary APIs
Pull Request
Description
Describe the changes in this pull request:
Checklist
Linear
Fixes TH-4580
Related Issues
Closes #<issue_number>