feat(api): generate Experiments REST bindings - #701
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f9c2512bcc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
Implements step 3 of #683 and fixes #639. This adds the second public generated REST resource: - `BraintrustOpenApiClient.experiments`: all 10 operations selected by the Experiments OpenAPI tag, including `get_experiment_id_summarize` - `braintrust.api.types`: public experiment request and response types Codegen now partitions models across multiple resources. Models reached by one resource remain in its resource-specific module, while shared definitions are emitted once in `models/common.py` and imported explicitly. Against the pinned specification, Projects plus Experiments generates 15 operations and 45 reachable component schemas. Logical POST reads use a reviewed `safe_reads` allowlist, while generated GETs retain mechanical `SAFE_READ` classification and writes remain non-retrying unless explicitly classified. `Experiment.summarize()` now uses the generated summarize binding. Successful and intentionally skipped summaries are represented by `SummarySuccess` and `SummarySkipped`, with `comparison` as the primary result. The deprecated read-only `scores` and `metrics` bridges remain serialized for compatibility. Summary retrieval errors are no longer swallowed: transient failures retry through the policy-aware transport and final failures raise typed API errors. Structured summaries support tagged deep deserialization and legacy payloads containing only top-level score and metric maps. Coverage includes deterministic multi-resource codegen, retry-policy validation, exact wire behavior for all generated Experiments methods, additive responses, retry exhaustion, framework propagation, static and runtime typing, structured-summary round trips, and real-backend VCR flows for implicit and explicit comparison selection.
f9c2512 to
bd49da8
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bd49da87f8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".
| "Projects", | ||
| "Experiments" |
There was a problem hiding this comment.
if I understand correctly, I wonder if we will ever leak unwanted endpoints into the generated client
There was a problem hiding this comment.
or accidentally remove endpoints because someone removed the tag and break them
There was a problem hiding this comment.
I did the allowlist this way so that we very explicitly add only the endpoints we want to the generated client.
Removing tags should also break tests so I'm not too worried about that.
2 participants
Implements step 3 of #683 and fixes #639.
AI Summary
This adds the second public generated REST resource:
BraintrustOpenApiClient.experiments: all 10 operations selected by the Experiments OpenAPI tag, includingget_experiment_id_summarizebraintrust.api.types: public experiment request and response types