OpenAPI-generated Python API client plan

[AbhiPrasad]

Tracking issue for generating the public Braintrust REST API client from
braintrust-openapi and exposing it through
braintrust.api without changing existing authentication, routing, retries, or high-level SDK behavior.

This supersedes #673. Keep #673 as the detailed runtime/call-site audit and #682 as the reference for
the Experiments behavior and tests. This issue is the implementation source of truth.

Implementation checklist

• Foundation: transport (ref(api): establish transport boundary for API client #641), policy-aware errors/retries (feat(api): add policy-aware HTTP transport #653), and client/service facade (feat(api): add client bootstrap and service facade #655)
• Generator foundation: measurement spike, pinned spec/models/drift CI (feat(api): generate REST models from pinned OpenAPI spec #690), and cross-resource partitioning
• Projects: generate bindings and migrate registration and project lookup (bindings landed in feat(api): generate Projects REST bindings #697)
• Experiments: generate bindings and migrate registration, lookup, fetching, and summaries (bindings and summarize landed in feat(api): generate Experiments REST bindings #701)
• Datasets: generate bindings and migrate registration, lookup, fetching, and summaries
• Prompts: generate bindings and migrate load_prompt() retrieval
• Functions: generate metadata bindings and migrate function/parameter retrieval; keep invocation specialized
• Generate all remaining reviewed REST tags; keep intentionally unsupported tags documented and exclude Proxy
• Document the public REST resources, types, and stability policy
• Automate pinned-spec update PRs after 3–4 manual updates

Next: complete Datasets end to end with generated bindings plus SDK migration and behavior-parity coverage.

Goal

Generate a synchronous, typed REST client from a pinned OpenAPI spec while reusing the SDK's existing
requests transport, auth, organization selection, endpoint routing, retry policies, and error types.
Generated source is committed; install, sdist, and wheel builds remain offline and do not run codegen.

Out of scope

• Replacing braintrust.generated_types or its backend generator.
• Async APIs, Pydantic, httpx, or another HTTP pool.
• Proxy/streaming, log ingestion, attachments/object storage, cache fallback, or other multi-step
  workflows already handled by specialized SDK paths.
• Observable changes to existing high-level braintrust APIs, except the explicit Experiment.summarize() silently discards scores/metrics on any fetch failure instead of surfacing or retrying #639 fix in Step 3.

Settled design

1. Keep both public clients with distinct roles. BraintrustClient owns one transport and router shared by
   handwritten auth and its non-owning openapi view; BraintrustOpenApiClient is the direct REST
   constructor and owns its transport only when it creates one. Both constructors are network-free.
   Organization discovery is explicit through client.auth.login(); direct OpenAPI construction requires
   a configured api_url.
2. Keep the hand-written shell: client.py, _transport.py, _routing.py, _service.py, auth,
   errors, retry policies, and specialized workflows. Generate selected REST resource classes.
3. Generate only models transitively reachable from selected endpoint tags as dependency-free
   TypedDicts/type aliases. Export only types reachable from documented public methods through
   braintrust.api.types; never re-export them from braintrust.
4. Selected spec operationIds define the public resource method names mechanically after snake-case
   normalization. Generated inline-response names are derived mechanically from them as well.
5. Generate sync only. A future async client requires a native async transport and separate design.
6. Add no runtime dependency beyond the standard library, typing_extensions, and existing
   braintrust.api modules.
7. Commit openapi/spec.json, a full spec commit SHA, and its SHA-256. Normal generation is hermetic;
   BRAINTRUST_OPENAPI_ROOT is the local-checkout override.
8. Select endpoint tags through an explicit allowlist. Proxy remains unselected; revisit it only
   in a separate proposal after upstream fixes proxy{path+} and if generation benefits streaming.
9. Responses use generated mappings, not a parallel dataclass/raw convention. Unknown/additive keys
   must survive.
10. braintrust.api._generated remains an implementation namespace and is not imported directly by
    users. Generated resource methods exposed through BraintrustOpenApiClient and exports in
    braintrust.api.types.__all__ define the reviewed compatibility surface.

Required shape

openapi/
├── README.md
├── config.json                 # spec/tool pins, selected tags, idempotent writes
└── spec.json                   # committed full snapshot
py/scripts/
├── fetch-openapi-spec.py
├── generate-api-client.py         # regeneration and --check drift mode
└── openapi_codegen.py             # shared validation/generation implementation
py/src/braintrust/api/
├── <hand-written client, transport, routing, service, auth, errors, and policies>
├── types/__init__.py           # deliberate public REST type exports
└── _generated/
    ├── __init__.py
    ├── models/
    │   ├── __init__.py
    │   └── projects.py         # reachable models for the selected tag
    └── projects.py             # operation registry + generated ProjectsAPI resource

Every generated package directory needs __init__.py; setuptools currently uses regular package
discovery. Wheel tests must import the generated package from an installed wheel.

The scripts under py/scripts/ resolve repository paths relative to themselves. py/Makefile remains the
SDK entry point:

cd py
make generate-api-client
make check-api-client-codegen

generate-api-client writes to a temporary directory, emits final ruff-formatted output, atomically
replaces emitted files, removes stale generated artifacts/directories, and leaves handwritten files alone.
check-api-client-codegen uses the same script in --check mode to regenerate and diff without dirtying
the worktree.

Generator contract

Inputs and determinism

Pin in openapi/config.json:

• full braintrust-openapi commit SHA and spec SHA-256;
• datamodel-code-generator==0.72.4;
• the ruff version, asserted equal to .pre-commit-config.yaml;
• generator Python version (recorded, not enforced); and
• endpoint-generator schema/version, selected tags, and verified idempotent-write operation IDs.

Generated headers contain the spec SHA, spec hash, generator versions, and a content hash. They must
not contain timestamps, hostnames, or absolute paths. Mark py/src/braintrust/api/_generated/** as
generated in .gitattributes and exclude it from pre-commit; the generator owns formatting and syntax.

Model flags proven by Step 0:

--input-file-type openapi
--output-model-type typing.TypedDict
--target-python-version 3.10
--use-union-operator
--enum-field-as-literal all
--use-generic-container-types
--use-field-description
--strict-nullable
--naming-strategy primary-first
--no-use-closed-typed-dict
--disable-future-imports
--formatters ruff-format
--custom-file-header "..."

Do not copy backend cleanup that collapses “optional” into “nullable”; request types must preserve the
OpenAPI 3.0 distinction between missing fields and nullable: true.

Spec validation

Before generation, fail with an actionable error unless all of these hold:

• all operation IDs remain globally unique, and selected operations have valid IDs and usable tags;
• local $refs reachable from selected operations resolve transitively, including referenced parameters;
• component, operation-derived, and inline-response names do not collide after Python normalization;
• selected success responses and request bodies use supported status/media types;
• selected operations use only the intentionally supported path/query parameter shapes;
• path templates map to declared scalar parameters; and
• idempotent-write operation IDs reference selected write operations.

Slice the full pinned spec to selected operations and their transitive component closure before model
generation. Report selected operation/schema counts; do not hard-code them. Unsupported tags are not
validated or generated merely because they exist in the pinned upstream snapshot.

Generated resources and runtime service

Each selected operation records only the runtime metadata currently needed: operation ID, method,
relative path, path/query parameters, request-body presence, success/JSON statuses, and resolved
RetryMode. One generated module per selected tag contains both the operation registry and a typed
resource class whose method names are derived from operationId. The hand-written ResourceAPI base
handles serialization and HTTP behavior.

The initial ResourceAPI intentionally supports the Projects wire shapes: API-target requests, JSON
bodies and responses, scalar path parameters, scalar query parameters, and exploded query arrays.
Broaden it only when another selected endpoint requires a new OpenAPI shape.

ResourceAPI must preserve existing routing, auth, injected transports, custom deployments, typed
errors/request IDs, and opaque non-2xx bodies. Ignore the spec's absolute servers URL. Never send
Braintrust credentials to signed object-storage URLs.

Runtime policy guardrails

OpenAPI does not encode replay safety or product fallback behavior. Generate resolved metadata from
conservative defaults plus explicit semantic allowlists:

Operation	Default
GET / HEAD	SAFE_READ
writes	NONE
POST logical reads	explicit safe-read allowlist when required
verified idempotent writes	membership in idempotent_writes
ingestion	existing specialized path / LOG_INGESTION; never a second retry loop

Every generated operation must have an explicit resolved mode; coverage fails on unclassified
operations. Keep payload-dependent or multi-call behavior hand-written, including conditional
registration, base-experiment HTTP 400 -> None, attachment reconciliation, invocation, signed
uploads, caching/fallback, and lenient summary behavior. The API layer reports errors; higher layers
own visibly marked fallback. #673 remains the detailed replay-safety audit.

Import and type-surface guardrails

logger.py imports the handwritten client module during import braintrust, so that module must not import
any generated resources at module import time. BraintrustOpenApiClient imports generated resource classes
only while initializing an actual client instance. A bare import braintrust must leave every
braintrust.api._generated.* module absent from sys.modules.

Keep the two type surfaces separate:

• braintrust.generated_types: SDK logging/eval payload types.
• braintrust.api.types: generated REST request/response types.

Add a test that inventories overlapping public names and makes shape changes review-visible.

Rollout

Each step is independently landable and revertible. Generated output and the infrastructure that
creates it should be separate commits. Do not widen a step while its exit criteria are unmet.

Step 0 — measurement spike ✅

Measured against the then-current public spec with datamodel-code-generator==0.72.4:

• 6,815 lines / ~200 KB; 392 TypedDicts + 114 aliases; 506 unique names, no collisions.
• Byte-identical repeated generation and byte-identical output on Python 3.10 vs 3.13.
• Warm import: 18.9 ms on 3.13, 13.9 ms on 3.10; cold import: 37.9 ms.
• --disable-future-imports is required by repository policy.
• --formatters ruff-format is required for output clean under ruff 0.15.21.
• One models.py was sufficient for the measurement spike; Step 2 replaced the full model file with
  resource-scoped reachable models. Lazy loading remains required.

Step 1 — pinned spec, validator, models, drift CI ✅ (#690)

Deliver:

• committed config/spec snapshot, fetch/local-override flow, validator, and deterministic model generator;
• private _generated/models.py, imported by nothing at runtime at this stage (Step 2 later replaced it with
  _generated/models/projects.py);
• Make targets, ruff-pin sync check, .gitattributes, and pre-commit exclusion;
• dedicated Ubuntu drift job outside the nox shard matrix and included in checks-passed;
• generator tests for determinism, hash/pin validation, filtering/skip exactness, collisions, invalid IDs,
  media types, $ref parameters, nullable-vs-missing, composition types, and JSON-compatible scalars;
• Python 3.10–3.14 import/type coverage, lazy-import guard, and installed-wheel content test.

Commit order: .gitattributes first; infrastructure/tests second; generated output last.

Exit: a clean checkout passes make check-api-client-codegen; import braintrust loads no generated
module; build/test do not access the network; installed wheel contains _generated.

Step 2 — minimal Projects vertical slice ✅ (#697)

Landed the five current Projects operations end-to-end through BraintrustOpenApiClient.projects:
post_project, get_project, get_project_id, patch_project_id, and delete_project_id. The reviewed
public REST types are CreateProject, PatchProject, Project, and GetProjectResponse.

Decision: keep generation, but keep it deliberately narrow. Projects is deterministic metadata plus
shared templates with no operation-specific executable branches. The landed generator/runtime:

• selects exactly one configured OpenAPI tag and rejects selecting a second until model ownership is
  designed explicitly;
• slices the full pinned spec to that tag's transitive component closure;
• emits models/projects.py plus one combined operation-registry/ProjectsAPI module;
• derives method, operation constant, and inline response names mechanically from operationId, rejecting
  collisions in each emitted namespace;
• forwards request bodies and parameters transparently without injecting client context or implicit
  org_name defaults; callers pass organization fields explicitly when needed;
• classifies postProject declaratively as IDEMPOTENT_WRITE, GETs mechanically as SAFE_READ, and the
  remaining writes as NONE;
• shares BraintrustClient's transport/router between explicit auth.login() and the non-owning OpenAPI
  view, while retaining a direct, network-free BraintrustOpenApiClient constructor; and
• supports only the request/response shapes exercised by Projects, including scalar path/query parameters,
  exploded query arrays, exact declared success statuses, JSON bodies/responses, and additive response keys.

This reduces the generated model surface from the full 238-schema snapshot to 13 reachable schemas; seven
referenced parameter components are retained in the sliced spec as inputs. Codegen reports 5 selected
operations and 13 reachable schemas. Add explicit cross-resource model partitioning before selecting a
second tag, and broaden ResourceAPI only in response to a concrete endpoint shape.

Exit met: all five operations have cassette-backed real-backend coverage, exact-wire local HTTP coverage,
router/transport lifecycle coverage, additive-field checks, static/runtime type tests, lazy-import checks,
codegen drift checks, and installed-wheel coverage.

Step 3 — generated Experiments bindings and #639 ✅ (#701)

Landed all ten operations selected by the pinned spec's Experiments tag through
BraintrustOpenApiClient.experiments, including get_experiment_id_summarize. The reviewed public REST
request and response types are exported through braintrust.api.types.

Codegen now supports deterministic cross-resource model partitioning: definitions reached by one resource
remain resource-owned, while shared definitions are emitted once in models/common.py and imported
explicitly. Projects plus Experiments generates 15 operations and 45 reachable component schemas. Generated
GETs resolve mechanically to SAFE_READ; confirmed logical POST reads use the reviewed safe_reads
allowlist; other writes remain non-retrying unless explicitly classified.

Experiment.summarize() now uses the generated summarize binding. Successful and intentionally skipped
summaries are represented by SummarySuccess and SummarySkipped, with comparison as the primary result;
the deprecated read-only scores and metrics bridges remain serialized for compatibility. Summary
retrieval errors are no longer swallowed: transient failures retry through the policy-aware transport and
final failures raise typed API errors. Structured summaries support tagged deep deserialization and legacy
payloads containing only top-level score and metric maps.

Exit met: coverage includes deterministic multi-resource codegen, complete operation/retry classification,
exact-wire behavior for all generated Experiments methods, additive response fields, retry exhaustion,
framework propagation, static/runtime typing, structured-summary round trips, and real-backend VCR flows
for implicit and explicit comparison selection. #639 is fixed without
changing unrelated high-level behavior.

Step 4 — SDK-used resources: generate and migrate

Generate each resource and migrate its existing high-level SDK call sites in the same resource-focused PR. Use red -> green and VCR-backed coverage. Remove handwritten wire shaping only after the generated binding has equivalent coverage, and preserve existing behavior and fallbacks.

Tag	SDK migration scope
Projects	Registration and project lookup
Experiments	Registration, lookup, fetching, and summaries; summarize already uses the generated binding
Datasets	Registration, lookup, fetching, and summaries
Prompts	load_prompt() retrieval
Functions	Function/parameter metadata retrieval only

Selecting a tag still publishes every supported operation carrying that tag; do not add a per-operation publication map. Before selecting Functions, separate the streaming invocation operation upstream (or otherwise resolve its tag shape) so metadata generation does not publish an unsupported generic invocation path. Specialized invocation, attachment/storage, proxy/streaming, log-ingestion, cache fallback, environment resolution, and other multi-step workflows remain handwritten.

Exit per resource: all selected operations are generated and policy-classified; the listed SDK call sites use the generated resource; VCR and focused runtime/type coverage prove behavior parity; unrelated high-level behavior is unchanged.

Step 5 — remaining supported bindings

After the SDK-used resources are generated and migrated, expand the selected-tag allowlist resource by resource for the rest of the reviewed public REST surface. Add ResourceAPI capabilities only when required by concrete endpoint shapes, plus selected-spec-to-registry operation coverage and policy-completeness tests. Keep specialized/off-spec workflows handwritten and do not select Proxy.

Exit: every reviewed supported tag and operation is generated; intentionally unsupported tags remain outside the allowlist with a documented rationale.

Step 6 — public REST resources and types

Document the reviewed generated resource methods and only their reachable types. Document preview/stability policy and the distinction between braintrust.api.types and braintrust.generated_types. Presence in the spec is not automatic publication; selecting a tag is the explicit publication decision.

Exit: generated resource methods, braintrust.api.__all__, type exports, docs, and type tests agree on the supported surface.

Step 7 — automated spec updates

After 3–4 manual pin bumps, add a scheduled/manual workflow that updates the SHA and snapshot,
regenerates, runs codegen/runtime/type tests, and opens a PR with operation/schema summaries and the
upstream spec diff. Never auto-merge generated API changes.

Agent execution checklist

For every implementation PR:

1. Work from py/; inspect py/noxfile.py, py/pyproject.toml,
   py/src/braintrust/integrations/versioning.py when relevant, and .github/workflows/checks.yaml.
2. Add the smallest failing test first. Provider response-shape behavior is VCR-first; record real
   cassettes rather than using mocks as the primary regression test.
3. Modify generator/template/runtime source, never generated output directly.
4. Run the narrowest test first, then the exact nox/type/wheel sessions affected by the change.
5. Run make check-api-client-codegen and verify git status stays clean after the check.
6. In the PR body, report generated operation/schema deltas, retry allowlists, exclusions, test
   commands, and any measured import change.

Definition of done

• One offline command regenerates byte-identical committed output from a hash-verified full spec SHA.
• Every selected operation has a generated resource method; unsupported tags remain absent from the
  explicit allowlist.
• Every generated operation has deterministic route/retry metadata and uses the existing
  transport/error stack.
• import braintrust loads no generated module and has no material import regression.
• Builds never fetch or generate; installed wheels contain the generated packages.
• Public exports are deliberate, typed, documented, and separated from braintrust.generated_types.
• Auth, org selection, custom routing, transport injection, retries, and specialized workflows retain
  existing behavior.
• Representative real responses are VCR-covered; runtime, type, lint, and wheel checks pass on Python
  3.10–3.14.