schema: classify transcription models + add gpt-transcribe + audio media types - #1121
schema: classify transcription models + add gpt-transcribe + audio media types#1121paultancre-bt wants to merge 2 commits into
Conversation
Add an additive optional `transcription` flag to ModelSchema (backwards compatible; avoids widening the closed `flavor` enum which would break older consumers). Flag all speech-to-text models (gpt-4o-transcribe family, whisper-1, fireworks whisper) so the app can keep them out of the chat playground picker and, later, surface them in a dedicated transcription UI. Also add the missing `gpt-transcribe` model. It is priced per-minute with no per-token pricing, which the current token-only schema can't represent, so no cost fields are set (per-minute cost is deferred; tracked in GATE-30). Pairs with the gateway /v1/audio/transcriptions endpoint. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
getSupportedMediaTypes now includes audio types when the model is flagged `transcription`, so the playground file picker allows audio uploads for gpt-4o-transcribe etc. Widen the `model` params to string (ModelName is a subtype) so callers can pass the resolved model name. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7fedd1984c
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "Codex (@codex) review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "Codex (@codex) address that feedback".
| // Transcription models take an audio file regardless of provider format. | ||
| if (model && getAvailableModels()[model]?.transcription) { | ||
| Object.assign(baseSupport, toMediaTypeSupport(AUDIO_MEDIA_TYPES)); |
There was a problem hiding this comment.
This model-aware media filtering determines which files the transcription request path accepts, so it is request/upload shaping in the deprecated proxy. The root AGENTS.md assigns that behavior to the parent gateway/ implementation and permits only the catalog JSON exception here; keeping active behavior in this package risks the gateway and deprecated proxy diverging. Move this logic to the paired gateway change and retain only the catalog update here.
AGENTS.md reference: AGENTS.md:L6-L15
Useful? React with 👍 / 👎.
| // Transcription models take an audio file regardless of provider format. | ||
| if (model && getAvailableModels()[model]?.transcription) { | ||
| Object.assign(baseSupport, toMediaTypeSupport(AUDIO_MEDIA_TYPES)); |
There was a problem hiding this comment.
When a flagged OpenAI transcription model is selected, baseSupport already contains PDFs and images, and this assignment merely adds audio types. Consequently, calls such as getSupportedMediaTypes("openai", "whisper-1") advertise application/pdf and image/jpeg, allowing the upload validator to accept files that the transcription endpoint will reject. In the active gateway implementation, short-circuit to the transcription-specific media set instead of unioning it with the format defaults.
AGENTS.md reference: AGENTS.md:L6-L12
Useful? React with 👍 / 👎.
| // Transcription models take an audio file regardless of provider format. | ||
| if (model && getAvailableModels()[model]?.transcription) { | ||
| Object.assign(baseSupport, toMediaTypeSupport(AUDIO_MEDIA_TYPES)); |
There was a problem hiding this comment.
When a browser reports a supported .mp4, .webm, or .mpeg upload using its standard video/mp4, video/webm, or video/mpeg MIME type, this branch merges only AUDIO_MEDIA_TYPES, so the transcription helper rejects the file despite supporting those container formats. The aliases already exist in VIDEO_MEDIA_TYPES; include the applicable container MIME aliases in the transcription-specific set in the active gateway implementation.
AGENTS.md reference: AGENTS.md:L6-L12
Useful? React with 👍 / 👎.
1 participant
Catalog support for speech-to-text transcription models. Paired with the braintrust main-repo branch
paultancre/transcription-endpoint(gateway/v1/audio/transcriptionsendpoint + playground support). Tracked in GATE-30.Changes
transcription: booleanonModelSchema(backwards compatible; avoids widening the closedflavorenum, which would break older deployed proxies/SDKs).gpt-4o-transcribefamily,whisper-1, fireworks whisper) and add the missinggpt-transcribe(per-minute priced, so no per-token cost fields — noted in GATE-30).getSupportedMediaTypes/isMediaTypeSupportedinclude audio when the model istranscription, and widen themodelparam tostring.AUDIO_MEDIA_TYPESexpanded to OpenAI's full set: wav, mp3, mpeg, mp4, webm, flac, ogg, m4a.ref - https://github.com/braintrustdata/braintrust/pull/19029