Report over-large requests as 400, not 500 - #264
Merged
Conversation
Some rejections that are the caller's to fix surface as 500 server_error. The message names an internal graph or cache, so there is nothing actionable in it, and the caller goes looking for a server fault that is not there. Add engine::runtime::CapacityError for "this device or model cannot serve a request of this size", convert two sites across two engines (the Qwen3-ASR prefill allocation, and VoxCPM2 prompt and generation cache-length checks), and map it to 400 invalid_request_error in the server dispatch. Each message now carries the numbers, so the remedy is arithmetic rather than guesswork. Follows the ServerBusyError -> 503 precedent: the layer that knows the condition throws a typed exception, the server decides the status. CapacityError lives in the engine runtime headers because the throw sites are under src/models/ and the mapping is in app/server/, and both already include engine/framework/*. Deliberately not a sweep. Around 30 models throw "failed to allocate", but they are not uniform -- several allocate model weights at load time, which is a genuine fault and should stay a 500. Whether a site is caller-size-driven needs per-site judgement, so this converts only the two engines exercised end to end on hardware.
Owner
|
@derekja Merged. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #262, as suggested there.
Some rejections that are the caller's to fix surface as
500 server_error. The message names an internal graph or cache, so there is nothing actionable in it, and the caller goes looking for a server fault that isn't there.engine::runtime::CapacityErrorfor "this device or model cannot serve a request of this size" — distinct from a genuine fault, because the caller can fix it by sending less.400 invalid_request_errorin the server dispatch.This follows the
ServerBusyError→ 503 precedent: the layer that knows the condition throws a typed exception, the server decides the status.CapacityErrorlives in the engine runtime headers because the throw sites are undersrc/models/and the mapping is inapp/server/— both already includeengine/framework/*.Deliberately not a sweep. Around 30 models throw
failed to allocate, but they are not uniform — several allocate model weights at load time, which is a genuine fault and should stay a 500. Whether a given site is caller-size-driven needs per-site judgement, so this converts only the two engines I could exercise end to end on real hardware. The type and the mapping make the rest mechanical for whoever knows each model.Validation
Two servers side by side, CUDA / H100 12GB slice, one at
mainand one patched.main500 server_errorfailed to allocate Qwen3 ASR thinker prefill graph400 invalid_request_errorQwen3 ASR prefill graph does not fit in device memory at this size (1173 prompt steps, of which 13 are audio tokens); shorten the transcription prompt or the audio500 server_errorVoxCPM2 generation exceeds model cache length400 invalid_request_errorVoxCPM2 generation exceeds the model cache length (4104 prefill rows + 4096 requested tokens, limit 8192); shorten the input textThe VoxCPM2 numbers are the point:
4104 + 4096 = 8200against a limit of8192tells the caller exactly how much to cut.CapacityErroris thrown from three sites in two files, so no existing 500 changes meaning elsewhere.