Skip to content

Report over-large requests as 400, not 500 - #264

Merged
0xShug0 merged 1 commit into
0xShug0:mainfrom
derekja:fix/capacity-error-not-500
Aug 17, 2026
Merged

Report over-large requests as 400, not 500#264
0xShug0 merged 1 commit into
0xShug0:mainfrom
derekja:fix/capacity-error-not-500

Conversation

@derekja

@derekja derekja commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #262, as suggested there.

Some rejections that are the caller's to fix surface as 500 server_error. The message names an internal graph or cache, so there is nothing actionable in it, and the caller goes looking for a server fault that isn't there.

  • Add engine::runtime::CapacityError for "this device or model cannot serve a request of this size" — distinct from a genuine fault, because the caller can fix it by sending less.
  • Convert two sites across two engines: the Qwen3-ASR prefill allocation, and VoxCPM2's prompt and generation cache-length checks. Each message now carries the numbers, so the remedy is arithmetic rather than guesswork.
  • Map it to 400 invalid_request_error in the server dispatch.

This follows the ServerBusyError → 503 precedent: the layer that knows the condition throws a typed exception, the server decides the status. CapacityError lives in the engine runtime headers because the throw sites are under src/models/ and the mapping is in app/server/ — both already include engine/framework/*.

Deliberately not a sweep. Around 30 models throw failed to allocate, but they are not uniform — several allocate model weights at load time, which is a genuine fault and should stay a 500. Whether a given site is caller-size-driven needs per-site judgement, so this converts only the two engines I could exercise end to end on real hardware. The type and the mapping make the rest mechanical for whoever knows each model.

Validation

Two servers side by side, CUDA / H100 12GB slice, one at main and one patched.

engine request main this branch
Qwen3-ASR no prompt 200 200
Qwen3-ASR 400-char prompt 200 200
Qwen3-ASR 4000-char prompt 500 server_error
failed to allocate Qwen3 ASR thinker prefill graph
400 invalid_request_error
Qwen3 ASR prefill graph does not fit in device memory at this size (1173 prompt steps, of which 13 are audio tokens); shorten the transcription prompt or the audio
VoxCPM2 60-char text 200, 506924 B pcm 200, 506924 B pcm (identical)
VoxCPM2 20000-char text 500 server_error
VoxCPM2 generation exceeds model cache length
400 invalid_request_error
VoxCPM2 generation exceeds the model cache length (4104 prefill rows + 4096 requested tokens, limit 8192); shorten the input text

The VoxCPM2 numbers are the point: 4104 + 4096 = 8200 against a limit of 8192 tells the caller exactly how much to cut.

CapacityError is thrown from three sites in two files, so no existing 500 changes meaning elsewhere.

Some rejections that are the caller's to fix surface as 500 server_error.
The message names an internal graph or cache, so there is nothing actionable
in it, and the caller goes looking for a server fault that is not there.

Add engine::runtime::CapacityError for "this device or model cannot serve a
request of this size", convert two sites across two engines (the Qwen3-ASR
prefill allocation, and VoxCPM2 prompt and generation cache-length checks),
and map it to 400 invalid_request_error in the server dispatch. Each message
now carries the numbers, so the remedy is arithmetic rather than guesswork.

Follows the ServerBusyError -> 503 precedent: the layer that knows the
condition throws a typed exception, the server decides the status.
CapacityError lives in the engine runtime headers because the throw sites are
under src/models/ and the mapping is in app/server/, and both already include
engine/framework/*.

Deliberately not a sweep. Around 30 models throw "failed to allocate", but
they are not uniform -- several allocate model weights at load time, which is
a genuine fault and should stay a 500. Whether a site is caller-size-driven
needs per-site judgement, so this converts only the two engines exercised end
to end on hardware.
@0xShug0
0xShug0 merged commit eaede92 into 0xShug0:main Aug 17, 2026
6 checks passed
@0xShug0

0xShug0 commented Aug 17, 2026

Copy link
Copy Markdown
Owner

@derekja Merged. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants