Skip to content

fix(api): harden OpenAI streaming/non-streaming error handling - #1426

Open
sufubao wants to merge 1 commit into
ModelTC:mainfrom
sufubao:upstream-pr/openai-stream-error-hardening
Open

fix(api): harden OpenAI streaming/non-streaming error handling#1426
sufubao wants to merge 1 commit into
ModelTC:mainfrom
sufubao:upstream-pr/openai-stream-error-hardening

Conversation

@sufubao

@sufubao sufubao commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

背景

OpenAI 兼容接口(/v1/chat/completions/v1/completions)在两个层面缺少对未预期异常的兜底:

  1. 非流式 handlerchat_completions / completions):只有 ValueError / ServerBusyError / ClientDisconnected 的捕获,其它异常会冒泡到 Starlette,返回 500 + 长堆栈,且没有结构化错误体。而同文件里的 /generate/generate_stream handler 已经有 except Exception 兜底,行为不一致。
  2. 流式 _safe_stream_wrapper:只捕获 ValueErrorClientDisconnected。后端推理过程中抛出的 ServerBusyError 或其它异常同样会冒泡到 Starlette,客户端只能收到被中断的 SSE 流,拿不到结构化错误;且这条失败路径不会累计 lightllm_request_failure 指标(非流式路径经由 create_error_response 是会累计的)。

改动

  • lightllm/server/api_http.py:给 chat_completionscompletions 各加一个 except Exception 兜底,返回结构化的 EXPECTATION_FAILED (417) 响应,并 logger.error(..., exc_info=True) 记录堆栈。与 /generate handler 对齐。
  • lightllm/server/api_openai.py:扩展 _safe_stream_wrapper
    • ServerBusyError → 输出 ServerBusyError SSE 错误块(带 code=503),并累计失败指标;
    • asyncio.CancelledError → 直接 raise,不吞掉取消;
    • 其它 Exception → 输出 InternalServerError SSE 错误块,并累计失败指标;
    • ValueError 分支同样补上失败指标累计,行为与 create_error_response 一致。
    • 抽出 _record_request_failure_metric / _stream_error_chunk 两个小辅助函数复用。
  • unit_tests/server/test_openai_stream_error_handling.py:新增单测,覆盖 Exception / ServerBusyError / ValueError / ClientDisconnected 四种情况下的 SSE 输出与指标累计。沿用仓库现有的 asyncio.run 风格,未引入 pytest-asyncio 依赖。

兼容性

  • 纯错误路径增强,正常请求行为不变。
  • 失败指标 lightllm_request_failure 此前在流式失败时漏统计,本 PR 补齐,监控数值会因此上升(更准确)。

测试

pytest unit_tests/server/test_openai_stream_error_handling.py 4 passed;pre-commit run(black 21.12b0 + flake8)通过。

Unexpected exceptions raised inside chat_completions/completions handlers
and inside the SSE stream wrapper currently propagate to Starlette,
producing a 500 with a long traceback and no structured error body. The
non-OpenAI /generate handlers already catch the broad Exception case;
this aligns the OpenAI endpoints with that pattern.

- api_http.py: add a final `except Exception` arm to chat_completions and
  completions, returning a structured EXPECTATION_FAILED (417) response.
- api_openai.py: extend _safe_stream_wrapper to also handle
  ServerBusyError (503 SSE error), re-raise asyncio.CancelledError, and
  convert any other exception into an InternalServerError SSE chunk.
  Streaming failures now increment lightllm_request_failure, matching the
  non-streaming create_error_response path.
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant