fix(api): harden OpenAI streaming/non-streaming error handling - #1426
Open
sufubao wants to merge 1 commit into
Open
fix(api): harden OpenAI streaming/non-streaming error handling#1426sufubao wants to merge 1 commit into
sufubao wants to merge 1 commit into
Conversation
Unexpected exceptions raised inside chat_completions/completions handlers and inside the SSE stream wrapper currently propagate to Starlette, producing a 500 with a long traceback and no structured error body. The non-OpenAI /generate handlers already catch the broad Exception case; this aligns the OpenAI endpoints with that pattern. - api_http.py: add a final `except Exception` arm to chat_completions and completions, returning a structured EXPECTATION_FAILED (417) response. - api_openai.py: extend _safe_stream_wrapper to also handle ServerBusyError (503 SSE error), re-raise asyncio.CancelledError, and convert any other exception into an InternalServerError SSE chunk. Streaming failures now increment lightllm_request_failure, matching the non-streaming create_error_response path.
Contributor
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
OpenAI 兼容接口(
/v1/chat/completions、/v1/completions)在两个层面缺少对未预期异常的兜底:chat_completions/completions):只有ValueError/ServerBusyError/ClientDisconnected的捕获,其它异常会冒泡到 Starlette,返回 500 + 长堆栈,且没有结构化错误体。而同文件里的/generate、/generate_streamhandler 已经有except Exception兜底,行为不一致。_safe_stream_wrapper:只捕获ValueError和ClientDisconnected。后端推理过程中抛出的ServerBusyError或其它异常同样会冒泡到 Starlette,客户端只能收到被中断的 SSE 流,拿不到结构化错误;且这条失败路径不会累计lightllm_request_failure指标(非流式路径经由create_error_response是会累计的)。改动
lightllm/server/api_http.py:给chat_completions和completions各加一个except Exception兜底,返回结构化的EXPECTATION_FAILED (417)响应,并logger.error(..., exc_info=True)记录堆栈。与/generatehandler 对齐。lightllm/server/api_openai.py:扩展_safe_stream_wrapper:ServerBusyError→ 输出ServerBusyErrorSSE 错误块(带code=503),并累计失败指标;asyncio.CancelledError→ 直接raise,不吞掉取消;Exception→ 输出InternalServerErrorSSE 错误块,并累计失败指标;ValueError分支同样补上失败指标累计,行为与create_error_response一致。_record_request_failure_metric/_stream_error_chunk两个小辅助函数复用。unit_tests/server/test_openai_stream_error_handling.py:新增单测,覆盖Exception/ServerBusyError/ValueError/ClientDisconnected四种情况下的 SSE 输出与指标累计。沿用仓库现有的asyncio.run风格,未引入pytest-asyncio依赖。兼容性
lightllm_request_failure此前在流式失败时漏统计,本 PR 补齐,监控数值会因此上升(更准确)。测试
pytest unit_tests/server/test_openai_stream_error_handling.py4 passed;pre-commit run(black 21.12b0 + flake8)通过。