FIX: warn instead of raising on reasoning-model token truncation#2166
FIX: warn instead of raising on reasoning-model token truncation#2166romanlutz wants to merge 2 commits into
Conversation
When a reasoning model hits max_completion_tokens, the API returns finish_reason='length' with empty visible content. Previously this raised EmptyResponseException (misleadingly reported as 'Status Code: 204') and triggered the 10x retry storm. _validate_response now logs a warning explaining that reasoning models consume tokens on hidden reasoning, and returns a graceful empty response (error='empty') instead of raising, so runs continue. Non-length empty responses still raise as before. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
| if not (has_content or has_audio or has_tool_calls): | ||
| return construct_response_from_request( | ||
| request=request, | ||
| response_text_pieces=[""], | ||
| response_type="text", | ||
| error="empty", | ||
| ) | ||
| return None |
There was a problem hiding this comment.
Is this the right behavior? based on how I understand this, if there is any content, we would just ignore it? Also, there is no test for this currently
| "if you expected complete content." | ||
| ) | ||
| if not (has_content or has_audio or has_tool_calls): | ||
| return construct_response_from_request( |
There was a problem hiding this comment.
2 points:
1- before this change, the _validate_response always returns None in the "happy path", and never returns a Message , so I think the return type should just be None and not Message|None
2- with above in mind, I think all we want to do, is consider the response valid and not raise an exception when it has a finish_reason == 'length' ... and instead _construct_message_from_response_async should know how to handle such responses (this is the function called right after successful validation in openai_target.py:457
Description
Reasoning models spend completion tokens on hidden reasoning before emitting any visible output. When
max_completion_tokensis set too low, the API can returnfinish_reason="length"with empty visible content.OpenAIChatTarget._validate_responsetreated that empty content as a hard failure and raisedEmptyResponseException(surfaced with a misleading "Status Code: 204"), which also triggered the full retry loop even though retrying with the same token budget cannot succeed.Since a low token budget can be a deliberate configuration choice, this changes the behavior to warn rather than raise:
finish_reason="length", log a warning explaining that reasoning models consume tokens on hidden reasoning and thatmax_completion_tokensmay need to be increased.error="empty") so the run continues instead of raising and retrying.EmptyResponseExceptionas before.The final stored result for the empty case is unchanged from today (an
error="empty"piece). The difference is no exception, no retry storm, and a clear warning pointing at the token limit.Tests and Documentation
test_validate_response_success_lengthto assert the truncation warning is emitted when content is present.test_validate_response_length_empty_returns_empty_responseasserting that a truncated-and-empty response returns aMessagewitherror="empty"and raises nothing.tests/unit/prompt_target/target/test_openai_chat_target.pysuite passes (101 tests).No documentation changes needed. JupyText not run (no doc or notebook changes).