fix(evaluation): avoid crash on first turn with empty user content#6405
Closed
anxkhn wants to merge 1 commit into
Closed
fix(evaluation): avoid crash on first turn with empty user content#6405anxkhn wants to merge 1 commit into
anxkhn wants to merge 1 commit into
Conversation
`_evaluate_first_turn` guarded only against a `None` `user_content`, then called `get_text_from_content(...).strip()`. When `user_content` is a non-None `Content` whose parts are empty or `None`, `get_text_from_content` returns `None`, so `.strip()` raised `AttributeError: 'NoneType' object has no attribute 'strip'`. Compute the extracted text once and return a `NOT_EVALUATED` result when it is `None`, mirroring the existing guard for a `None` `user_content` above it and the `None`/empty guard in `_convert_llm_response_to_score`. Add a regression test covering `parts=[]` and `parts=None`. Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
copybara-service Bot
pushed a commit
that referenced
this pull request
Jul 16, 2026
Merge #6405 PiperOrigin-RevId: 949112659
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
1. Link to an existing issue (if applicable):
No existing issue. Describing the bug below per the issue-template structure, as
CONTRIBUTING permits for a PR without an associated issue.
2. Or, if no issue exists, describe the change:
Problem:
PerTurnUserSimulatorQualityV1._evaluate_first_turncrashes withAttributeError: 'NoneType' object has no attribute 'strip'when the firstinvocation's
user_contentis a non-NoneContentwhose parts are empty(
parts=[]) orNone(types.Content()).The method guards only the
first_invocation.user_content is Nonecase, thendoes:
get_text_from_content(llm_as_judge_utils.py) returnsNonewhen the contenthas no text parts, because
if content and content.parts:falls through forempty/
Noneparts:So
.strip()is called onNoneand raises. This is reachable through thepublic
Evaluator.evaluate_invocations, which calls_evaluate_first_turn(actual_invocations[0], ...): any evaluated conversationwhose first user turn was built with empty parts (or carries only non-text parts
after text extraction returns
None) crashes the evaluation instead of producinga result.
Reproduction (text extraction results that reach
_evaluate_first_turn):Solution:
Extract the text once and return a
NOT_EVALUATEDPerInvocationResultwhen itis
None, before the comparison. This mirrors the two guards already in thisfile: the
user_content is Noneguard immediately above, and_convert_llm_response_to_score'sif response_text is None or not response_text: return AutoRaterScore().The fix is behavior-preserving for every case that did not crash: the guard is
is Noneonly, so a non-text part (text extraction returns"", notNone) isstill scored and legitimately marked
FAILED. Only the two previously-crashinginputs (
parts=[],types.Content()) now returnNOT_EVALUATED.Testing Plan
Unit Tests:
Added a parametrized regression test
test_evaluate_first_turn_not_evaluated_when_user_content_has_no_textcoveringtypes.Content(role="user", parts=[])andtypes.Content(), asserting theresult is
NOT_EVALUATED(scoreNone) instead of raising.Effectiveness check: temporarily reverting the new guard makes the two new
parametrized cases fail with the exact
AttributeError: 'NoneType' object has no attribute 'strip'; restoring the guardmakes them pass. The test is not vacuous.
Manual End-to-End (E2E) Tests:
Not applicable; this is a unit-level crash guard in the evaluation module,
covered by the added regression test. The behavior is demonstrated by the
reproduction table above (empty/
Noneparts extract toNone, which previouslyhit
.strip()).Checklist
comments needed; the guard mirrors the existing ones directly above/below it.)
the regression test rather than an E2E flow.)
Additional context
The two new
NOT_EVALUATEDcases are consistent with how the aggregator alreadyhandles non-evaluated turns (
num_evaluated == 0yields anEvaluationResultwithout an
overall_score), so a conversation whose only turn is empty no longercrashes and simply contributes nothing to the score.
Diff summary
src/google/adk/evaluation/simulation/per_turn_user_simulator_quality_v1.py(+10/-2): compute
user_text = get_text_from_content(...)once; returnPerInvocationResult(eval_status=NOT_EVALUATED)whenuser_text is None; reuseuser_textin the comparison.tests/unittests/evaluation/simulation/test_per_turn_user_simulation_quality_v1.py(+26): parametrized regression test over
types.Content(role="user", parts=[])and
types.Content().