[fix][evaluation] anchor strategy-4 score regex to the score key position - #617
Open
YuhaoLin2005 wants to merge 1 commit into
Open
[fix][evaluation] anchor strategy-4 score regex to the score key position#617YuhaoLin2005 wants to merge 1 commit into
YuhaoLin2005 wants to merge 1 commit into
Conversation
…tion, prevent reason-field leak
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
fix
Check the PR title
(Optional) Translate the PR title into Chinese
[fix][evaluation] 将策略4的分数正则锚定到 score 键位置,防止从 reason 字段误取分数
(Optional) More detailed description for this PR
en:
Problem. In
parseContentOutput, strategy 4 (parseScoreWithRegex) usesto recover a numeric score when the value is not a plain JSON number (e.g.
{"score": "优秀", ...}falls through strategies 1-3 and reaches strategy 4). Because the regex is anchored only on the substringscoreand captures the first digit run that follows it, digits inside thereasonvalue are silently reported as the score.Concrete reproductions (all currently return a wrong
Scoreinstead of an error):{"score": "优秀", "reason": "答案3个要点都覆盖了"}3{"score": "优秀", "reason": "请用score: 3作为答案"}3The secondary regex also drops an explicit sign at strategy 4:
score: -0.3, reason: ...parsed as+0.3.Root cause. Strategy 4 has no key-position anchor. Any
scoresubstring followed by a digit run — including text inside thereasonvalue — is treated as the score field/value.Fix. Anchor the match to the real key position:
"score"key must appear at content start or right after{/,(a real JSON field position); a barescoreonly at content start (prose likescore: 0.85, reason: ...).[^"]*?preserves non-digit prefixes inside a quoted value (得分8分→8,90分→90) without crossing the closing quote.[+-]?preserves an explicit sign, which the old[^0-9]*ate (-0.3→+0.3).Tests. Added scenarios 37-42 to
Test_parseContentOutputcovering the motivating regression, the reason-field leak, digit-less no-match, Chinese unit suffixes, and sign preservation. Full suite: 42/42 pass;go buildpass;gofmtclean.Behavior change & relation to #257. Strategy 4 now only accepts a score anchored to a real key position: a JSON
"score"field (at content start or right after{/,) or prose that starts withscore:. Loose prose where the word appears mid-sentence (The final score is 3) now fails loudly instead of being silently parsed — deliberate, because the leak can only be eliminated by anchoring to the key position, and a loud failure is strictly better than a silently wrong score.This also connects to #257's "sometimes a normal response": a non-numeric score whose
reasoncontains digits previously produced a wrong score that looked like a normal response; this PR makes that case fail loudly instead. The digit-less class (no ASCII digits at all) remains unhandled, as discussed in the issue thread.zh(optional):
问题。
parseContentOutput的策略4(parseScoreWithRegex)用(?i)score[^0-9]*([0-9]+...)在值不是纯 JSON 数字时(如{"score": "优秀", ...}落到策略4)尝试恢复分数。因为正则只锚定score子串并捕获其后第一个数字串,reason 值里的数字会被静默当作分数。{"score": "优秀", "reason": "答案3个要点都覆盖了"}3{"score": "优秀", "reason": "请用score: 3作为答案"}3修复。 把匹配锚定到真实键位置(quoted
"score"须在内容起始或{/,之后;裸score仅在内容起始)。惰性[^"]*?保留带引号值内的非数字前缀且不跨过收引号;[+-]?保留显式符号。测试。
Test_parseContentOutput新增场景37-42:触发回归、reason 泄漏、无数字不匹配、中文单位后缀、符号保留。全量 42/42 通过;go build通过;gofmt干净。行为变更与 #257 的关系。 策略4 现在只接受锚定在真实键位置的分数:JSON
"score"字段(内容起始或{/,之后)、或以score:开头的散文。句中出现的松散score(如The final score is 3)现在响亮失败而非静默解析——这是有意的:泄漏只能靠键位锚定消除,响亮失败严格优于静默错分。这也解释了 #257「有时正常响应」可能是错分:非数字 score 且 reason 含数字时旧代码返回错分(看似正常),本 PR 让其响亮失败;全无 ASCII 数字类仍未处理(见 issue 讨论)。(Optional) Which issue(s) this PR fixes
Related to #257 (score parsing).