Skip to content

[fix][evaluation] anchor strategy-4 score regex to the score key position - #617

Open
YuhaoLin2005 wants to merge 1 commit into
coze-dev:mainfrom
YuhaoLin2005:fix/score-regex-strategy4
Open

[fix][evaluation] anchor strategy-4 score regex to the score key position#617
YuhaoLin2005 wants to merge 1 commit into
coze-dev:mainfrom
YuhaoLin2005:fix/score-regex-strategy4

Conversation

@YuhaoLin2005

@YuhaoLin2005 YuhaoLin2005 commented Aug 13, 2026

Copy link
Copy Markdown

What type of PR is this?

fix

Check the PR title

  • This PR title match the format: [<type>][<scope>] <description>. For example: [fix][backend] flaky fix
  • The description of this PR title is user-oriented and clear enough for others to understand.
  • Add documentation if the current PR requires user awareness at the usage level.
  • This PR is written in English. PRs not in English will not be reviewed.

(Optional) Translate the PR title into Chinese

[fix][evaluation] 将策略4的分数正则锚定到 score 键位置,防止从 reason 字段误取分数

(Optional) More detailed description for this PR

en:

Problem. In parseContentOutput, strategy 4 (parseScoreWithRegex) uses

(?i)score[^0-9]*([0-9]+(?:\.[0-9]+)?)

to recover a numeric score when the value is not a plain JSON number (e.g. {"score": "优秀", ...} falls through strategies 1-3 and reaches strategy 4). Because the regex is anchored only on the substring score and captures the first digit run that follows it, digits inside the reason value are silently reported as the score.

Concrete reproductions (all currently return a wrong Score instead of an error):

Input Old behavior (Score) Expected
{"score": "优秀", "reason": "答案3个要点都覆盖了"} 3 error (no numeric score)
{"score": "优秀", "reason": "请用score: 3作为答案"} 3 error (leak from reason)

The secondary regex also drops an explicit sign at strategy 4: score: -0.3, reason: ... parsed as +0.3.

Root cause. Strategy 4 has no key-position anchor. Any score substring followed by a digit run — including text inside the reason value — is treated as the score field/value.

Fix. Anchor the match to the real key position:

(?i)(?:(?:^|[{,])\s*"score"|^\s*score)\s*:\s*"?[^"]*?([+-]?[0-9]+(?:\.[0-9]+)?)"?
  • A quoted "score" key must appear at content start or right after {/, (a real JSON field position); a bare score only at content start (prose like score: 0.85, reason: ...).
  • The lazy [^"]*? preserves non-digit prefixes inside a quoted value (得分8分8, 90分90) without crossing the closing quote.
  • [+-]? preserves an explicit sign, which the old [^0-9]* ate (-0.3+0.3).

Tests. Added scenarios 37-42 to Test_parseContentOutput covering the motivating regression, the reason-field leak, digit-less no-match, Chinese unit suffixes, and sign preservation. Full suite: 42/42 pass; go build pass; gofmt clean.

Behavior change & relation to #257. Strategy 4 now only accepts a score anchored to a real key position: a JSON "score" field (at content start or right after {/,) or prose that starts with score:. Loose prose where the word appears mid-sentence (The final score is 3) now fails loudly instead of being silently parsed — deliberate, because the leak can only be eliminated by anchoring to the key position, and a loud failure is strictly better than a silently wrong score.

This also connects to #257's "sometimes a normal response": a non-numeric score whose reason contains digits previously produced a wrong score that looked like a normal response; this PR makes that case fail loudly instead. The digit-less class (no ASCII digits at all) remains unhandled, as discussed in the issue thread.

zh(optional):

问题。 parseContentOutput 的策略4(parseScoreWithRegex)用 (?i)score[^0-9]*([0-9]+...) 在值不是纯 JSON 数字时(如 {"score": "优秀", ...} 落到策略4)尝试恢复分数。因为正则只锚定 score 子串并捕获其后第一个数字串,reason 值里的数字会被静默当作分数。

输入 旧行为(Score) 预期
{"score": "优秀", "reason": "答案3个要点都覆盖了"} 3 报错(无数字分数)
{"score": "优秀", "reason": "请用score: 3作为答案"} 3 报错(从 reason 泄漏)

修复。 把匹配锚定到真实键位置(quoted "score" 须在内容起始或 {/, 之后;裸 score 仅在内容起始)。惰性 [^"]*? 保留带引号值内的非数字前缀且不跨过收引号;[+-]? 保留显式符号。

测试。 Test_parseContentOutput 新增场景37-42:触发回归、reason 泄漏、无数字不匹配、中文单位后缀、符号保留。全量 42/42 通过;go build 通过;gofmt 干净。

行为变更与 #257 的关系。 策略4 现在只接受锚定在真实键位置的分数:JSON "score" 字段(内容起始或 {/, 之后)、或以 score: 开头的散文。句中出现的松散 score(如 The final score is 3)现在响亮失败而非静默解析——这是有意的:泄漏只能靠键位锚定消除,响亮失败严格优于静默错分。这也解释了 #257「有时正常响应」可能是错分:非数字 score 且 reason 含数字时旧代码返回错分(看似正常),本 PR 让其响亮失败;全无 ASCII 数字类仍未处理(见 issue 讨论)。

(Optional) Which issue(s) this PR fixes

Related to #257 (score parsing).

@CLAassistant

CLAassistant commented Aug 13, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants