fix(scorers): detect AI SDK v5 tool parts in toolCallAccuracy - #1364
fix(scorers): detect AI SDK v5 tool parts in toolCallAccuracy#1364VihaanAgarwal wants to merge 1 commit into
Conversation
The code-based tool-call-accuracy scorer read tool calls from message parts only when a part's type was exactly "tool_call". AI SDK v5 encodes a tool call in message parts as `type: "tool-<toolName>"`, so those parts were skipped and the scorer reported the expected tool as not called even when it was. Accept tool-typed parts as well, using the existing extraction that already reads `tool-<name>` from tool-call lists; tool results and streaming events stay excluded via shouldExtractToolNameFromType.
🦋 Changeset detectedLatest commit: ce5afd5 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughExtends toolCallAccuracy AI SDK v5 part type support
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
2 issues found across 3 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/scorers/src/tool-call-accuracy.ts">
<violation number="1" location="packages/scorers/src/tool-call-accuracy.ts:283">
P1: AI SDK v5 dynamic tool parts with `type: "dynamic-tool"` and a `toolName` field are still skipped because `normalizeMessageType` produces `"dynamic_tool"`, which fails both `"tool_call"` and `shouldExtractToolNameFromType` (`startsWith("tool")` is false). `extractToolName` would otherwise read the `toolName` from these parts, so dynamic tool calls will still score as not called.</violation>
<violation number="2" location="packages/scorers/src/tool-call-accuracy.ts:283">
P1: The stream-event guard `shouldExtractToolNameFromType` rejects valid AI SDK v5 `tool-<toolName>` parts when the tool name collides with excluded prefixes or suffixes (e.g., `input_validator` → `tool_input_validator` matches the `tool_input_` prefix, `generate_delta` matches the `_delta` suffix). This causes false negatives where the scorer silently skips actually-called tools, undermining the fix for AI SDK v5 support.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| if (partType !== "tool_call" && !shouldExtractToolNameFromType(partType)) { | ||
| continue; | ||
| } | ||
|
|
There was a problem hiding this comment.
P1: AI SDK v5 dynamic tool parts with type: "dynamic-tool" and a toolName field are still skipped because normalizeMessageType produces "dynamic_tool", which fails both "tool_call" and shouldExtractToolNameFromType (startsWith("tool") is false). extractToolName would otherwise read the toolName from these parts, so dynamic tool calls will still score as not called.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/scorers/src/tool-call-accuracy.ts, line 283:
<comment>AI SDK v5 dynamic tool parts with `type: "dynamic-tool"` and a `toolName` field are still skipped because `normalizeMessageType` produces `"dynamic_tool"`, which fails both `"tool_call"` and `shouldExtractToolNameFromType` (`startsWith("tool")` is false). `extractToolName` would otherwise read the `toolName` from these parts, so dynamic tool calls will still score as not called.</comment>
<file context>
@@ -276,7 +276,11 @@ function extractToolNamesFromParts(parts: unknown[]): string[] {
+ // not `"tool_call"`. Accept those too (extractToolName reads the name from
+ // the type); tool results and streaming events are still excluded because
+ // shouldExtractToolNameFromType returns false for them.
+ if (partType !== "tool_call" && !shouldExtractToolNameFromType(partType)) {
continue;
}
</file context>
| if (partType !== "tool_call" && !shouldExtractToolNameFromType(partType)) { | |
| continue; | |
| } | |
| // AI SDK v5 encodes a tool call in message parts as `type: "tool-<name>"`, | |
| // not `"tool_call"`. Dynamic tools use `type: "dynamic-tool"` with a | |
| // `toolName` field. Accept those too (extractToolName reads the name from | |
| // the type or toolName); tool results and streaming events are still | |
| // excluded because shouldExtractToolNameFromType returns false for them. | |
| if (partType !== "tool_call" && partType !== "dynamic_tool" && !shouldExtractToolNameFromType(partType)) { | |
| continue; | |
| } |
| // not `"tool_call"`. Accept those too (extractToolName reads the name from | ||
| // the type); tool results and streaming events are still excluded because | ||
| // shouldExtractToolNameFromType returns false for them. | ||
| if (partType !== "tool_call" && !shouldExtractToolNameFromType(partType)) { |
There was a problem hiding this comment.
P1: The stream-event guard shouldExtractToolNameFromType rejects valid AI SDK v5 tool-<toolName> parts when the tool name collides with excluded prefixes or suffixes (e.g., input_validator → tool_input_validator matches the tool_input_ prefix, generate_delta matches the _delta suffix). This causes false negatives where the scorer silently skips actually-called tools, undermining the fix for AI SDK v5 support.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/scorers/src/tool-call-accuracy.ts, line 283:
<comment>The stream-event guard `shouldExtractToolNameFromType` rejects valid AI SDK v5 `tool-<toolName>` parts when the tool name collides with excluded prefixes or suffixes (e.g., `input_validator` → `tool_input_validator` matches the `tool_input_` prefix, `generate_delta` matches the `_delta` suffix). This causes false negatives where the scorer silently skips actually-called tools, undermining the fix for AI SDK v5 support.</comment>
<file context>
@@ -276,7 +276,11 @@ function extractToolNamesFromParts(parts: unknown[]): string[] {
+ // not `"tool_call"`. Accept those too (extractToolName reads the name from
+ // the type); tool results and streaming events are still excluded because
+ // shouldExtractToolNameFromType returns false for them.
+ if (partType !== "tool_call" && !shouldExtractToolNameFromType(partType)) {
continue;
}
</file context>
1 participant
While using the code-based
toolCallAccuracyscorer on agent outputs, I hit a case where it reported the expected tool as never called even though it was. The cause is in how tool calls are read from messageparts.What was wrong
extractToolNamesFromPartsonly accepted a part when itstypewas exactly"tool_call":AI SDK v5 encodes a tool call in message parts as
type: "tool-<toolName>"(for example"tool-getWeather"), which normalizes to"tool_getweather"and never equals"tool_call". So tool calls living inmessage.partswere skipped, and the scorer scored0for a tool that was actually called.This is inconsistent with the tool-call list path, which already extracts
tool-<name>/tool_<name>viaextractToolName(covered by the existing "keeps extracting from tool_ type in tool call lists" test).Fix
Accept tool-typed parts as well, reusing the existing
shouldExtractToolNameFromTypeguard so tool results and streaming events stay excluded:extractToolNamethen reads the name from thetype, exactly as it already does for lists.Test
Added a regression test that feeds an AI SDK v5 assistant message with a
type: "tool-searchProducts"part. It fails onmain(score0) and passes with this change (score1). Full@voltagent/scorerssuite stays green (77 tests).Summary by cubic
Fixes
toolCallAccuracyin@voltagent/scorersto detect tool calls encoded astype: "tool-<name>"in message parts, so expected tools aren’t reported as missing.tool-<name>part types when extracting tool names; still excludes tool results and streaming events viashouldExtractToolNameFromType.type: "tool-searchProducts"; full suite remains green.Written for commit ce5afd5. Summary will update on new commits.
Summary by CodeRabbit
tool-<name>format.