fix(anthropic): Record thinking tokens as metric - #2382
Open
Dylan Pulver (dylanpulver) wants to merge 4 commits into
Open
fix(anthropic): Record thinking tokens as metric#2382Dylan Pulver (dylanpulver) wants to merge 4 commits into
Dylan Pulver (dylanpulver) wants to merge 4 commits into
Conversation
…etion_thinking_tokens
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Anthropic Messages API returns
usage.output_tokens_details.thinking_tokenswhen extended or adaptive thinking is active. It reports how many of the billed output tokens the model spent on internal reasoning, and it is always less than or equal tooutput_tokens. TodayparseMetricsFromUsagein the Anthropic plugin readsinput_tokens,output_tokens, both cache token fields, the per-TTL cache creation breakdown, and every numeric field underserver_tool_use, but it never looks atoutput_tokens_details. The value falls through the index signature onAnthropicUsageand never reaches span metrics, so someone running Claude models with thinking enabled cannot see what share of their completion tokens went to reasoning.This is a parity gap rather than a new idea. The OpenAI plugin already maps
output_tokens_details.reasoning_tokenstocompletion_reasoning_tokensthrough the generic*_tokens_detailshandling inopenai-utils.ts, and the Google GenAI and GitHub Copilot plugins emit that same metric from their own provider fields. Anthropic was the remaining provider where the reasoning share of billed output tokens got dropped. I named the metriccompletion_thinking_tokensbecause Anthropic bills and documents the field as thinking tokens, and keeping the provider's own vocabulary avoids implying the number is computed the same way as OpenAI's reasoning tokens.Changes:
js/src/instrumentation/plugins/anthropic-plugin.ts:parseMetricsFromUsagenow readsoutput_tokens_details.thinking_tokensand records it ascompletion_thinking_tokens. The value is a subset ofoutput_tokensrather than extra tokens, so it is deliberately not folded intoprompt_tokensortokensbyfinalizeAnthropicTokens, andcompletion_tokensstays the inclusive billing total.js/src/vendor-sdk-types/anthropic.ts:AnthropicUsagenow declaresoutput_tokens_details?: AnthropicOutputTokensDetails | null. The new interface follows the shape already used forAnthropicCacheCreationUsage, including the index signature, so later sub-fields stay assignable.Both the non-streaming and the streaming paths pick this up, because
aggregateAnthropicStreamChunksroutesmessage_startandmessage_deltausage through the sameparseMetricsFromUsage.Coverage
js/src/instrumentation/plugins/anthropic-plugin.test.tsgains cases mirroring how the OpenAI reasoning token mapping is tested inopenai-plugin.test.ts. One asserts the direct mapping and thatcompletion_tokensstays the inclusive total. One asserts that a null breakdown and a non-numericthinking_tokensare both ignored, matching the existing treatment of a nullcache_creation. One drives the streaming aggregator with amessage_deltacarrying the breakdown and checks thatcompletion_thinking_tokenssurvives finalization whiletokensstays at the prompt plus completion total. All of them fail without the plugin change.On the e2e side, the recorded
anthropic-v0-latestcassette already contains a thinking response carryingoutput_tokens_details, so the metric now shows up on theanthropic-stream-thinking-operationspan. The paired.span-tree.jsonand.span-tree.txtsnapshots for that variant were regenerated together throughpnpm run test:e2e:update, and the diff is one added metric line per file with no other drift.assertions.tsgets a check on the streaming thinking span that the metric is numeric and does not exceedcompletion_tokens, guarded by a presence check in the style of the existingserver_tool_use_web_search_requestsassertion, since the older pinned cassettes predate the field and do not carry it.Fixes #2175
Verification run locally
pnpm testinjs/pnpm run check:typingsinjs/pnpm run lintinjs/mainpnpm exec prettier --checkon the touched filesnode ./scripts/run-e2e-tests.mjs anthropic-instrumentation anthropic-bedrock-instrumentationA changeset was added at
.changeset/anthropic-thinking-tokens-metric.mdmarking abraintrustpatch, matching how recentfix(...)PRs in this repo are versioned.AI-disclosure compliance
The repo has no
CONTRIBUTING.md, no code of conduct, and no pull request template.gh api .../community/profilereturns null for all of those, thebraintrustdata/.githuborg repo does not exist, and neitherAGENTS.md,CLAUDE.md,README.md, norPUBLISHING.mdmentions AI disclosure. Recent merged PR bodies carry no disclosure line, so there is no policy to comply with and nothing was added to the body on that front.