Skip to content

fix(anthropic): Record thinking tokens as metric - #2382

Open
Dylan Pulver (dylanpulver) wants to merge 4 commits into
braintrustdata:mainfrom
dylanpulver:fix/anthropic-thinking-tokens
Open

fix(anthropic): Record thinking tokens as metric#2382
Dylan Pulver (dylanpulver) wants to merge 4 commits into
braintrustdata:mainfrom
dylanpulver:fix/anthropic-thinking-tokens

Conversation

@dylanpulver

Copy link
Copy Markdown

The Anthropic Messages API returns usage.output_tokens_details.thinking_tokens when extended or adaptive thinking is active. It reports how many of the billed output tokens the model spent on internal reasoning, and it is always less than or equal to output_tokens. Today parseMetricsFromUsage in the Anthropic plugin reads input_tokens, output_tokens, both cache token fields, the per-TTL cache creation breakdown, and every numeric field under server_tool_use, but it never looks at output_tokens_details. The value falls through the index signature on AnthropicUsage and never reaches span metrics, so someone running Claude models with thinking enabled cannot see what share of their completion tokens went to reasoning.

This is a parity gap rather than a new idea. The OpenAI plugin already maps output_tokens_details.reasoning_tokens to completion_reasoning_tokens through the generic *_tokens_details handling in openai-utils.ts, and the Google GenAI and GitHub Copilot plugins emit that same metric from their own provider fields. Anthropic was the remaining provider where the reasoning share of billed output tokens got dropped. I named the metric completion_thinking_tokens because Anthropic bills and documents the field as thinking tokens, and keeping the provider's own vocabulary avoids implying the number is computed the same way as OpenAI's reasoning tokens.

Changes:

  • js/src/instrumentation/plugins/anthropic-plugin.ts: parseMetricsFromUsage now reads output_tokens_details.thinking_tokens and records it as completion_thinking_tokens. The value is a subset of output_tokens rather than extra tokens, so it is deliberately not folded into prompt_tokens or tokens by finalizeAnthropicTokens, and completion_tokens stays the inclusive billing total.
  • js/src/vendor-sdk-types/anthropic.ts: AnthropicUsage now declares output_tokens_details?: AnthropicOutputTokensDetails | null. The new interface follows the shape already used for AnthropicCacheCreationUsage, including the index signature, so later sub-fields stay assignable.

Both the non-streaming and the streaming paths pick this up, because aggregateAnthropicStreamChunks routes message_start and message_delta usage through the same parseMetricsFromUsage.

Coverage

js/src/instrumentation/plugins/anthropic-plugin.test.ts gains cases mirroring how the OpenAI reasoning token mapping is tested in openai-plugin.test.ts. One asserts the direct mapping and that completion_tokens stays the inclusive total. One asserts that a null breakdown and a non-numeric thinking_tokens are both ignored, matching the existing treatment of a null cache_creation. One drives the streaming aggregator with a message_delta carrying the breakdown and checks that completion_thinking_tokens survives finalization while tokens stays at the prompt plus completion total. All of them fail without the plugin change.

On the e2e side, the recorded anthropic-v0-latest cassette already contains a thinking response carrying output_tokens_details, so the metric now shows up on the anthropic-stream-thinking-operation span. The paired .span-tree.json and .span-tree.txt snapshots for that variant were regenerated together through pnpm run test:e2e:update, and the diff is one added metric line per file with no other drift. assertions.ts gets a check on the streaming thinking span that the metric is numeric and does not exceed completion_tokens, guarded by a presence check in the style of the existing server_tool_use_web_search_requests assertion, since the older pinned cassettes predate the field and do not carry it.

Fixes #2175


Verification run locally

Check Command Result
Unit suite pnpm test in js/ 1646 passed, 109 files, 0 failed
Typecheck pnpm run check:typings in js/ clean
Lint pnpm run lint in js/ 0 errors, 1167 warnings, identical to the baseline count on main
Formatting pnpm exec prettier --check on the touched files clean
E2E node ./scripts/run-e2e-tests.mjs anthropic-instrumentation anthropic-bedrock-instrumentation 74 passed, run twice, no drift
E2E (Claude Agent SDK, shares the thinking cassettes) same runner 44 passed

A changeset was added at .changeset/anthropic-thinking-tokens-metric.md marking a braintrust patch, matching how recent fix(...) PRs in this repo are versioned.

AI-disclosure compliance

The repo has no CONTRIBUTING.md, no code of conduct, and no pull request template. gh api .../community/profile returns null for all of those, the braintrustdata/.github org repo does not exist, and neither AGENTS.md, CLAUDE.md, README.md, nor PUBLISHING.md mentions AI disclosure. Recent merged PR bodies carry no disclosure line, so there is no policy to comply with and nothing was added to the body on that front.

@lforst Luca Forstner (lforst) changed the title fix(anthropic): Record output_tokens_details.thinking_tokens as completion_thinking_tokens fix(anthropic): Record thinking tokens as metric Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bot] Anthropic usage.output_tokens_details.thinking_tokens metric not extracted

2 participants