Skip to content

docs(ai): add LLM observability page - #4568

Merged
D-K-P merged 4 commits into
mainfrom
llm-observability-docs
Aug 11, 2026
Merged

docs(ai): add LLM observability page#4568
D-K-P merged 4 commits into
mainfrom
llm-observability-docs

Conversation

@D-K-P

@D-K-P D-K-P commented Aug 11, 2026

Copy link
Copy Markdown
Member

Summary

Adds a docs page for LLM observability: every opted-in Vercel AI SDK call inside a task becomes its own span in the run trace, carrying the model, provider, token counts, cost, and latency. The page covers turning it on per call with experimental_telemetry: { isEnabled: true }, what each span inspector tab shows (Overview, Messages, Tools, and a Prompt tab when linked), linking a call to its prompt version with toAISDKTelemetry(), and querying usage across runs with TRQL against the llm_metrics table.

It sits in the AI dropdown under Features, next to Prompts, and cross-links the Query page.

It is explicit that capture is opt-in per call (not automatic) and only covers Vercel AI SDK calls, and notes the @ai-sdk/otel requirement on AI SDK 7. Every API name, span tab, and TRQL column was checked against the SDK and the live query schema.

Also in this PR

Corrects one bullet in the AI Agents overview: it claimed an in-progress chat resumes on the new version after a redeploy, which contradicts the version-upgrades and backend pages. Chat agent runs are pinned to the version they started on; moving onto new code is an explicit version upgrade.

D-K-P added 2 commits August 11, 2026 15:27
The overview said an in-progress chat resumes on the new version after a redeploy, which contradicts the version-upgrades and backend pages. Chat agent runs are pinned to the version they started on; moving onto new code is an explicit version upgrade.
Document capturing Vercel AI SDK calls as spans in the run trace: turning it on per call, what each span inspector tab shows, linking a call to its prompt version, and querying usage across runs with TRQL.
@changeset-bot

changeset-bot Bot commented Aug 11, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: ab043c5

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@coderabbitai

coderabbitai Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 3b9beecb-ee7e-42da-b570-6c8d2f66997f

📥 Commits

Reviewing files that changed from the base of the PR and between 88fe26b and ab043c5.

📒 Files selected for processing (1)
  • docs/docs.json
🚧 Files skipped from review as they are similar to previous changes (1)
  • docs/docs.json
📜 Recent review details
⏰ Context from checks skipped due to timeout. (3)
  • GitHub Check: Analyze (javascript-typescript)
  • GitHub Check: Analyze (actions)
  • GitHub Check: Build and publish previews

Walkthrough

Added AI observability documentation for Vercel AI SDK telemetry, span inspection, prompt linking, custom metadata, and TRQL usage queries. Added the page to the Agents features navigation. Updated chat deployment documentation to describe version pinning during redeploys and explicit upgrades.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely identifies the main change: adding an LLM observability documentation page.
Description check ✅ Passed The description clearly explains the documentation scope, technical behavior, navigation update, and related chat-version correction.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch llm-observability-docs

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

coderabbitai[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

AI SDK 7 auto-registration of @ai-sdk/otel only happens for chat agents, so plain tasks must register the OpenTelemetry integration themselves. Also note that chat agents auto-capture only when a prompt is set.
@D-K-P
D-K-P enabled auto-merge (squash) August 11, 2026 14:35
@pkg-pr-new

pkg-pr-new Bot commented Aug 11, 2026

Copy link
Copy Markdown

Open in StackBlitz

@trigger.dev/build

npm i https://pkg.pr.new/@trigger.dev/build@ab043c5

trigger.dev

npm i https://pkg.pr.new/trigger.dev@ab043c5

@trigger.dev/core

npm i https://pkg.pr.new/@trigger.dev/core@ab043c5

@trigger.dev/python

npm i https://pkg.pr.new/@trigger.dev/python@ab043c5

@trigger.dev/react-hooks

npm i https://pkg.pr.new/@trigger.dev/react-hooks@ab043c5

@trigger.dev/redis-worker

npm i https://pkg.pr.new/@trigger.dev/redis-worker@ab043c5

@trigger.dev/rsc

npm i https://pkg.pr.new/@trigger.dev/rsc@ab043c5

@trigger.dev/schema-to-json

npm i https://pkg.pr.new/@trigger.dev/schema-to-json@ab043c5

@trigger.dev/sdk

npm i https://pkg.pr.new/@trigger.dev/sdk@ab043c5

commit: ab043c5

@D-K-P
D-K-P merged commit 336f515 into main Aug 11, 2026
25 checks passed
@D-K-P
D-K-P deleted the llm-observability-docs branch August 11, 2026 14:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants