feat(llmobs): add annotateAgentManifest manual API to LLMObs SDK - #12318
feat(llmobs): add annotateAgentManifest manual API to LLMObs SDK#12318yahya-mouman wants to merge 4 commits into
Conversation
- Add AgentManifest immutable value class with Builder pattern in LLMObs.java - Add AgentTool immutable value class in LLMObs.java - Add AGENT_MANIFEST constant to LLMObsTags - Add annotateAgentManifest() default method to LLMObsSpan interface - Add comprehensive builder tests for AgentManifest and AgentTool Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Stores agent manifest fields (name, instructions, model, model_settings, tools) as an internal tag `_ml_obs_tag.agent_manifest`. Only applies to agent spans; warns and no-ops on other span kinds. Null manifest is silently ignored. Tools with null/empty names are skipped with a warning. A second call overwrites the previous manifest. Framework field "AgentObs SDK" is always added when any manifest fields are present. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add AGENT_MANIFEST_KEY byte constant, include the tag in TAGS_FOR_REMAPPING, and handle it in the meta serialization loop as a msgpack map. Add two tests: one verifying all manifest fields appear in meta, one verifying the tag does not leak into the tags list. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
🎯 Code Coverage (details) 🔗 Commit SHA: b5f56f1 | Docs | View more details | Give us feedback! |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b5f56f11d6
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| String name = | ||
| manifest.getName() != null ? manifest.getName() : (sn != null ? sn.toString() : null); | ||
| if (name != null && !name.isEmpty()) { |
There was a problem hiding this comment.
Fall back to the span name for empty manifest names
When a caller supplies .name(""), such as by forwarding an optional configuration value, this selects the empty string instead of the span-name fallback, and the following guard then omits name entirely. The serialized agent manifest is therefore nameless despite the API's default-to-span-name behavior; treat both null and empty names as absent before selecting the fallback.
Useful? React with 👍 / 👎.
ncybul
left a comment
There was a problem hiding this comment.
A couple minor suggestions but overall looks good to me! Would be nice to see a manual verification of setting the manifest via the SDK.
| * | ||
| * <p>Build via {@link AgentManifest#builder()} and pass to {@link | ||
| * LLMObsSpan#annotateAgentManifest(AgentManifest)}. Only applied on agent spans; ignored on other | ||
| * span kinds. A subsequent call on the same span overwrites the previous manifest. |
There was a problem hiding this comment.
This is different from the Python implementation, right? I think in Python, we were merging the two manifests together whenever possible. It's probably best that we align the implementations and choose one approach.
Personally, I would lean towards merging the fields whenever possible.
| String name = | ||
| manifest.getName() != null ? manifest.getName() : (sn != null ? sn.toString() : null); |
There was a problem hiding this comment.
What happens when name is an empty string here?
| private static final String METADATA = LLMOBS_TAG_PREFIX + LLMObsTags.METADATA; | ||
| private static final String TOOL_DEFINITIONS = LLMOBS_TAG_PREFIX + LLMObsTags.TOOL_DEFINITIONS; | ||
| private static final String AGENT_MANIFEST = LLMOBS_TAG_PREFIX + LLMObsTags.AGENT_MANIFEST; | ||
| private static final String MANUAL_FRAMEWORK = "AgentObs SDK"; |
There was a problem hiding this comment.
Idk if "AgentObs SDK" makes sense here or if we should default to "custom" or "manual" since the former makes it seem like we are determining the agent manifest via the SDK rather than the user supplying it. (same comment applies to the python PR)
Summary
Adds
annotateAgentManifest(LLMObs.AgentManifest)to the Java LLMObs SDK, allowing users to manually declare an agent span's configuration (name, instructions, model, model_settings, tools). Mirrors the Python implementation (DataDog/dd-trace-py#19771) and follows theannotatePromptpattern from #12161.Changes
LLMObsTags.java— addsAGENT_MANIFEST = "agent_manifest"constantLLMObs.java— adds two public immutable builder classes:LLMObs.AgentTool— represents a single tool (name, optionaldescription, optionalparameters)LLMObs.AgentManifest— builder withname,instructions,model,model_settings,toolsLLMObsSpan.java— addsdefault void annotateAgentManifest(LLMObs.AgentManifest)(no-op default for backwards compat)NoOpLLMObsSpan.java— explicit@Overrideno-opDDLLMObsSpan.java— real implementation: validates span kind (agent only), builds manifest map, stores as_ml_obs_tag.agent_manifestLLMObsSpanMapper.java— addsagent_manifesttoTAGS_FOR_REMAPPING; serializes tometa.agent_manifestas a msgpack mapBehaviour
agentspan kind; other span kinds emit a log warning and no-opframeworkis set to"AgentObs SDK"automatically by the SDKnamedefaults to the span name if not providedmodel_settingskeys are forwarded as-is (no allowlist in this initial PR)Test plan
./gradlew :dd-trace-api:test --tests "datadog.trace.api.llmobs.LLMObsTest"— 27 tests pass (4 new)./gradlew :dd-java-agent:agent-llmobs:test --tests "datadog.trace.llmobs.domain.DDLLMObsSpanTest"— 39 tests pass (7 new)./gradlew :dd-trace-core:test --tests "datadog.trace.llmobs.writer.ddintake.LLMObsSpanMapperTest"— 18 tests pass (2 new)🤖 Generated with Claude Code