Skip to content

feat(llmobs): add annotateAgentManifest manual API to LLMObs SDK - #12318

Draft
yahya-mouman wants to merge 4 commits into
masterfrom
yahya/llmobs-agent-manifest-manual
Draft

feat(llmobs): add annotateAgentManifest manual API to LLMObs SDK#12318
yahya-mouman wants to merge 4 commits into
masterfrom
yahya/llmobs-agent-manifest-manual

Conversation

@yahya-mouman

Copy link
Copy Markdown

Summary

Adds annotateAgentManifest(LLMObs.AgentManifest) to the Java LLMObs SDK, allowing users to manually declare an agent span's configuration (name, instructions, model, model_settings, tools). Mirrors the Python implementation (DataDog/dd-trace-py#19771) and follows the annotatePrompt pattern from #12161.

Changes

  • LLMObsTags.java — adds AGENT_MANIFEST = "agent_manifest" constant
  • LLMObs.java — adds two public immutable builder classes:
    • LLMObs.AgentTool — represents a single tool (name, optional description, optional parameters)
    • LLMObs.AgentManifest — builder with name, instructions, model, model_settings, tools
  • LLMObsSpan.java — adds default void annotateAgentManifest(LLMObs.AgentManifest) (no-op default for backwards compat)
  • NoOpLLMObsSpan.java — explicit @Override no-op
  • DDLLMObsSpan.java — real implementation: validates span kind (agent only), builds manifest map, stores as _ml_obs_tag.agent_manifest
  • LLMObsSpanMapper.java — adds agent_manifest to TAGS_FOR_REMAPPING; serializes to meta.agent_manifest as a msgpack map

Behaviour

LLMObs.AgentManifest manifest = LLMObs.AgentManifest.builder()
    .name("travel_desk")
    .instructions("Book travel for the user.")
    .model("gpt-4o")
    .modelSettings(Map.of("temperature", 0.7))
    .tools(List.of(LLMObs.AgentTool.from("get_weather", "Look up weather", null)))
    .build();

agentSpan.annotateAgentManifest(manifest);
  • Only applies to agent span kind; other span kinds emit a log warning and no-op
  • Calling twice on the same span overwrites the previous manifest (no merge)
  • framework is set to "AgentObs SDK" automatically by the SDK
  • name defaults to the span name if not provided
  • model_settings keys are forwarded as-is (no allowlist in this initial PR)
  • Null manifest is silently ignored

Test plan

  • ./gradlew :dd-trace-api:test --tests "datadog.trace.api.llmobs.LLMObsTest" — 27 tests pass (4 new)
  • ./gradlew :dd-java-agent:agent-llmobs:test --tests "datadog.trace.llmobs.domain.DDLLMObsSpanTest" — 39 tests pass (7 new)
  • ./gradlew :dd-trace-core:test --tests "datadog.trace.llmobs.writer.ddintake.LLMObsSpanMapperTest" — 18 tests pass (2 new)

🤖 Generated with Claude Code

yahya-mouman and others added 4 commits August 27, 2026 13:36
- Add AgentManifest immutable value class with Builder pattern in LLMObs.java
- Add AgentTool immutable value class in LLMObs.java
- Add AGENT_MANIFEST constant to LLMObsTags
- Add annotateAgentManifest() default method to LLMObsSpan interface
- Add comprehensive builder tests for AgentManifest and AgentTool

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Stores agent manifest fields (name, instructions, model, model_settings,
tools) as an internal tag `_ml_obs_tag.agent_manifest`. Only applies to
agent spans; warns and no-ops on other span kinds. Null manifest is
silently ignored. Tools with null/empty names are skipped with a warning.
A second call overwrites the previous manifest. Framework field
"AgentObs SDK" is always added when any manifest fields are present.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add AGENT_MANIFEST_KEY byte constant, include the tag in TAGS_FOR_REMAPPING,
and handle it in the meta serialization loop as a msgpack map. Add two
tests: one verifying all manifest fields appear in meta, one verifying
the tag does not leak into the tags list.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@yahya-mouman yahya-mouman added the type: feature Enhancements and improvements label Aug 27, 2026
@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Aug 27, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 82.42%
Overall Coverage: 58.36% (-0.42%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b5f56f1 | Docs | View more details | Give us feedback!

@ncybul

ncybul commented Aug 27, 2026

Copy link
Copy Markdown

@codex

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b5f56f11d6

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +316 to +318
String name =
manifest.getName() != null ? manifest.getName() : (sn != null ? sn.toString() : null);
if (name != null && !name.isEmpty()) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Fall back to the span name for empty manifest names

When a caller supplies .name(""), such as by forwarding an optional configuration value, this selects the empty string instead of the span-name fallback, and the following guard then omits name entirely. The serialized agent manifest is therefore nameless despite the API's default-to-span-name behavior; treat both null and empty names as absent before selecting the fallback.

Useful? React with 👍 / 👎.

@ncybul ncybul left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple minor suggestions but overall looks good to me! Would be nice to see a manual verification of setting the manifest via the SDK.

*
* <p>Build via {@link AgentManifest#builder()} and pass to {@link
* LLMObsSpan#annotateAgentManifest(AgentManifest)}. Only applied on agent spans; ignored on other
* span kinds. A subsequent call on the same span overwrites the previous manifest.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is different from the Python implementation, right? I think in Python, we were merging the two manifests together whenever possible. It's probably best that we align the implementations and choose one approach.

Personally, I would lean towards merging the fields whenever possible.

Comment on lines +316 to +317
String name =
manifest.getName() != null ? manifest.getName() : (sn != null ? sn.toString() : null);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens when name is an empty string here?

private static final String METADATA = LLMOBS_TAG_PREFIX + LLMObsTags.METADATA;
private static final String TOOL_DEFINITIONS = LLMOBS_TAG_PREFIX + LLMObsTags.TOOL_DEFINITIONS;
private static final String AGENT_MANIFEST = LLMOBS_TAG_PREFIX + LLMObsTags.AGENT_MANIFEST;
private static final String MANUAL_FRAMEWORK = "AgentObs SDK";

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Idk if "AgentObs SDK" makes sense here or if we should default to "custom" or "manual" since the former makes it seem like we are determining the agent manifest via the SDK rather than the user supplying it. (same comment applies to the python PR)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants