Skip to content

Python: Best effort to serialize tool def to Json for observability#7029

Open
TaoChenOSU wants to merge 5 commits into
mainfrom
fix/tool-def-serialization-in-observability
Open

Python: Best effort to serialize tool def to Json for observability#7029
TaoChenOSU wants to merge 5 commits into
mainfrom
fix/tool-def-serialization-in-observability

Conversation

@TaoChenOSU

@TaoChenOSU TaoChenOSU commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Motivation & Context

A bug was discovered while working on a task that uses an agent that has the code interpreter tool. The observability module reported "TypeError: Object of type AutoCodeInterpreterToolParam is not JSON serializable". This happens because the observability module tries to serialize the tool definition to json while the object is not serilizable.

Description & Review Guide

  • What are the major changes?
    • Add logic to serialize Azure SDK type
    • Best effort to serialize such that serialization failures don't fail the application (warning will be given)
  • What is the impact of these changes?
    • Bug fix
  • What do you want reviewers to focus on?
    • The serialization logic

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR is linked to an issue and there is no other open PR for this issue (see Related Issue above).
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

@TaoChenOSU TaoChenOSU self-assigned this Jul 9, 2026
Copilot AI review requested due to automatic review settings July 9, 2026 19:51
@TaoChenOSU TaoChenOSU added python Usage: [Issues, PRs], Target: Python observability Usage: [Issues, PRs], Target: observability related features labels Jul 9, 2026
@github-actions github-actions Bot changed the title Best effort to serialize tool def to Json for observability Python: Best effort to serialize tool def to Json for observability Jul 9, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens Python OpenTelemetry (OTel) observability by making tool-definition serialization best-effort, preventing telemetry capture from raising when tools contain non-JSON-serializable values (notably Azure SDK tool models and nested mappings).

Changes:

  • Add best-effort serialization for gen_ai.tool.definitions, including per-tool fallback that skips only the offending tool and logs a warning with its name.
  • Improve tool-to-OTel conversion for Azure SDK Mapping-based tool models by using as_dict() when available to recursively produce JSON-serializable dicts.
  • Add regression and behavior tests covering Azure Foundry code-interpreter tools and partial/all unserializable tool lists.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
python/packages/foundry/tests/foundry/test_foundry_chat_client.py Adds a regression test ensuring Foundry CodeInterpreterTool telemetry tool definitions serialize correctly.
python/packages/core/tests/core/test_observability.py Adds unit tests for skipping/omitting unserializable tool definitions while preserving serializable ones and warning with tool names.
python/packages/core/agent_framework/observability.py Implements best-effort tool-definition serialization and as_dict() handling for Azure SDK tool model mappings; wires new serializer into span attribute extraction.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated Code Review

Reviewers: 5 | Confidence: 92% | Result: All clear

Reviewed: Correctness, Security Reliability, Test Coverage, Failure Modes, Design Approach


Automated review by TaoChenOSU's agents

@github-actions

github-actions Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Python Test Coverage

Python Test Coverage Report •
FileStmtsMissCoverMissing
packages/core/agent_framework
   observability.py10379590%419, 421–422, 425, 428, 431–432, 437–438, 444–445, 451–452, 459, 461–462, 465, 468, 471–472, 477–478, 484–485, 491–492, 499, 676–677, 887, 891–893, 895, 903–904, 908, 952, 954, 965–967, 969–971, 975, 983, 1107–1108, 1343, 1622–1623, 1884, 1927–1928, 2117, 2349–2350, 2354, 2394–2395, 2399, 2441–2444, 2450, 2481–2482, 2498, 2514–2527, 2659, 2662, 2674, 2691, 2695–2696, 2699, 2705, 2825, 3042, 3044
TOTAL44164527288% 

Python Unit Test Overview

Tests Skipped Failures Errors Time
8856 33 💤 0 ❌ 0 🔥 2m 18s ⏱️

@TaoChenOSU TaoChenOSU marked this pull request as ready for review July 9, 2026 21:02

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated Code Review

Reviewers: 4 | Confidence: 90%

✓ Correctness

This PR correctly fixes a bug where Azure SDK tool models (non-dict Mappings) caused TypeError during JSON serialization in the observability module. The fix uses as_dict() for recursive conversion of Azure SDK models and adds a best-effort serialization wrapper that gracefully skips individual unserializable tools. The implementation is well-structured with appropriate error handling, and the tests thoroughly validate both the happy path and fallback behavior.

✓ Test Coverage

The PR adds good test coverage for the new serialization logic. The two new core tests cover partial and total serialization failure, verifying both graceful degradation and that warnings name the offending tool. The foundry regression test exercises the real Azure SDK model end-to-end through the as_dict() path. One minor gap: the defensive except around _tools_to_dict itself raising (lines 2338-2345) has no dedicated test, and the as_dict() branch in _build_tool_otel_definition has no isolated unit test in core (only the integration test in foundry). These are nice-to-haves, not blocking.

✓ Failure Modes

This PR correctly implements best-effort serialization for telemetry tool definitions. The error handling is properly layered: conversion failures are caught at the top level, JSON serialization failures trigger a per-tool fallback that preserves good tools while skipping and identifying bad ones. The as_dict() integration for Azure SDK types is sound — if it fails for any tool, the exception propagates to the outer try/except in _serialize_tool_definitions which logs a warning and returns None rather than crashing. No silent data corruption or exception leakage paths were found.

✓ Design Approach

I did not find a design-level issue that meets the required evidence bar. The change addresses the stated regression by converting Azure SDK mapping models through as_dict() before observability reshaping, and the new best-effort serializer is consistent with existing behavior that already omits unparseable tool definitions rather than failing the caller.

Suggestions

  • Consider adding a unit test in test_observability.py for when _tools_to_dict itself raises an exception (the try/except at observability.py:2338-2345). You could mock normalize_tools to raise and verify that _serialize_tool_definitions returns None with a warning, without the caller crashing.
  • Consider adding a unit test in the core package for the as_dict() branch of _build_tool_otel_definition (observability.py:2458-2459) using a mock Mapping object with an as_dict method, so the core tests don't depend on the foundry package to cover this path.

Automated review by TaoChenOSU's agents

TaoChenOSU added a commit that referenced this pull request Jul 9, 2026
Comment thread python/packages/core/agent_framework/observability.py
Comment thread python/packages/core/agent_framework/observability.py Outdated
@TaoChenOSU TaoChenOSU enabled auto-merge July 10, 2026 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

observability Usage: [Issues, PRs], Target: observability related features python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants