Skip to content

fix(tracing): stop set_tracing_processor_configs deadlocking on its own lock - #543

Open
michaelxu2288 wants to merge 1 commit into
scaleapi:mainfrom
michaelxu2288:fix/tracing-config-deadlock
Open

michaelxu2288 wants to merge 1 commit into
scaleapi:mainfrom
michaelxu2288:fix/tracing-config-deadlock

Conversation

@michaelxu2288

@michaelxu2288 michaelxu2288 commented Oct 1, 2026 •

Copy link
Copy Markdown

Problem

set_tracing_processor_configs() (exported from agentex.lib.core.tracing.tracing_processor_manager) never returns. TracingProcessorManager.set_processor_configs takes self.lock and then calls add_processor_config for each config, which takes the same lock again. self.lock is a threading.Lock, which is not reentrant, so the first config blocks forever:

from agentex.lib.core.tracing.tracing_processor_manager import set_tracing_processor_configs
set_tracing_processor_configs([SGPTracingProcessorConfig(...)])  # hangs

Called at ACP startup, the server never finishes booting; called before a worker starts, it never reaches worker.run(). Nothing inside the SDK calls the plural API today, which is how it went unnoticed, but it is exported next to add_tracing_processor_config.

Fix

self.lock becomes a threading.RLock. The batch still registers under one lock acquisition and the nested add_processor_config can re-enter it. add_processor_config itself is unchanged. Two-line diff.

set_processor_configs appends to the registered processors, the same as calling add_processor_config once per config. I kept that behaviour rather than making it replace the existing list; happy to change that if "set" was meant literally.

Verification

  • New tests/lib/core/tracing/test_tracing_processor_manager.py registers two configs from a thread with a 5 s join.
    • Before: AssertionError: set_processor_configs blocked on the manager's own lock.
    • After: passes, and both the sync and the async processor lists hold one processor per config.
  • uv run pytest -n 0 tests/lib/core/tracing/test_tracing_processor_manager.py tests/lib/core/tracing/test_span_queue.py tests/lib/core/tracing/processors: 89 passed.
  • ruff check and pyright clean on both files.

RetriggerConfidence Score: 4/5

The lock fix appears sound, but the test must satisfy the repository’s named-number rule before merging.

Fix All in CursorFindings

  1. P2 Test deadline has no name ▶
  2. P2 Worker errors look successful ▶
Fix with agent prompt
### Issue 1
tests/lib/core/tracing/test_tracing_processor_manager.py:27
`_finishes` uses `5.0` as an inline timeout. The repository requires magic numbers to be stored as class or instance variables with descriptive names. Name this deadline to satisfy that requirement before merging.

### Issue 2
tests/lib/core/tracing/test_tracing_processor_manager.py:31
If registration raises in the worker thread, `_finishes` returns `True` because the thread has stopped. The test then fails on a later list check instead of showing the registration error. Capture the worker error and raise it after the join.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

The tracing processor manager now uses a reentrant lock so batch config registration can finish while it keeps registration protected.

  • Adds tests for batch registration and one-at-a-time registration.

Reviews (1) · Last reviewed commit: "fix(tracing): stop set_tracing_processor..."

…wn lock

TracingProcessorManager.set_processor_configs takes self.lock and then
calls add_processor_config for each config, which takes the same lock
again. self.lock was a threading.Lock, which is not reentrant, so the
first call to the exported set_tracing_processor_configs() blocked
forever: on an ACP server it never finishes startup, on a worker it never
reaches worker.run().

Use an RLock so the batch still registers under one lock acquisition and
the nested add can re-enter it. add_processor_config is unchanged.

The new test registers two configs from a thread and fails on the old
lock (the thread is still blocked after 5 s); it passes with the RLock.
return manager


def _finishes(target: Any, timeout: float = 5.0) -> bool:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Test deadline has no name

_finishes uses 5.0 as an inline timeout. The repository requires magic numbers to be stored as class or instance variables with descriptive names. Name this deadline to satisfy that requirement before merging.

Rule Used: Store magic numbers as class or instance variables with descriptive names rather than using them inline in the code. (source)

Learned From
scaleapi/scaleapi#126388

Prompt To Fix With AI
This is a comment left during a code review.
Path: tests/lib/core/tracing/test_tracing_processor_manager.py
Line: 27

Comment:
**Test deadline has no name**

`_finishes` uses `5.0` as an inline timeout. The repository requires magic numbers to be stored as class or instance variables with descriptive names. Name this deadline to satisfy that requirement before merging.

**Rule Used:** Store magic numbers as class or instance variables with descriptive names rather than using them inline in the code. ([source](https://app.greptile.com/scale-ai/-/custom-context?memory=002e0051-41ad-46c1-9098-47433c580150))

**Learned From**
[scaleapi/scaleapi#126388](https://github.com/scaleapi/scaleapi/pull/126388)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Fix in Cursor Fix in Claude Code Fix in Codex

thread = threading.Thread(target=target, daemon=True)
thread.start()
thread.join(timeout)
return not thread.is_alive()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Worker errors look successful

If registration raises in the worker thread, _finishes returns True because the thread has stopped. The test then fails on a later list check instead of showing the registration error. Capture the worker error and raise it after the join.

Prompt To Fix With AI
This is a comment left during a code review.
Path: tests/lib/core/tracing/test_tracing_processor_manager.py
Line: 31

Comment:
**Worker errors look successful**

If registration raises in the worker thread, `_finishes` returns `True` because the thread has stopped. The test then fails on a later list check instead of showing the registration error. Capture the worker error and raise it after the join.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant