feat: OpenTelemetry (OTel) observability for the community API#9419
Open
sriramveeraghanta wants to merge 7 commits into
Open
feat: OpenTelemetry (OTel) observability for the community API#9419sriramveeraghanta wants to merge 7 commits into
sriramveeraghanta wants to merge 7 commits into
Conversation
sriramveeraghanta
requested review from
dheeru0198,
mguptahub and
pablohashescobar
as code owners
Contributor
📝 WalkthroughWalkthroughAdds environment-gated OpenTelemetry tracing, metrics, instrumentation, trace-correlated logging, startup bootstrapping, deployment settings, collector documentation, and unit tests across the API and worker stack. ChangesOpenTelemetry API observability
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant APIStartup
participant configure_otel
participant DjangoCelery
participant OTLPCollector
APIStartup->>configure_otel: initialize when OTEL is enabled
configure_otel->>DjangoCelery: instrument requests and tasks
DjangoCelery->>OTLPCollector: export traces and metrics
DjangoCelery->>OTLPCollector: emit correlated logs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
Adds OpenTelemetry (OTel) observability to the community Django API, including request/task tracing, metrics export, and trace-correlated JSON logs, gated by environment variables for opt-in self-hosting.
Changes:
- Introduces
plane.observabilitybootstrap (configure_otel/flush_otel) and log correlation filter (TraceContextFilter), wired into WSGI/ASGI/Celery/manage entrypoints. - Extends Django and Celery JSON logging to optionally include
trace_id/span_id/service_namefields. - Adds OTel instrumentation/exporter dependencies plus self-hoster docs and deployment env wiring.
Reviewed changes
Copilot reviewed 20 out of 20 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| docs/otel-api-observability/README.md | Self-hoster guide for enabling OTLP traces/metrics/log correlation. |
| docs/otel-api-observability/otel-collector.yaml | Reference collector config (OTLP receivers + filelog pipeline). |
| deployments/cli/community/variables.env | Documents new OTEL_* env vars (off by default). |
| deployments/cli/community/docker-compose.yml | Wires OTEL_* env into api/worker/beat-worker services. |
| deployments/aio/community/variables.env | Documents new OTEL_* env vars (off by default). |
| apps/api/requirements/base.txt | Adds required OTel instrumentation/exporter packages. |
| apps/api/plane/wsgi.py | Calls configure_otel() before Django WSGI initialization. |
| apps/api/plane/asgi.py | Calls configure_otel() before Django ASGI initialization. |
| apps/api/manage.py | Calls configure_otel() before executing Django management commands. |
| apps/api/plane/celery.py | Boots OTel for Celery, adds flush hook, and conditionally enriches worker logs. |
| apps/api/plane/settings/production.py | Conditionally extends JSON log formatter + installs TraceContextFilter. |
| apps/api/plane/settings/local.py | Conditionally extends JSON log formatter + installs TraceContextFilter. |
| apps/api/plane/observability/setup.py | New OTel bootstrap (providers, exporters, instrumentation, flush). |
| apps/api/plane/observability/logging.py | New TraceContextFilter for trace-context log enrichment. |
| apps/api/plane/observability/init.py | Initializes the new observability package. |
| apps/api/plane/tests/unit/observability/conftest.py | Fixtures to isolate OTEL env/global state across tests. |
| apps/api/plane/tests/unit/observability/test_setup.py | Unit tests for gating, wiring, idempotency, exporter selection. |
| apps/api/plane/tests/unit/observability/test_logging.py | Unit tests for TraceContextFilter behavior. |
| apps/api/plane/tests/unit/observability/init.py | Initializes the new unit test package. |
| apps/api/.env.example | Documents OTEL_* env vars for local/self-host configuration. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (3)
apps/api/plane/settings/local.py (1)
96-105: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winOTEL logging block is duplicated verbatim across
local.pyandproduction.py. Both files contain an identical 10-line block that gates onOTEL_ENABLEDand mutates theLOGGINGdict. Theonomission above shows how duplication invites drift. Extract this into a shared helper (e.g.,plane.observability.logging.extend_logging_config(LOGGING)) and call it from both settings modules.
apps/api/plane/settings/local.py#L96-L105: Replace inline block with a call to the shared helper.apps/api/plane/settings/production.py#L106-L115: Same — replace inline block with a call to the shared helper.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/api/plane/settings/local.py` around lines 96 - 105, Extract the duplicated OTEL logging mutation into a shared helper such as extend_logging_config in plane.observability.logging, preserving the OTEL_ENABLED gate and existing formatter, filter, and handler behavior. Replace the inline blocks with helper calls in apps/api/plane/settings/local.py lines 96-105 and apps/api/plane/settings/production.py lines 106-115.apps/api/plane/tests/unit/observability/conftest.py (1)
23-37: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winReset
_TRACER_PROVIDERand_METER_PROVIDERin the fixture.The fixture resets
_CONFIGUREDbut not the provider references. After a test that callsconfigure_otel(),_TRACER_PROVIDERand_METER_PROVIDERare set to real provider objects and persist into subsequent tests. No current test checks these, but addingmonkeypatch.setattrfor them prevents latent test pollution.♻️ Proposed fix
`@pytest.fixture`(autouse=True) def isolate_otel_state(monkeypatch): from plane.observability import setup as otel_setup monkeypatch.setattr(otel_setup, "_CONFIGURED", False, raising=False) + monkeypatch.setattr(otel_setup, "_TRACER_PROVIDER", None, raising=False) + monkeypatch.setattr(otel_setup, "_METER_PROVIDER", None, raising=False)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/api/plane/tests/unit/observability/conftest.py` around lines 23 - 37, Update the isolate_otel_state fixture to also reset observability setup’s _TRACER_PROVIDER and _METER_PROVIDER globals to their unconfigured state via monkeypatch, alongside _CONFIGURED, so provider objects cannot persist between tests.apps/api/plane/observability/setup.py (1)
134-149: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winUnhandled instrumentor exceptions can prevent application startup.
If any instrumentor raises (e.g., version mismatch, missing optional dependency), the exception propagates through
configure_otel()to the caller —manage.py,asgi.py,wsgi.py, andcelery.pyall call it at startup. An observability failure should degrade gracefully rather than take down the application.♻️ Proposed fix: wrap each instrumentor in try/except
def _instrument_libraries() -> None: - """Patch Django + Celery + downstream client libraries. - - DjangoInstrumentor emits HTTP server spans + http.server.* metrics. - CeleryInstrumentor emits a span per task with celery.action, task_name, - task_id, state, and propagates traceparent across the queue so a - request that enqueues a task is linked to that task's execution span. - The others add child spans so latency can be decomposed (SQL query, - Redis op, outbound HTTP). - """ - DjangoInstrumentor().instrument() - CeleryInstrumentor().instrument() - PsycopgInstrumentor().instrument(enable_commenter=False) - RedisInstrumentor().instrument() - RequestsInstrumentor().instrument() - HTTPXClientInstrumentor().instrument() + """Patch Django + Celery + downstream client libraries. + + Each instrumentor is wrapped independently so a single failure + (version mismatch, missing optional dependency) doesn't prevent + the rest from loading or crash the application. + """ + _instrumentors = ( + (DjangoInstrumentor, {}), + (CeleryInstrumentor, {}), + (PsycopgInstrumentor, {"enable_commenter": False}), + (RedisInstrumentor, {}), + (RequestsInstrumentor, {}), + (HTTPXClientInstrumentor, {}), + ) + for cls, kwargs in _instrumentors: + try: + cls().instrument(**kwargs) + except Exception as exc: + logger.warning("Failed to instrument %s: %s", cls.__name__, exc)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/api/plane/observability/setup.py` around lines 134 - 149, Update _instrument_libraries to isolate each instrumentor call with its own try/except so failures from optional dependencies or incompatible versions are caught and do not propagate through configure_otel or block application startup. Continue attempting the remaining instrumentors after an individual failure, and log each exception using the module’s existing observability logging mechanism.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/api/plane/celery.py`:
- Line 36: Align the _OTEL_LOG_ENABLED check in celery.py with
configure_otel()’s _is_enabled() by accepting “on” as a truthy value, and apply
the same update to the Django LOGGING gates in local.py and production.py.
Prefer reusing a shared truthiness helper from plane.observability.setup if
available or introduce one to prevent these checks from diverging.
In `@apps/api/plane/observability/setup.py`:
- Around line 36-38: Centralize OTEL flag parsing in an exported
is_otel_enabled() function in setup.py, using _TRUTHY_VALUES so “on” is accepted
consistently. Replace the duplicated environment checks in celery.py’s Celery
log-correlation gate and the Django settings OTEL gating with this function,
preserving existing behavior for other values.
In `@apps/api/plane/settings/local.py`:
- Line 96: Accept “on” as a truthy OTEL_ENABLED value in the settings checks at
apps/api/plane/settings/local.py:96 and
apps/api/plane/settings/production.py:106, preserving the existing accepted
values. Keep docs/otel-api-observability/README.md:27 unchanged because “on”
should remain documented as supported.
In `@deployments/cli/community/docker-compose.yml`:
- Around line 64-73: Update the x-otel-env anchor to pass through
OTEL_TRACES_SAMPLER, OTEL_TRACES_SAMPLER_ARG, and OTEL_RESOURCE_ATTRIBUTES using
the corresponding environment-variable defaults, matching the documented
.env.example configuration.
---
Nitpick comments:
In `@apps/api/plane/observability/setup.py`:
- Around line 134-149: Update _instrument_libraries to isolate each instrumentor
call with its own try/except so failures from optional dependencies or
incompatible versions are caught and do not propagate through configure_otel or
block application startup. Continue attempting the remaining instrumentors after
an individual failure, and log each exception using the module’s existing
observability logging mechanism.
In `@apps/api/plane/settings/local.py`:
- Around line 96-105: Extract the duplicated OTEL logging mutation into a shared
helper such as extend_logging_config in plane.observability.logging, preserving
the OTEL_ENABLED gate and existing formatter, filter, and handler behavior.
Replace the inline blocks with helper calls in apps/api/plane/settings/local.py
lines 96-105 and apps/api/plane/settings/production.py lines 106-115.
In `@apps/api/plane/tests/unit/observability/conftest.py`:
- Around line 23-37: Update the isolate_otel_state fixture to also reset
observability setup’s _TRACER_PROVIDER and _METER_PROVIDER globals to their
unconfigured state via monkeypatch, alongside _CONFIGURED, so provider objects
cannot persist between tests.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: fafaa717-4b00-41a8-8a1d-ac68f6e903a7
📒 Files selected for processing (20)
apps/api/.env.exampleapps/api/manage.pyapps/api/plane/asgi.pyapps/api/plane/celery.pyapps/api/plane/observability/__init__.pyapps/api/plane/observability/logging.pyapps/api/plane/observability/setup.pyapps/api/plane/settings/local.pyapps/api/plane/settings/production.pyapps/api/plane/tests/unit/observability/__init__.pyapps/api/plane/tests/unit/observability/conftest.pyapps/api/plane/tests/unit/observability/test_logging.pyapps/api/plane/tests/unit/observability/test_setup.pyapps/api/plane/wsgi.pyapps/api/requirements/base.txtdeployments/aio/community/variables.envdeployments/cli/community/docker-compose.ymldeployments/cli/community/variables.envdocs/otel-api-observability/README.mddocs/otel-api-observability/otel-collector.yaml
…entors, pass sampler vars through compose - Extract shared is_otel_enabled() + extend_logging_config() into plane.observability.logging so the bootstrap, Celery, and Django LOGGING gates all accept the documented tokens (incl. 'on') identically. - Isolate each instrumentor in _instrument_libraries() so one failure can't block startup. - Pass OTEL_TRACES_SAMPLER / _ARG / OTEL_RESOURCE_ATTRIBUTES through the x-otel-env compose anchor (documented in .env.example). - Reset _TRACER_PROVIDER/_METER_PROVIDER in the test fixture.
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
apps/api/plane/observability/logging.py (1)
75-76: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winPreserve existing filters on log handlers.
Currently, this assignment overwrites any existing filters on the handlers. While the default
local.pyandproduction.pydon't define filters on their handlers, custom deployments or future additions (likerequire_debug_true) will have their existing filters silently removed when OpenTelemetry is enabled.Consider appending to the existing filters rather than overwriting them.
♻️ Proposed fix
for handler in logging_config["handlers"].values(): - handler["filters"] = ["trace_context"] + filters = handler.get("filters", []) + handler["filters"] = list(filters) + ["trace_context"]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/api/plane/observability/logging.py` around lines 75 - 76, Update the handler loop in the logging configuration setup to preserve each handler’s existing filters while adding the "trace_context" filter. Reuse the current filters when present, avoid duplicate entries if trace_context is already configured, and retain the existing behavior for handlers without filters.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@apps/api/plane/observability/logging.py`:
- Around line 75-76: Update the handler loop in the logging configuration setup
to preserve each handler’s existing filters while adding the "trace_context"
filter. Reuse the current filters when present, avoid duplicate entries if
trace_context is already configured, and retain the existing behavior for
handlers without filters.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: b7a0cf88-b8d7-4bd6-b998-5f5d8ac0cc05
📒 Files selected for processing (7)
apps/api/plane/celery.pyapps/api/plane/observability/logging.pyapps/api/plane/observability/setup.pyapps/api/plane/settings/local.pyapps/api/plane/settings/production.pyapps/api/plane/tests/unit/observability/conftest.pydeployments/cli/community/docker-compose.yml
🚧 Files skipped from review as they are similar to previous changes (4)
- deployments/cli/community/docker-compose.yml
- apps/api/plane/tests/unit/observability/conftest.py
- apps/api/plane/observability/setup.py
- apps/api/plane/celery.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds end-to-end OpenTelemetry (OTel) observability to the community API. Self-hosters can get OTLP-compatible APM for the Django API — a server span per request with DB / Redis / HTTP-client child spans, a span per Celery task (traceparent propagated through the queue), HTTP server metrics, and trace-correlated JSON logs — by setting two environment variables and pointing at any OTLP-compatible backend (Jaeger, Tempo, Datadog Agent, Honeycomb, Grafana Cloud, …).
Everything is off by default (
OTEL_ENABLED=0) and gated behindOTEL_ENABLED+OTEL_EXPORTER_OTLP_ENDPOINT, so existing deployments are unaffected until they opt in. The off path is byte-for-byte unchanged (no OTLP connections, log schema untouched).Scope is the Django API server. Node/browser/SSR instrumentation and browser-side telemetry are out of scope for this PR.
Highlights:
apps/api/plane/observability/package —configure_otel()(idempotent, gated) +flush_otel()+TraceContextFilter. Sets up aTracerProvider(parent-based 10% head sampler) andMeterProvider, selects grpc/http exporters byOTEL_EXPORTER_OTLP_PROTOCOL, instruments Django / Celery / Psycopg / Redis / Requests / HTTPX, and pins noisyopentelemetry.*loggers to WARNING.configure_otel()is called at the top ofwsgi.py,asgi.py,manage.py, andcelery.pybefore Django wires up, soDjangoInstrumentorpatches the WSGI/ASGI handler with zero view/serializer changes.LOGGINGdict (both settings files). Because the Celery workers attach their own JSON handlers (they don't route through Django'sdictConfig), a small gated filter incelery.pygives worker logs the sametrace_id/span_id/service_namefields — the off-path handler setup is unchanged.worker_process_shutdownflushes buffered spans/metrics (prefork children exit viaos._exitand skipatexit).opentelemetry-instrumentation-*/ OTLP-HTTP pins torequirements/base.txt(the 5 base OTel packages were already present).api/worker/beat-workerin thecli/communitycompose (notmigrator), and documents the vars in thecli/communityandaio/communityvariables.envfiles.docs/otel-api-observability/.Type of Change
Screenshots and Media (if applicable)
Test Scenarios
Off by default (no regression):
OTEL_ENABLEDunset → normal startup, no OTLP connection attempts, and JSON logs keep their existing schema (notrace_id/span_id). Verified:manage.py checkis clean, and the full test suite collects 494 tests with no import errors.Unit tests (
plane/tests/unit/observability/, 32 tests, all green):0, enabled-without-endpoint warns and returns, truthy tokens1/true/yes/on), provider + instrumentation wiring,PsycopgInstrumentor(enable_commenter=False), default application + operator overrides, quiet loggers, idempotency, and protocol-aware exporter selection (grpc default, http/protobuf → http). PlusTraceContextFilterbehavior.Enabled (
OTEL_ENABLED=1+ collector reachable):http.route/http.status_codeplus child Postgres/Redis/HTTP-client spans;http.server.durationmetrics arrive.trace_id/span_id/service_name.OTEL_ENABLED=1withOTEL_EXPORTER_OTLP_ENDPOINTunset → exactly one WARNING at boot; the app continues without instrumentation.Deployment:
docker compose -f deployments/cli/community/docker-compose.yml configrenders theOTEL_*vars onapi/worker/beat-workeronly (notmigrator).References
docs/otel-api-observability/README.md,docs/otel-api-observability/otel-collector.yaml🤖 PR description generated with Claude Code
Summary by CodeRabbit
New Features
Documentation
Tests