diff --git a/docs/superpowers/followups/2026-06-03-otel-span-linkage-followup.md b/docs/superpowers/followups/2026-06-03-otel-span-linkage-followup.md new file mode 100644 index 00000000000..a1fd6d426e6 --- /dev/null +++ b/docs/superpowers/followups/2026-06-03-otel-span-linkage-followup.md @@ -0,0 +1,80 @@ +# Follow-up: server span links to the operation span, not the command span + +- **Ticket:** [DRIVERS-3454](https://jira.mongodb.org/browse/DRIVERS-3454) +- **Status:** Known limitation of the POC — fix after the POC is validated. +- **Date:** 2026-06-03 +- **Related:** `docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md`, + `docs/superpowers/specs/2026-06-02-otel-e2e-jaeger-design.md` + +## Symptom (observed in Jaeger) + +For a single client operation, the exported trace nests as: + +``` +insert test.ping (client OPERATION span, id=4d5f339b) + ├─ insert (client COMMAND span, id=d49879c6, parent=4d5f339b) + └─ insert (mongod SERVER span, id=b89795e6, parent=4d5f339b) +``` + +The `mongod` server span is parented to the client **operation** span, making it a **sibling** of +the client **command** span. The expected/ideal hierarchy is +`operation span → command span → server span` (server span as a child of the command span). Sibling +spans render in Jaeger by start time, which is why the server span can appear "before" the command +span. + +## Root cause + +The OP_MSG trace-context section is written during message **encoding**, which happens **before** the +per-command span is created. At encode time the only span available in the `OperationContext` is the +operation span, so that is the context propagated on the wire. + +Code path (current `nabil_otel_context`): + +1. `InternalStreamConnection.sendAndReceiveInternal` (~line 445): `message.encode(bsonOutput, operationContext)` + runs first. `CommandMessage.writeOtelTraceContextSection` executes inside `encode` and reads + `operationContext.getTracingSpan()`. +2. `operationContext.getTracingSpan()` returns the **operation** span, set earlier by + `TracingManager.createOperationSpan` → `operationContext.setTracingSpan(span)` + (`TracingManager.java` ~line 284). +3. `InternalStreamConnection.sendAndReceiveInternal` (~lines 446–455): the **command** span is created + **after** encode via `createTracingSpan(...)`, parented to the operation span + (`TracingManager.java` ~lines 191–192: `addSpan(MONGODB_COMMAND, …, operationSpan.context())`). + +So the `traceparent` carries the operation span's id; the server attaches its span to the operation +span; the command span is a separate sibling under the same operation span. + +## Why the command span is the better parent + +The command span represents an individual wire RPC, and one operation can issue several commands +(retries, `getMore`, split bulk batches). Parenting each server span under the corresponding command +span ties it to the exact RPC that produced it. Under the operation span, multiple server spans pile +up as indistinguishable siblings. + +## Why it is not a trivial reorder (chicken-and-egg) + +`createTracingSpan` currently derives the command name from `message.getCommandDocument(bsonOutput)`, +i.e. it reads the **already-encoded** bytes. So the command span cannot simply be created before +`encode()` without changing how it obtains the command name. + +## Candidate fixes (after the POC) + +1. **Create the command span before `encode()` using the in-memory command name.** The command name + is available from `CommandMessage`'s in-memory command `BsonDocument` (its first key), without + re-parsing the encoded output. Create the command span first, then have + `writeOtelTraceContextSection` propagate the command span's context. Requires refactoring + `createTracingSpan` so the name/parent no longer depend on `getCommandDocument(bsonOutput)`. + *Preferred — keeps the section inside encoding and yields the correct parent.* + +2. **Append the section after the command span is created.** Move section-writing out of + `CommandMessage.writeOpMsg` into a post-`createTracingSpan` step in `InternalStreamConnection`, + appending the kind-3 section and re-backpatching the OP_MSG message length. More invasive to wire + framing; risks interacting with compression and `getCommandDocument` re-parsing. + +3. **Accept operation-span parenting for the POC.** The trace is still correctly linked end to end — + just one level shallower than ideal. No change. + +## Decision + +Defer. The POC's goal (end-to-end propagation visible in Jaeger) is met with operation-span +parenting. Revisit with option 1 when productionizing, alongside removing the test-only +`OtelTracePropagationTestToggle` and adding the real `hello` `tracingSupport` negotiation. diff --git a/docs/superpowers/plans/2026-06-02-otel-e2e-jaeger.md b/docs/superpowers/plans/2026-06-02-otel-e2e-jaeger.md new file mode 100644 index 00000000000..eb55898a016 --- /dev/null +++ b/docs/superpowers/plans/2026-06-02-otel-e2e-jaeger.md @@ -0,0 +1,507 @@ +# End-to-end OTel Trace Visualization (driver toggle + Spring Boot + Jaeger) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** See a single client→server trace in Jaeger: a Spring Boot app's MongoDB operation, the driver's propagated `traceparent` (forced on via a test toggle), and the POC server's child span, all exported to one Jaeger instance. + +**Architecture:** Add a test-only static toggle in the driver to bypass the server-capability gate; publish the driver to Maven Local; run Jaeger in Docker; reconnect the POC mongod to export OTLP to Jaeger; build a Spring Boot app (Spring Data MongoDB + Micrometer→OTel→Jaeger) that wires the driver's internal tracing via `MongoClientSettingsBuilderCustomizer`, flips the toggle, and exposes a REST trigger. + +**Tech Stack:** Java 8 driver-core (toggle), Gradle (publish), Docker (Jaeger + mongod), Spring Boot 3.3.x / Java 17 / Maven (app), Micrometer Tracing + OpenTelemetry OTLP. + +**Spec:** `docs/superpowers/specs/2026-06-02-otel-e2e-jaeger-design.md` + +> **Standing instruction:** the user wants driver changes STAGED, not committed. For driver tasks, replace any `git commit` with `git add` (stage only). The Spring Boot app lives OUTSIDE the driver repo (`~/MongoDB/otel-poc-server/otel-poc-client/`) and is not added to driver git at all. + +> **Environment facts (already true):** +> - POC mongod binaries extracted at `~/MongoDB/otel-poc-server/dist-test/`; Docker image `otel-poc-mongod` built; container `otel-poc-mongod-run` currently running on the default bridge with port 27017. +> - `featureFlagTracing` is enabled in that build; `opentelemetryHttpEndpoint` is a startup setParameter. +> - Driver OP_MSG kind-3 propagation POC is already staged (gate lives in `CommandMessage.writeOtelTraceContextSection`). + +--- + +## File map + +| File | Responsibility | Change | +|---|---|---| +| `driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java` | Test-only force switch | Create | +| `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java` | OP_MSG encoding gate | Modify (one clause) | +| `driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java` | Toggle behavior test | Modify (add 1 test) | +| `~/MongoDB/otel-poc-server/otel-poc-client/pom.xml` | App build, driver override, mavenLocal | Create | +| `~/MongoDB/otel-poc-server/otel-poc-client/src/main/java/com/example/otelpoc/OtelPocApplication.java` | Main + toggle activation | Create | +| `~/MongoDB/otel-poc-server/otel-poc-client/src/main/java/com/example/otelpoc/MongoTracingConfig.java` | `MongoClientSettingsBuilderCustomizer` wiring | Create | +| `~/MongoDB/otel-poc-server/otel-poc-client/src/main/java/com/example/otelpoc/PingController.java` | REST trigger | Create | +| `~/MongoDB/otel-poc-server/otel-poc-client/src/main/resources/application.properties` | URI, sampling, OTLP endpoint | Create | + +--- + +## Task 1: Driver test-only force toggle + +**Files:** +- Create: `driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java` +- Modify: `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java` +- Test: `driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java` + +- [ ] **Step 1: Write the failing test** + +Add this method to `CommandMessageOtelTraceContextTest` (it reuses the existing `buildCommandMessage`, `buildOperationContext`, `encodeToBytes`, `containsOtelSection`, and `TRACEPARENT` helpers/constants already in the class). Add the import `import com.mongodb.internal.observability.micrometer.OtelTracePropagationTestToggle;` at the top. + +```java + @Test + void writesSectionWhenForcedEvenIfCapabilityAbsent() { + OtelTracePropagationTestToggle.FORCE_PROPAGATION = true; + try { + CommandMessage message = buildCommandMessage(false); // server did NOT advertise tracingSupport + TraceContext traceContext = () -> TRACEPARENT; + Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); + OperationContext operationContext = buildOperationContext(span); + + byte[] encoded = encodeToBytes(message, operationContext); + + assertTrue(containsOtelSection(encoded, TRACEPARENT), + "With FORCE_PROPAGATION the section must be sent even when the server did not advertise support"); + } finally { + OtelTracePropagationTestToggle.FORCE_PROPAGATION = false; + } + } +``` + +- [ ] **Step 2: Run the test to verify it fails to compile** + +Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.connection.CommandMessageOtelTraceContextTest.writesSectionWhenForcedEvenIfCapabilityAbsent" -PskipCryptVerify=true` +Expected: FAIL — `OtelTracePropagationTestToggle` does not exist. + +- [ ] **Step 3: Create the toggle class** + +Create `driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java`: + +```java +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.mongodb.internal.observability.micrometer; + +/** + * TEST-ONLY switch (DRIVERS-3454): when {@code true}, the driver writes the OP_MSG OpenTelemetry + * trace-context section even if the server did not advertise {@code tracingSupport} in its + * {@code hello} response. The sampled-{@code traceparent} requirement still applies. + * + *

This exists only to exercise end-to-end propagation against a server that does not yet advertise + * the capability. Remove before any production use.

+ */ +public final class OtelTracePropagationTestToggle { + public static volatile boolean FORCE_PROPAGATION = false; + + private OtelTracePropagationTestToggle() { + } +} +``` + +- [ ] **Step 4: Relax the capability gate in CommandMessage** + +In `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java`, add the import (alphabetically within the `com.mongodb.internal.observability.micrometer` group, next to the existing `Span` import): + +```java +import com.mongodb.internal.observability.micrometer.OtelTracePropagationTestToggle; +``` + +In `writeOtelTraceContextSection`, change the first guard from: + +```java + if (!getSettings().isTracingSupported()) { + return; + } +``` + +to: + +```java + if (!getSettings().isTracingSupported() && !OtelTracePropagationTestToggle.FORCE_PROPAGATION) { + return; + } +``` + +- [ ] **Step 5: Run the test to verify it passes** + +Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.connection.CommandMessageOtelTraceContextTest" -PskipCryptVerify=true` +Expected: PASS (all cases, including the new one and the existing `omitsSectionWhenCapabilityAbsent` which does NOT set the toggle, so it still passes). + +- [ ] **Step 6: Stage (do NOT commit)** + +```bash +git add driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java \ + driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java \ + driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java +``` + +--- + +## Task 2: Publish the driver to Maven Local + +**Files:** none (build action) + +- [ ] **Step 1: Publish all modules as 5.9.0-SNAPSHOT** + +Run from the repo root: +```bash +./gradlew publishToMavenLocal -PskipCryptVerify=true +``` +Expected: BUILD SUCCESSFUL. + +- [ ] **Step 2: Verify the artifacts landed in the local repo** + +Run: +```bash +ls ~/.m2/repository/org/mongodb/mongodb-driver-sync/5.9.0-SNAPSHOT/ \ + ~/.m2/repository/org/mongodb/mongodb-driver-core/5.9.0-SNAPSHOT/ \ + ~/.m2/repository/org/mongodb/bson/5.9.0-SNAPSHOT/ +``` +Expected: each directory contains a `*-5.9.0-SNAPSHOT.jar`. If `bson-record-codec` is referenced transitively, confirm it too: +```bash +ls ~/.m2/repository/org/mongodb/bson-record-codec/5.9.0-SNAPSHOT/ 2>/dev/null || echo "(bson-record-codec not published — fine unless the app fails to resolve it)" +``` + +--- + +## Task 3: Start Jaeger in Docker + +**Files:** none (infra) + +- [ ] **Step 1: Create the shared network and run Jaeger** + +```bash +docker network create otel-poc-net 2>/dev/null || true +docker rm -f jaeger 2>/dev/null || true +docker run -d --name jaeger --network otel-poc-net \ + -e COLLECTOR_OTLP_ENABLED=true \ + -p 16686:16686 -p 4317:4317 -p 4318:4318 \ + jaegertracing/all-in-one:1.62.0 +``` + +- [ ] **Step 2: Verify the UI is up** + +```bash +until curl -sf http://localhost:16686/ >/dev/null; do sleep 1; done; echo "Jaeger UI reachable" +``` +Expected: `Jaeger UI reachable`. + +--- + +## Task 4: Reconnect mongod to export OTLP to Jaeger + +**Files:** none (infra) + +- [ ] **Step 1: Recreate the mongod container on the shared network with the OTLP endpoint** + +```bash +cd ~/MongoDB/otel-poc-server +docker rm -f otel-poc-mongod-run 2>/dev/null || true +docker run -d --name otel-poc-mongod-run --platform linux/arm64 --network otel-poc-net \ + -v "$PWD/dist-test:/opt/mongo:ro" \ + -v "$PWD/data:/data/db" \ + -p 27017:27017 \ + otel-poc-mongod \ + /opt/mongo/bin/mongod --dbpath /data/db --bind_ip_all --port 27017 \ + --setParameter opentelemetryHttpEndpoint=http://jaeger:4318/v1/traces \ + --setParameter openTelemetryExportIntervalMillis=1000 +``` + +- [ ] **Step 2: Wait for readiness and confirm the endpoint was accepted** + +```bash +until docker logs otel-poc-mongod-run 2>&1 | grep -q "Waiting for connections" || ! docker ps -q --filter name=otel-poc-mongod-run | grep -q .; do sleep 1; done +docker ps --filter name=otel-poc-mongod-run --format '{{.Status}}' +docker logs otel-poc-mongod-run 2>&1 | grep -iE "opentelemetry|otel|trace|export|endpoint" | tail -15 +``` +Expected: container `Up`; no fatal OTLP/endpoint parse error in logs. + +- [ ] **Step 3: Fallback if the endpoint form is rejected** + +If mongod refuses to start or logs an OTLP endpoint error, re-run Step 1 replacing the endpoint with the file exporter (already proven working): +``` + --setParameter opentelemetryTraceDirectory=/data/db/otel-traces +``` +and note in the final report that server spans are inspected as JSONL under `~/MongoDB/otel-poc-server/data/otel-traces/` rather than in the Jaeger UI. Then continue. + +--- + +## Task 5: Scaffold the Spring Boot app (pom + properties + main) + +**Files (all under `~/MongoDB/otel-poc-server/otel-poc-client/`, OUTSIDE the driver repo):** +- Create: `pom.xml` +- Create: `src/main/resources/application.properties` +- Create: `src/main/java/com/example/otelpoc/OtelPocApplication.java` + +- [ ] **Step 1: Create the project directory and Maven wrapper** + +```bash +mkdir -p ~/MongoDB/otel-poc-server/otel-poc-client/src/main/java/com/example/otelpoc +mkdir -p ~/MongoDB/otel-poc-server/otel-poc-client/src/main/resources +``` + +- [ ] **Step 2: Create `pom.xml`** + +```xml + + + 4.0.0 + + + org.springframework.boot + spring-boot-starter-parent + 3.3.5 + + + + com.example + otel-poc-client + 0.0.1-SNAPSHOT + + + 17 + + 5.9.0-SNAPSHOT + + + + + org.springframework.boot + spring-boot-starter-web + + + org.springframework.boot + spring-boot-starter-data-mongodb + + + org.springframework.boot + spring-boot-starter-actuator + + + io.micrometer + micrometer-tracing-bridge-otel + + + io.opentelemetry + opentelemetry-exporter-otlp + + + + + + + org.springframework.boot + spring-boot-maven-plugin + + + + +``` + +> Maven consults the local `~/.m2` repository by default, so the `5.9.0-SNAPSHOT` artifacts published in Task 2 resolve without any extra `` entry. + +- [ ] **Step 3: Create `src/main/resources/application.properties`** + +```properties +spring.application.name=otel-poc-client +server.port=8080 +spring.data.mongodb.uri=mongodb://localhost:27017/test +management.tracing.sampling.probability=1.0 +management.otlp.tracing.endpoint=http://localhost:4318/v1/traces +# Suppress Spring's auto Mongo command instrumentation: the driver's internal tracer is the ONLY +# source of Mongo command spans (no duplicate/competing spans). +management.metrics.enable.mongodb=false +spring.autoconfigure.exclude=org.springframework.boot.actuate.autoconfigure.metrics.mongo.MongoMetricsAutoConfiguration +``` + +- [ ] **Step 4: Create the main class with toggle activation** + +`src/main/java/com/example/otelpoc/OtelPocApplication.java`: + +```java +package com.example.otelpoc; + +import com.mongodb.internal.observability.micrometer.OtelTracePropagationTestToggle; +import org.springframework.boot.SpringApplication; +import org.springframework.boot.autoconfigure.SpringBootApplication; + +@SpringBootApplication +public class OtelPocApplication { + public static void main(String[] args) { + // TEST-ONLY: force the driver to send the OP_MSG trace-context section even though this POC + // server does not advertise tracingSupport in hello. Set before any MongoClient is used. + OtelTracePropagationTestToggle.FORCE_PROPAGATION = true; + SpringApplication.run(OtelPocApplication.class, args); + } +} +``` + +- [ ] **Step 5: Generate the Maven wrapper** + +```bash +cd ~/MongoDB/otel-poc-server/otel-poc-client +mvn -N wrapper:wrapper -Dmaven=3.9.9 2>&1 | tail -3 || echo "(if 'mvn' is absent, install it or run with a system Maven in Task 7)" +``` +Expected: `mvnw` and `.mvn/` created. If no system `mvn` exists, skip — Task 7 notes the alternative. + +--- + +## Task 6: Tracing wiring bean + REST trigger + +**Files (under `~/MongoDB/otel-poc-server/otel-poc-client/src/main/java/com/example/otelpoc/`):** +- Create: `MongoTracingConfig.java` +- Create: `PingController.java` + +- [ ] **Step 1: Create the `MongoClientSettingsBuilderCustomizer` wiring** + +`MongoTracingConfig.java`: + +```java +package com.example.otelpoc; + +import com.mongodb.observability.micrometer.MicrometerObservabilitySettings; +import io.micrometer.observation.ObservationRegistry; +import org.springframework.boot.autoconfigure.mongo.MongoClientSettingsBuilderCustomizer; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +@Configuration +public class MongoTracingConfig { + + /** + * Activates the driver's INTERNAL tracing (the path that sets operationContext.tracingSpan, which + * traceParent() reads for OP_MSG propagation) and routes its spans through Spring's OTel-bridged + * ObservationRegistry so they export to Jaeger. + */ + @Bean + public MongoClientSettingsBuilderCustomizer tracingCustomizer(final ObservationRegistry registry) { + return builder -> builder.observabilitySettings( + MicrometerObservabilitySettings.builder() + .observationRegistry(registry) + .build()); + } +} +``` + +- [ ] **Step 2: Create the REST trigger** + +`PingController.java`: + +```java +package com.example.otelpoc; + +import org.bson.Document; +import org.springframework.data.mongodb.core.MongoTemplate; +import org.springframework.web.bind.annotation.GetMapping; +import org.springframework.web.bind.annotation.RestController; + +import java.util.Date; + +@RestController +public class PingController { + + private final MongoTemplate mongoTemplate; + + public PingController(final MongoTemplate mongoTemplate) { + this.mongoTemplate = mongoTemplate; + } + + @GetMapping("/ping") + public String ping() { + mongoTemplate.getCollection("ping").insertOne(new Document("at", new Date())); + long count = mongoTemplate.getCollection("ping").countDocuments(); + return "ok, count=" + count + "\n"; + } +} +``` + +- [ ] **Step 3: Compile the app to verify wiring resolves** + +```bash +cd ~/MongoDB/otel-poc-server/otel-poc-client +./mvnw -q -DskipTests compile 2>&1 | tail -20 || mvn -q -DskipTests compile 2>&1 | tail -20 +``` +Expected: BUILD SUCCESS. If compilation fails to resolve `MicrometerObservabilitySettings` or `OtelTracePropagationTestToggle`, confirm Task 2 published `mongodb-driver-core` (the toggle and settings live there) and that `mongodb.version` is `5.9.0-SNAPSHOT`. + +--- + +## Task 7: End-to-end run and Jaeger verification + +**Files:** none (run + verify). Prereqs: Tasks 1–6 done; Jaeger (Task 3) and mongod-on-network (Task 4) running. + +- [ ] **Step 1: Start the Spring Boot app** + +```bash +cd ~/MongoDB/otel-poc-server/otel-poc-client +./mvnw spring-boot:run 2>&1 | tee /tmp/otel-poc-client.log & +# wait until it is listening on 8080 +until curl -sf http://localhost:8080/ping >/dev/null 2>&1; do sleep 2; done +echo "app up" +``` +(If there is no `mvnw`, use `mvn spring-boot:run`.) Expected: `app up`. + +- [ ] **Step 2: Generate a traced operation** + +```bash +curl -s http://localhost:8080/ping +``` +Expected: `ok, count=`. + +- [ ] **Step 3: Confirm the client exported a trace to Jaeger** + +```bash +sleep 3 +curl -s "http://localhost:16686/api/services" | tr ',' '\n' | grep -i "otel-poc-client" && echo "client service present in Jaeger" +``` +Expected: `otel-poc-client` appears in the services list. + +- [ ] **Step 4: Confirm a linked server span exists in the same trace** + +```bash +TRACE_JSON=$(curl -s "http://localhost:16686/api/traces?service=otel-poc-client&limit=1") +echo "$TRACE_JSON" | python3 -c "import sys,json; d=json.load(sys.stdin); \ +spans=d['data'][0]['spans']; procs=d['data'][0]['processes']; \ +print('services in trace:', sorted({procs[s['processID']]['serviceName'] for s in spans})); \ +print('span names:', [s['operationName'] for s in spans])" +``` +Expected: the services set includes BOTH `otel-poc-client` and the mongod service (e.g. `mongod`), proving the client span and the server span share one trace. + +- [ ] **Step 5: Visual confirmation** + +Open **http://localhost:16686**, select service `otel-poc-client`, open the most recent trace. Confirm a span tree: HTTP `GET /ping` → driver mongo command span → `mongod` server span (same traceId; server span's parent is the client mongo span). + +- [ ] **Step 6: Negative check (capability gate still works)** + +Stop the app, set the toggle off by editing `OtelPocApplication.main` to `FORCE_PROPAGATION = false` (or comment the line), `./mvnw spring-boot:run` again, `curl /ping`, and confirm in Jaeger the new client trace has **no** `mongod` span (server starts its own unrelated trace). Restore the line afterward. + +- [ ] **Step 7: Record the result** + +If server spans appear in Jaeger: success — note it. If Task 4 used the file-exporter fallback, instead show the server span and matching traceId from the latest `~/MongoDB/otel-poc-server/data/otel-traces/*.jsonl` (grep for the client's traceId) and note that server spans are file-based rather than in the Jaeger UI. + +--- + +## Self-Review + +**Spec coverage:** +- §3 toggle → Task 1 (class + gate + test). §4 publish → Task 2. §5 Jaeger → Task 3. §6 mongod OTLP (+ file fallback) → Task 4 (Steps 1–3). §7 app: deps/version override → Task 5 Step 2; properties → Task 5 Step 3; toggle activation → Task 5 Step 4; `MongoClientSettingsBuilderCustomizer` wiring → Task 6 Step 1; REST trigger → Task 6 Step 2. §8 run & verification → Task 7. §10 risks: OTLP endpoint form → Task 4 Step 3 fallback; duplicate Spring command spans → acceptable, see note below; driver-version override → Task 5 Step 2 + Task 6 Step 3 troubleshooting. + +**Duplicate-span suppression (spec §7, firm):** Spring's auto Mongo command instrumentation is suppressed in `application.properties` (Task 5 Step 3) via `spring.autoconfigure.exclude=…MongoMetricsAutoConfiguration` + `management.metrics.enable.mongodb=false`, so the driver's internal tracer is the only source of Mongo command spans. If startup logs show the excluded class still contributing (version drift), the implementer confirms the exact actuator Mongo autoconfig class for Spring Boot 3.3.5 and excludes that instead. + +**Placeholder scan:** no TBD/TODO; every code/file step has full content; the only conditional is the documented OTLP-endpoint fallback (Task 4 Step 3) and the no-`mvnw` alternative (system `mvn`). + +**Type/name consistency:** `OtelTracePropagationTestToggle.FORCE_PROPAGATION` (Tasks 1, 5); `MicrometerObservabilitySettings.builder().observationRegistry(...)` and `.observabilitySettings(...)` (Task 6, matches driver API verified in spec §7); package `com.example.otelpoc` and class names match the file map; `mongodb.version=5.9.0-SNAPSHOT` consistent (Tasks 2, 5, 6). + +**Staging vs commit:** Task 1 stages only (driver repo); Tasks 5–6 create files outside the driver repo (never added to driver git). diff --git a/docs/superpowers/plans/2026-06-02-otel-opmsg-propagation-poc.md b/docs/superpowers/plans/2026-06-02-otel-opmsg-propagation-poc.md new file mode 100644 index 00000000000..4c443b34720 --- /dev/null +++ b/docs/superpowers/plans/2026-06-02-otel-opmsg-propagation-poc.md @@ -0,0 +1,815 @@ +# OTel Trace-Context Propagation over OP_MSG (driver-sync POC) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make the synchronous MongoDB Java driver inject the active client span's W3C `traceparent` into outgoing OP_MSG messages as a new section (kind 3), but only when the server advertised support — validating the wire contract from the DRIVERS-3454 spec. + +**Architecture:** The driver already builds OP_MSG sections in `CommandMessage.writeOpMsg()` and creates per-operation Micrometer spans accessible via `OperationContext.getTracingSpan()`. We (1) expose the W3C `traceparent` from the span's `TraceContext`, (2) read a `tracingSupport` capability from the `hello` handshake into `ConnectionDescription` → `MessageSettings`, and (3) write section kind 3 in `writeOpMsg()` gated on both the capability and a non-null sampled traceparent. Validation is phased: automated OP_MSG encode/parse round-trip tests (Phase 1), then an optional manual end-to-end runbook against the server POC (Phase 2). + +**Tech Stack:** Java 8 (driver-core baseline), Micrometer Observation (runtime) + Micrometer Tracing (extraction), JUnit 5 + Mockito, Gradle. + +**Spec:** `docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md` + +> **POC scope guards (do NOT exceed):** sync path only; no reactive/async; no `mongos`/load-balanced propagation; `tracestate` is pass-through only; no sampling rework. All code is internal — no breaking public-API change (only additive withers/getters). + +--- + +## File map + +| File | Responsibility | Change | +|---|---|---| +| `driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java` | Trace context abstraction | Add `@Nullable String traceParent()` | +| `driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java` | Micrometer impl | Implement `traceParent()` from the `Observation`'s tracing context | +| `driver-core/build.gradle.kts` | Module deps | Add `micrometer-tracing` as `optionalImplementation` | +| `gradle/libs.versions.toml` | Dependency catalog | Add `micrometer-tracing` library coordinate (main) | +| `driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java` | hello parsing | Parse `tracingSupport`, set on `ConnectionDescription` | +| `driver-core/src/main/com/mongodb/connection/ConnectionDescription.java` | Connection capabilities | Add `tracingSupport` field, `withTracingSupport()`, `isTracingSupport()` | +| `driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java` | Per-message settings | Add `tracingSupported` field + builder method + getter | +| `driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java` | Builds `MessageSettings` | Propagate `tracingSupport` from `ConnectionDescription` | +| `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java` | OP_MSG encoding | Add section kind 3, write it gated on settings + traceparent | +| `docs/superpowers/runbooks/otel-opmsg-e2e.md` | Phase 2 manual validation | New runbook | + +--- + +## Task 1: Expose `traceParent()` on the `TraceContext` abstraction + +**Files:** +- Modify: `driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java` + +- [ ] **Step 1: Add the method to the interface and make `EMPTY` return null** + +Replace the interface body so `EMPTY` implements the new method: + +```java +@SuppressWarnings("InterfaceIsType") +public interface TraceContext { + TraceContext EMPTY = new TraceContext() { + @Override + public String traceParent() { + return null; + } + }; + + /** + * The W3C {@code traceparent} string for this context + * ({@code 00-<32hex traceId>-<16hex spanId>-<2hex flags>}), + * or {@code null} if unavailable or the span is not sampled. + */ + @Nullable + String traceParent(); +} +``` + +Add the import `import com.mongodb.lang.Nullable;` below the package statement. + +- [ ] **Step 2: Compile to verify the interface change is consistent** + +Run: `./gradlew :driver-core:compileJava` +Expected: SUCCESS. (`Span.EMPTY.context()` returns `TraceContext.EMPTY`, which now implements `traceParent()`.) If any other anonymous `TraceContext` implementers exist they will fail to compile — fix each by adding `traceParent()` returning `null`. + +- [ ] **Step 3: Commit** + +```bash +git add driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java +git commit -m "DRIVERS-3454: add traceParent() to TraceContext" +``` + +--- + +## Task 2: Implement `traceParent()` in the Micrometer tracer + +The driver only has the Micrometer **Observation** at runtime; the W3C trace/span IDs live in Micrometer **Tracing**, which attaches a `TracingObservationHandler.TracingContext` to the observation's context when a tracing bridge is configured. We read that. + +**Files:** +- Modify: `gradle/libs.versions.toml` +- Modify: `driver-core/build.gradle.kts` +- Modify: `driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java` +- Test: `driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java` + +- [ ] **Step 1: Add the main `micrometer-tracing` library coordinate** + +In `gradle/libs.versions.toml`, under `[libraries]` add (the `micrometer-tracing` version `1.6.0-M3` already exists under `[versions]`): + +```toml +micrometer-tracing = { module = "io.micrometer:micrometer-tracing", version.ref = "micrometer-tracing" } +``` + +- [ ] **Step 2: Add it as an optional dependency of driver-core** + +In `driver-core/build.gradle.kts`, directly after line 59 (`optionalImplementation(libs.micrometer.observation)`), add: + +```kotlin +optionalImplementation(libs.micrometer.tracing) +``` + +(The existing `"io.micrometer.*;resolution:=optional"` OSGi import on line 105 already covers the new package.) + +- [ ] **Step 3: Write the failing test** + +Create `driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java`: + +```java +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.mongodb.internal.observability.micrometer; + +import io.micrometer.observation.ObservationRegistry; +import io.micrometer.tracing.test.simple.SimpleTracer; +import io.micrometer.tracing.handler.DefaultTracingObservationHandler; +import com.mongodb.observability.micrometer.MongodbObservation; +import org.junit.jupiter.api.Test; + +import java.util.regex.Pattern; + +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class MicrometerTraceParentTest { + private static final Pattern TRACEPARENT = + Pattern.compile("00-[0-9a-f]{32}-[0-9a-f]{16}-[0-9a-f]{2}"); + + @Test + void returnsTraceParentForSampledSpan() { + ObservationRegistry registry = ObservationRegistry.create(); + SimpleTracer tracer = new SimpleTracer(); + registry.observationConfig().observationHandler(new DefaultTracingObservationHandler(tracer)); + + MicrometerTracer micrometerTracer = new MicrometerTracer(registry, false, 1000, null); + Span span = micrometerTracer.nextSpan(MongodbObservation.COMMAND_OBSERVATION, "find", null, null); + span.openScope(); + try { + String traceParent = span.context().traceParent(); + assertNotNull(traceParent); + assertTrue(TRACEPARENT.matcher(traceParent).matches(), traceParent); + } finally { + span.closeScope(); + span.end(); + } + } + + @Test + void returnsNullWhenNoTracingBridgeConfigured() { + ObservationRegistry registry = ObservationRegistry.create(); + MicrometerTracer micrometerTracer = new MicrometerTracer(registry, false, 1000, null); + Span span = micrometerTracer.nextSpan(MongodbObservation.COMMAND_OBSERVATION, "find", null, null); + span.openScope(); + try { + assertNull(span.context().traceParent()); + } finally { + span.closeScope(); + span.end(); + } + } +} +``` + +> Note: confirm the enum constant name in `MongodbObservation` (e.g. `COMMAND_OBSERVATION`); if it differs, use the actual command-span constant. `SimpleTracer`/`DefaultTracingObservationHandler` come from the test-scoped `micrometer-tracing-integration-test` dependency already present. + +- [ ] **Step 4: Run the test to verify it fails to compile/fails** + +Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.observability.micrometer.MicrometerTraceParentTest"` +Expected: FAIL — `MicrometerTraceContext` does not yet implement `traceParent()`. + +- [ ] **Step 5: Implement `traceParent()` in `MicrometerTraceContext`** + +In `MicrometerTracer.java`, add imports: + +```java +import io.micrometer.tracing.TraceContext; +import io.micrometer.tracing.handler.TracingObservationHandler; +``` + +> The driver's own type is also named `TraceContext`; reference the Micrometer one by its fully-qualified name in code to avoid the clash (do NOT add the conflicting import). Use `io.micrometer.tracing.TraceContext` inline. + +Replace the `MicrometerTraceContext` inner class with: + +```java + /** + * Represents a Micrometer-based trace context. + */ + private static class MicrometerTraceContext implements TraceContext { + @Nullable + private final Observation observation; + + MicrometerTraceContext(@Nullable final Observation observation) { + this.observation = observation; + } + + @Override + @Nullable + public String traceParent() { + if (observation == null) { + return null; + } + TracingObservationHandler.TracingContext tracingContext = + observation.getContextView().getOrNull(TracingObservationHandler.TracingContext.class); + if (tracingContext == null || tracingContext.getSpan() == null) { + return null; + } + io.micrometer.tracing.TraceContext ctx = tracingContext.getSpan().context(); + if (ctx == null || ctx.traceId() == null || ctx.spanId() == null) { + return null; + } + Boolean sampled = ctx.sampled(); + if (sampled == null || !sampled) { + return null; + } + return "00-" + ctx.traceId() + "-" + ctx.spanId() + "-01"; + } + } +``` + +> `traceId()`/`spanId()` from Micrometer Tracing are already lowercase hex of the correct length. We emit flags `01` (sampled) because we only propagate sampled spans, matching spec §3.3. + +Note: the `import io.micrometer.tracing.TraceContext;` added above is unused if you reference it fully-qualified — remove it and keep only the `TracingObservationHandler` import to satisfy the no-unused-import check. + +- [ ] **Step 6: Run the test to verify it passes** + +Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.observability.micrometer.MicrometerTraceParentTest"` +Expected: PASS (both tests). + +- [ ] **Step 7: Commit** + +```bash +git add gradle/libs.versions.toml driver-core/build.gradle.kts \ + driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java \ + driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java +git commit -m "DRIVERS-3454: extract W3C traceparent from Micrometer span" +``` + +--- + +## Task 3: Read `tracingSupport` from `hello` into `ConnectionDescription` + +**Files:** +- Modify: `driver-core/src/main/com/mongodb/connection/ConnectionDescription.java` +- Modify: `driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java` +- Test: `driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java` + +> `ConnectionDescription` is public API. The change is **additive only** (new wither + getter; existing constructors keep delegating with `tracingSupport=false`), so it is binary-compatible. Do NOT change any existing public constructor signature. + +- [ ] **Step 1: Read the actual `ConnectionDescription` constructor chain and the `withServerType` wither** + +Run: `sed -n '40,210p' driver-core/src/main/com/mongodb/connection/ConnectionDescription.java` +Note the most-derived constructor (the one with `serviceId` + all fields) and how `withConnectionId`/`withServiceId`/`withServerType` build a copy. You will mirror that pattern exactly. + +- [ ] **Step 2: Write the failing test** + +Create `driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java`: + +```java +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.mongodb.internal.connection; + +import com.mongodb.ServerAddress; +import com.mongodb.connection.ClusterConnectionMode; +import com.mongodb.connection.ConnectionDescription; +import com.mongodb.connection.ConnectionId; +import com.mongodb.connection.ServerId; +import org.bson.BsonDocument; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class DescriptionHelperTracingTest { + private static final ConnectionId CONNECTION_ID = + new ConnectionId(new ServerId(new com.mongodb.connection.ClusterId(), new ServerAddress())); + + private static BsonDocument hello(final boolean tracing) { + BsonDocument doc = BsonDocument.parse( + "{ ok: 1, ismaster: true, maxWireVersion: 25, minWireVersion: 0," + + " maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, maxWriteBatchSize: 100000 }"); + if (tracing) { + doc.put("tracingSupport", org.bson.BsonBoolean.TRUE); + } + return doc; + } + + @Test + void parsesTracingSupportTrue() { + ConnectionDescription description = DescriptionHelper.createConnectionDescription( + ClusterConnectionMode.SINGLE, CONNECTION_ID, hello(true)); + assertTrue(description.isTracingSupport()); + } + + @Test + void defaultsTracingSupportFalseWhenAbsent() { + ConnectionDescription description = DescriptionHelper.createConnectionDescription( + ClusterConnectionMode.SINGLE, CONNECTION_ID, hello(false)); + assertFalse(description.isTracingSupport()); + } +} +``` + +> Confirm `createConnectionDescription`'s exact signature/visibility (Explore reported `static ConnectionDescription createConnectionDescription(ClusterConnectionMode, ConnectionId, BsonDocument)`). If the package-private method isn't visible, the test is already in the same `com.mongodb.internal.connection` package, so it is. + +- [ ] **Step 3: Run the test to verify it fails** + +Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.connection.DescriptionHelperTracingTest"` +Expected: FAIL — `isTracingSupport()` does not exist. + +- [ ] **Step 4: Add the field, wither, and getter to `ConnectionDescription`** + +Add the field next to the other `private final` fields (after `logicalSessionTimeoutMinutes`): + +```java + private final boolean tracingSupport; +``` + +In the most-derived constructor (the one with `serviceId` and all fields), add `, false` initialization by assigning `this.tracingSupport = false;` at the end of its body. For all other delegating constructors no change is needed (they call the most-derived one). + +Add a wither (mirror `withServerType`) — it constructs a copy via the most-derived constructor and then sets the flag. Since the constructor sets `tracingSupport=false`, implement the wither by copying through a private all-args path. Concretely, add: + +```java + /** + * Returns a copy of this {@code ConnectionDescription} with the given tracing-support capability. + * + * @param tracingSupport whether the server advertised OpenTelemetry trace-context support + * @return the new connection description + */ + public ConnectionDescription withTracingSupport(final boolean tracingSupport) { + ConnectionDescription copy = new ConnectionDescription(serviceId, connectionId, maxWireVersion, serverType, + maxBatchCount, maxDocumentSize, maxMessageSize, compressors, saslSupportedMechanisms, + logicalSessionTimeoutMinutes); + copy.tracingSupportOverride = tracingSupport; + return copy; + } + + /** + * @return whether the server advertised OpenTelemetry trace-context support in its hello response + */ + public boolean isTracingSupport() { + return tracingSupportOverride != null ? tracingSupportOverride : tracingSupport; + } +``` + +Because `tracingSupport` is `final`, add a separate non-final override field to keep the change additive without rewriting every constructor: + +```java + @Nullable + private Boolean tracingSupportOverride; +``` + +(`@Nullable` import `com.mongodb.lang.Nullable` already present in this file; verify and add if missing.) Remove the now-unnecessary `this.tracingSupport = false;` line and instead initialize the field inline at declaration: `private final boolean tracingSupport = false;` is illegal for a constructor-set final — so declare it `private final boolean tracingSupport;` and assign `this.tracingSupport = false;` in the most-derived constructor only. + +> Rationale: this keeps all five public constructors source- and binary-compatible. The override field carries the capability set post-construction by `withTracingSupport`. + +- [ ] **Step 5: Set the capability in `DescriptionHelper.createConnectionDescription`** + +Add a parser near the other private getters: + +```java + private static boolean getTracingSupport(final BsonDocument helloResult) { + return helloResult.getBoolean("tracingSupport", org.bson.BsonBoolean.FALSE).getValue(); + } +``` + +In `createConnectionDescription`, change the returned value to apply the wither. Find the `return connectionDescription;` (or the `new ConnectionDescription(...)` return) and wrap it: + +```java + return connectionDescription.withTracingSupport(getTracingSupport(helloResult)); +``` + +(If the method returns the `new ConnectionDescription(...)` directly, assign it to a local `ConnectionDescription connectionDescription = new ConnectionDescription(...);` first, then return the wither call.) + +- [ ] **Step 6: Run the test to verify it passes** + +Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.connection.DescriptionHelperTracingTest"` +Expected: PASS (both tests). + +- [ ] **Step 7: Commit** + +```bash +git add driver-core/src/main/com/mongodb/connection/ConnectionDescription.java \ + driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java \ + driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java +git commit -m "DRIVERS-3454: parse hello tracingSupport into ConnectionDescription" +``` + +--- + +## Task 4: Carry `tracingSupported` through `MessageSettings` + +**Files:** +- Modify: `driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java` +- Modify: `driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java` + +- [ ] **Step 1: Add the field + builder method + getter to `MessageSettings`** + +Add next to `sessionSupported` (field after line 60): + +```java + private final boolean tracingSupported; +``` + +In the `Builder` (after `sessionSupported` field, line 82): + +```java + private boolean tracingSupported; +``` + +Add the builder setter (mirror `sessionSupported(...)`, after line 137): + +```java + public Builder tracingSupported(final boolean tracingSupported) { + this.tracingSupported = tracingSupported; + return this; + } +``` + +In the private `MessageSettings(final Builder builder)` constructor (line ~197), add: + +```java + this.tracingSupported = builder.tracingSupported; +``` + +Add the getter (mirror `getMaxWireVersion()`, near line 181): + +```java + public boolean isTracingSupported() { + return tracingSupported; + } +``` + +- [ ] **Step 2: Populate it in `ProtocolHelper`** + +In `ProtocolHelper.java` at the `MessageSettings.builder()` chain (line 231), directly after the existing `.sessionSupported(connectionDescription.getLogicalSessionTimeoutMinutes() != null)` line (line 237), add: + +```java + .tracingSupported(connectionDescription.isTracingSupport()) +``` + +- [ ] **Step 3: Compile** + +Run: `./gradlew :driver-core:compileJava` +Expected: SUCCESS. + +- [ ] **Step 4: Commit** + +```bash +git add driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java \ + driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java +git commit -m "DRIVERS-3454: thread tracingSupported into MessageSettings" +``` + +--- + +## Task 5: Write OP_MSG section kind 3 in `CommandMessage.writeOpMsg()` + +The section format matches the server POC's `kSecurityToken`/`kOtelTelemetryContext`: a single section-kind byte followed by a CString (no 4-byte length prefix). The server reads it with `readCStr()`. + +**Files:** +- Modify: `driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java` + +- [ ] **Step 1: Add the section-kind constant** + +After `PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE` (line 82) add: + +```java + /** + * Specifies that the `OP_MSG` section payload is a W3C traceparent C-string (OpenTelemetry trace context). + */ + private static final byte PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT = 3; +``` + +- [ ] **Step 2: Add the import for the tracing Span** + +In the import block add: + +```java +import com.mongodb.internal.observability.micrometer.Span; +``` + +- [ ] **Step 3: Write the section in `writeOpMsg`, just before the flag bits are backpatched** + +In `writeOpMsg(...)`, immediately **before** the comment `// Write the flag bits` (line 280), insert: + +```java + writeOtelTraceContextSection(bsonOutput, operationContext); + +``` + +Then add the private method (place it after `writeOpMsg`, before `writeOpQuery`): + +```java + private void writeOtelTraceContextSection(final ByteBufferBsonOutput bsonOutput, final OperationContext operationContext) { + if (!getSettings().isTracingSupported()) { + return; + } + Span tracingSpan = operationContext.getTracingSpan(); + if (tracingSpan == null) { + return; + } + String traceParent = tracingSpan.context().traceParent(); + if (traceParent == null) { + return; + } + bsonOutput.writeByte(PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT); + bsonOutput.writeCString(traceParent); + } +``` + +> Rationale: at encode time only the operation-level span exists in `OperationContext` (the command span is created later, in `InternalStreamConnection`). The operation span is a valid parent for the server's RPC span. `context().traceParent()` returns `null` for no-op/unsampled spans, so a missing or unsampled span naturally omits the section (spec §3.3). Gating on `isTracingSupported()` ensures we never send the section to a server that would `uassert(40432)` (spec §4). + +- [ ] **Step 4: Compile** + +Run: `./gradlew :driver-core:compileJava` +Expected: SUCCESS. + +- [ ] **Step 5: Commit** + +```bash +git add driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java +git commit -m "DRIVERS-3454: write OP_MSG otel trace-context section (kind 3)" +``` + +--- + +## Task 6 (Phase 1 validation): OP_MSG round-trip encode test + +This is the automated proof of the wire contract: encode a command with a tracing-capable connection + a sampled span, then scan the produced bytes for section kind 3 + the exact traceparent C-string; assert it is absent when the capability is off or the span yields no traceparent. + +**Files:** +- Test: `driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java` + +- [ ] **Step 1: Write the failing test** + +Create `driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java`: + +```java +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.mongodb.internal.connection; + +import com.mongodb.MongoNamespace; +import com.mongodb.ReadConcern; +import com.mongodb.ReadPreference; +import com.mongodb.ServerApi; +import com.mongodb.connection.ClusterConnectionMode; +import com.mongodb.connection.ServerType; +import com.mongodb.internal.TimeoutContext; +import com.mongodb.internal.connection.MessageSequences.EmptyMessageSequences; +import com.mongodb.internal.observability.micrometer.Span; +import com.mongodb.internal.observability.micrometer.TraceContext; +import com.mongodb.internal.session.SessionContext; +import com.mongodb.internal.validator.NoOpFieldNameValidator; +import org.bson.BsonDocument; +import org.bson.BsonString; +import org.bson.ByteBuf; +import org.junit.jupiter.api.Test; + +import java.nio.charset.StandardCharsets; +import java.util.List; + +import static com.mongodb.internal.mockito.MongoMockito.mock; +import static com.mongodb.internal.operation.ServerVersionHelper.LATEST_WIRE_VERSION; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.when; + +class CommandMessageOtelTraceContextTest { + private static final MongoNamespace NAMESPACE = new MongoNamespace("db.test"); + private static final BsonDocument COMMAND = new BsonDocument("find", new BsonString(NAMESPACE.getCollectionName())); + private static final String TRACE_PARENT = "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"; + + private static CommandMessage commandMessage(final boolean tracingSupported) { + return new CommandMessage(NAMESPACE.getDatabaseName(), COMMAND, NoOpFieldNameValidator.INSTANCE, + ReadPreference.primary(), + MessageSettings.builder() + .maxWireVersion(LATEST_WIRE_VERSION) + .serverType(ServerType.REPLICA_SET_PRIMARY) + .sessionSupported(true) + .tracingSupported(tracingSupported) + .build(), + true, EmptyMessageSequences.INSTANCE, ClusterConnectionMode.MULTIPLE, (ServerApi) null); + } + + private static OperationContext operationContextWithSpan(final String traceParentOrNull) { + SessionContext sessionContext = mock(SessionContext.class, mock -> { + when(mock.getClusterTime()).thenReturn(null); + when(mock.hasSession()).thenReturn(false); + when(mock.getReadConcern()).thenReturn(ReadConcern.DEFAULT); + when(mock.notifyMessageSent()).thenReturn(true); + when(mock.hasActiveTransaction()).thenReturn(false); + when(mock.isSnapshot()).thenReturn(false); + }); + TimeoutContext timeoutContext = mock(TimeoutContext.class); + TraceContext traceContext = () -> traceParentOrNull; + Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); + return mock(OperationContext.class, mock -> { + when(mock.getSessionContext()).thenReturn(sessionContext); + when(mock.getTimeoutContext()).thenReturn(timeoutContext); + when(mock.getTracingSpan()).thenReturn(span); + }); + } + + private static byte[] encodeToBytes(final CommandMessage message, final OperationContext operationContext) { + try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { + message.encode(output, operationContext); + List buffers = output.getByteBuffers(); + byte[] bytes = new byte[output.getSize()]; + int pos = 0; + for (ByteBuf buf : buffers) { + int remaining = buf.remaining(); + buf.get(bytes, pos, remaining); + pos += remaining; + } + buffers.forEach(ByteBuf::release); + return bytes; + } + } + + private static boolean containsTraceParentSection(final byte[] message) { + // Section kind 3 byte immediately followed by the null-terminated traceparent. + byte[] needle = new byte[1 + TRACE_PARENT.length() + 1]; + needle[0] = 3; + byte[] tp = TRACE_PARENT.getBytes(StandardCharsets.UTF_8); + System.arraycopy(tp, 0, needle, 1, tp.length); + needle[needle.length - 1] = 0; + outer: + for (int i = 0; i + needle.length <= message.length; i++) { + for (int j = 0; j < needle.length; j++) { + if (message[i + j] != needle[j]) { + continue outer; + } + } + return true; + } + return false; + } + + @Test + void writesSectionWhenSupportedAndSampledSpanPresent() { + byte[] bytes = encodeToBytes(commandMessage(true), operationContextWithSpan(TRACE_PARENT)); + assertTrue(containsTraceParentSection(bytes), "expected OP_MSG section kind 3 with traceparent"); + } + + @Test + void omitsSectionWhenCapabilityAbsent() { + byte[] bytes = encodeToBytes(commandMessage(false), operationContextWithSpan(TRACE_PARENT)); + assertFalse(containsTraceParentSection(bytes), "must not send section to non-tracing server"); + } + + @Test + void omitsSectionWhenSpanHasNoTraceParent() { + byte[] bytes = encodeToBytes(commandMessage(true), operationContextWithSpan(null)); + assertFalse(containsTraceParentSection(bytes), "must omit section when span yields no traceparent"); + } +} +``` + +> If `ByteBufferBsonOutput` exposes a different accessor than `getByteBuffers()`/`getSize()`, mirror whatever the existing `CommandMessageTest`/`CommandMessageSpecification` uses to read encoded bytes (e.g. `getByteBuffers()` then `ByteBufNIO`); adjust `encodeToBytes` accordingly. The `TraceContext traceContext = () -> traceParentOrNull;` lambda works because `TraceContext` is now a single-abstract-method interface returning the traceparent. + +- [ ] **Step 2: Run to verify it fails** + +Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.connection.CommandMessageOtelTraceContextTest"` +Expected: FAIL before Task 5 is implemented; after Task 5 it should pass. (If running tasks in order, this confirms PASS.) + +- [ ] **Step 3: Run to verify it passes** + +Run: `./gradlew :driver-core:test --tests "com.mongodb.internal.connection.CommandMessageOtelTraceContextTest"` +Expected: PASS (all three). + +- [ ] **Step 4: Commit** + +```bash +git add driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java +git commit -m "DRIVERS-3454: round-trip test for OP_MSG otel trace-context section" +``` + +--- + +## Task 7 (Phase 2, optional/manual): end-to-end runbook against the server POC + +Not automated / not CI. Documents how to confirm the server starts a linked child span. + +**Files:** +- Create: `docs/superpowers/runbooks/otel-opmsg-e2e.md` + +- [ ] **Step 1: Write the runbook** + +Create `docs/superpowers/runbooks/otel-opmsg-e2e.md`: + +```markdown +# Runbook: OTel OP_MSG propagation end-to-end (manual) + +Validates DRIVERS-3454 end to end: a sync-driver client span linked to a server child span. + +## Prerequisites +- A local build of the server POC branch (10gen/mongo PR #49930), which accepts OP_MSG + section kind 3 and starts a server span from it. NOTE: the POC does NOT yet advertise + `tracingSupport` in hello. Until SERVER-107128 adds it, temporarily force the driver + capability on for this manual run (see step 3). +- An OpenTelemetry collector / exporter the server POC is configured to export traces to + (e.g. Jaeger via OTLP), plus a Micrometer Tracing OTel bridge on the client. + +## Steps +1. Build & run the server POC `mongod` with tracing enabled and sampling at 100%. +2. Start a Jaeger all-in-one (or OTLP collector) and point both server and client exporters at it. +3. In a scratch sync-driver program, configure a `MongoClient` with an `ObservationRegistry` + that has an OTel-backed `DefaultTracingObservationHandler` (so `traceParent()` is non-null + and sampled). Run a `find`. + - Temporary capability override for the run (POC server lacks the hello flag): start an + OTel root span yourself, then run the command. Because the server POC accepts kind 3 + unconditionally, set the driver `tracingSupported` to true by connecting to a server + whose hello you patch to include `tracingSupport: true`, OR temporarily hardcode + `isTracingSupport()`/`tracingSupported` to `true` on the local branch for the manual run + only (revert before commit). +4. In Jaeger, confirm a single trace contains BOTH the client `find` span and a server span, + with the server span's parent = the client span id sent in the traceparent. + +## Pass criteria +- One trace, two+ spans, correct parent/child linkage across the client→server boundary. +- With the driver capability off (default), no server span is created (negative check). + +## Notes +- This proves the wire format end to end; the automated Phase 1 test + (`CommandMessageOtelTraceContextTest`) is the regression guard. +- Findings (any format mismatch, tracestate handling, flags) feed back into the spec. +``` + +- [ ] **Step 2: Commit** + +```bash +git add docs/superpowers/runbooks/otel-opmsg-e2e.md +git commit -m "DRIVERS-3454: add Phase 2 e2e validation runbook" +``` + +--- + +## Task 8: Full module verification + +- [ ] **Step 1: Format + static checks + tests for driver-core** + +Run: `./gradlew :driver-core:spotlessApply :driver-core:check` +Expected: BUILD SUCCESSFUL. Fix any spotless/checkstyle issues in the files you touched (copyright headers, import order, no unused imports — particularly the Micrometer `TraceContext` import note in Task 2). + +- [ ] **Step 2: Confirm the new tests ran and passed** + +Run: `./gradlew :driver-core:test --tests "*OtelTraceContext*" --tests "*MicrometerTraceParent*" --tests "*DescriptionHelperTracing*"` +Expected: PASS. + +- [ ] **Step 3: Final commit if spotless changed anything** + +```bash +git add -A && git commit -m "DRIVERS-3454: apply spotless formatting" || echo "nothing to commit" +``` + +--- + +## Self-Review (completed during planning) + +**Spec coverage:** +- §3.1 section kind 3 → Task 5. §3.2 W3C traceparent format → Task 2 (`00-…-01`). §3.3 request-only / sparse / sampled-only → Task 2 (null unless sampled) + Task 5 (omit when null). §4 `tracingSupport` negotiation → Tasks 3–5 (parse → MessageSettings → gate). §6.1 `TraceContext.traceParent()` → Task 1. §6.2 read capability → Task 3. §6.3 inject → Task 5. §6.4 Phase 1 automated → Task 6; Phase 2 manual → Task 7. §7 testing → Tasks 2,3,6 + Task 8 check. +- Out-of-scope items (reactive, mongos, tracestate beyond pass-through, sampling rework) intentionally have no task — matches spec §6.5. + +**Known follow-ups (not blockers for the POC):** +- Server `hello` `tracingSupport` flag does not exist yet (SERVER-107128). Phase 1 fully validates the driver without it; Phase 2 documents the temporary override. +- The `ConnectionDescription.tracingSupportOverride` approach (Task 3) keeps public constructors binary-compatible; a production implementation would likely fold the field into a new constructor + builder during the non-POC work. + +**Type consistency:** `traceParent()` (Task 1/2/5/6), `isTracingSupport()` on `ConnectionDescription` (Task 3/4), `isTracingSupported()`/`tracingSupported(...)` on `MessageSettings` (Task 4/5/6), `PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT` (Task 5) — names are used consistently across tasks. diff --git a/docs/superpowers/runbooks/otel-opmsg-e2e.md b/docs/superpowers/runbooks/otel-opmsg-e2e.md new file mode 100644 index 00000000000..ac41c43ba79 --- /dev/null +++ b/docs/superpowers/runbooks/otel-opmsg-e2e.md @@ -0,0 +1,37 @@ +# Runbook: OTel OP_MSG propagation end-to-end (manual) + +Validates DRIVERS-3454 end to end: a sync-driver client span linked to a server child span. + +## Prerequisites +- A local build of the server POC branch (10gen/mongo PR #49930), which accepts OP_MSG + section kind 3 and starts a server span from it. NOTE: the POC does NOT yet advertise + `tracingSupport` in hello. Until SERVER-107128 adds it, temporarily force the driver + capability on for this manual run (see step 3). +- An OpenTelemetry collector / exporter the server POC is configured to export traces to + (e.g. Jaeger via OTLP), plus a Micrometer Tracing OTel bridge on the client. + +## Steps +1. Build & run the server POC `mongod` with tracing enabled and sampling at 100%. +2. Start a Jaeger all-in-one (or OTLP collector) and point both server and client exporters at it. +3. In a scratch sync-driver program, configure a `MongoClient` with an `ObservationRegistry` + that has an OTel-backed `DefaultTracingObservationHandler` (so `traceParent()` is non-null + and sampled). Run a `find`. + - Temporary capability override for the run (POC server lacks the hello flag): start an + OTel root span yourself, then run the command. Because the server POC accepts kind 3 + unconditionally, set the driver `tracingSupported` to true by connecting to a server + whose hello you patch to include `tracingSupport: true`, OR temporarily hardcode + `isTracingSupport()`/`tracingSupported` to `true` on the local branch for the manual run + only (revert before staging). +4. In Jaeger, confirm a single trace contains BOTH the client `find` span and a server span, + with the server span's parent = the client span id sent in the traceparent. + +## Pass criteria +- One trace, two+ spans, correct parent/child linkage across the client→server boundary. +- With the driver capability off (default), no server span is created (negative check). + +## Notes +- This proves the wire format end to end; the automated Phase 1 test + (`CommandMessageOtelTraceContextTest`) is the regression guard. In particular, + `getCommandDocumentIgnoresOtelSection` guards against the trailing kind-3 section + corrupting command-document reconstruction on the send path. +- Findings (any format mismatch, tracestate handling, flags) feed back into the spec. diff --git a/docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md b/docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md new file mode 100644 index 00000000000..415c31fbf73 --- /dev/null +++ b/docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md @@ -0,0 +1,221 @@ +# Design: OpenTelemetry Trace-Context Propagation over OP_MSG + +- **Ticket:** [DRIVERS-3454](https://jira.mongodb.org/browse/DRIVERS-3454) — Support trace context propagation to the server +- **Status:** Design (Investigating phase) +- **Author:** Nabil Hachicha +- **Date:** 2026-06-01 +- **Related:** [DRIVERS-719](https://jira.mongodb.org/browse/DRIVERS-719) (client-side OTel tracing), [SERVER-107128](https://jira.mongodb.org/browse/SERVER-107128) (define trace context in OP_MSG), server POC [10gen/mongo#49930](https://github.com/10gen/mongo/pull/49930) + +--- + +## 1. Summary + +DRIVERS-719 added client-side OpenTelemetry spans to the MongoDB drivers, but those +spans stop at the client boundary: the server starts a fresh, disconnected trace. This +work closes the gap by **propagating the client's active trace context to the server on +each wire message**, so the server can start a *child* span and produce one continuous +client → server timeline. + +Two deliverables, sequenced so the POC validates the spec: + +1. **Spec proposal** — defines the wire contract and negotiation mechanism, written so it + can become a `mongodb/specifications` PR shepherded through the DRIVERS process. +2. **Java driver POC** — a minimal, `driver-sync`-only implementation that exercises every + claim in the spec. Ambiguities in the spec should surface as failing tests or awkward + code; findings feed back into the spec. The POC is internal/throwaway quality and + touches no public API. + +--- + +## 2. Background & constraints + +- **Why transport layer, not command BSON.** Per SERVER-107128, the trace context is + carried in an OP_MSG *section* rather than inside the command document, so the server + can read it before parsing the command — enabling tracing of command-parse failures and + early network handling. +- **The server POC is a prototype.** PR #49930 is explicitly throwaway ("USING AI DO NOT + COPY"), intra-server focused, and unconditionally accepts the new section. It is the + authoritative reference for the **wire format only**, not for negotiation or production + behavior. +- **Process.** Spec changes must flow through DRIVERS tickets; the drivers team shepherds + them. This document is the basis for that proposal. +- **Status of dependencies (June 2026).** No OP_MSG-propagation spec exists yet. + SERVER-107128 is Open/Unassigned. The server `hello` capability flag described below is + **not** in the POC and must be added server-side for the negotiation to work end to end. + +--- + +## 3. Wire contract + +### 3.1 OP_MSG section + +A new OP_MSG section kind is defined: + +| Kind | Name | Status | +|------|-----------------------|------------| +| 0 | Body | existing | +| 1 | Document Sequence | existing | +| 2 | Security Token | existing | +| **3**| **OTel Trace Context**| **new** | + +This matches `kOtelTelemetryContext = 3` in the server POC +(`src/mongo/rpc/op_msg.cpp`). + +### 3.2 Payload format + +The section payload is a single **null-terminated C-string** containing a **W3C +`traceparent`** value: + +``` +00-<32 hex trace-id>-<16 hex span-id>-<2 hex trace-flags>[-] +``` + +- `00` — W3C version. +- trace-id — 16 bytes, 32 lowercase hex chars. +- span-id — 8 bytes, 16 lowercase hex chars (the client span that created the RPC). +- trace-flags — 1 byte, 2 hex chars (e.g. `01` = sampled). +- Base length is **55 chars**; an optional `-` suffix may follow + (server POC appends `span_context.trace_state()->ToHeader()`). + +The string is encoded with the BSON CString convention (UTF-8 bytes + trailing `\0`), +consistent with how the server reads it via `readCStr()`. + +### 3.3 Direction & optionality + +- **Request-only.** No trace data is added to responses; server spans are exported + independently via the OTel pipeline and correlated by trace-id. +- **Sparse.** The section is present only when the client has an **active, sampled** span. + When there is no span, or the span is not sampled, the section is omitted entirely. +- Server behavior (informational): if a trace context is present, the server starts a span + subject to its own external tracing rate limiter. + +--- + +## 4. Negotiation (the key addition over the POC) + +Older or mixed-version servers `uassert(40432)` ("Unknown section kind") on an +unrecognized OP_MSG section. The driver therefore must **not** send section kind 3 to a +server that does not understand it. + +**Mechanism:** the server advertises support in its `hello` response: + +``` +{ ..., "tracingSupport": true } +``` + +Rules: + +- The driver reads `tracingSupport` from each `hello`/handshake response and stores it as a + per-connection / per-server capability (default **false** when absent). +- The driver sends the section **only** on connections whose server advertised + `tracingSupport: true`. +- The capability is re-evaluated on reconnect/failover; a server that stops advertising it + (downgrade) stops receiving the section. + +> **Open question / dependency.** This flag is not in server POC #49930. SERVER-107128 must +> add it. Until then, the POC validates the section via a controlled negotiation +> (see §6). An alternative considered was gating on a `maxWireVersion` bump; rejected for +> the spec because it couples the feature to a wire-version release and is coarser than an +> explicit capability. To be confirmed with the server team. + +--- + +## 5. Edge cases (spec-level, noted; not all in POC scope) + +- **Invalid/empty traceparent.** If the driver cannot produce a valid 55-char traceparent, + it omits the section (never sends a malformed one). The server treats a sub-55-char or + malformed value as absent. +- **tracestate size.** Pass-through only in the POC; the spec should reference W3C limits + (512 chars) and define truncation/drop behavior — to be finalized with the server team. +- **Compression / OP_COMPRESSED.** The section is part of the OP_MSG body that gets + compressed; no special handling expected, but the spec calls this out for confirmation. +- **mongos / load-balanced passthrough.** Out of scope for this driver work; the server is + responsible for forwarding context to downstream services (noted only). + +--- + +## 6. Java POC (driver-sync only) + +Four focused changes in `driver-core`. All internal; no public API change. + +### 6.1 Expose the trace context + +`com.mongodb.internal.observability.micrometer.TraceContext` is currently an empty marker +interface, and `Span.context()` returns it. Extend it minimally: + +```java +public interface TraceContext { + TraceContext EMPTY = new TraceContext() { ... }; + @Nullable String traceParent(); // W3C traceparent, or null if unavailable/unsampled +} +``` + +- Implement in `MicrometerTraceContext`/`MicrometerSpan` by reading `traceId()`, + `spanId()`, and `sampled()` from the underlying Micrometer `io.micrometer.tracing.TraceContext` + and formatting the `00-…` string. Return `null` when the span is no-op or not sampled. +- `TraceContext.EMPTY` and `Span.EMPTY` return `null`. + +### 6.2 Read the capability + +- In `DescriptionHelper`, parse `tracingSupport` from the `hello` result. +- Store on `ServerDescription` (and `ConnectionDescription` as needed) with an + `isTracingSupport()` accessor, following the existing `helloOk` / `maxWireVersion` + patterns. + +### 6.3 Inject the section + +- In `CommandMessage`, add `PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT = 3`. +- In `writeOpMsg()`, after the body/sequence sections, write the new section **iff**: + 1. the connection's server advertised `tracingSupport`, **and** + 2. the active operation `Span` yields a non-null (sampled) `traceParent()`. +- The send path (`InternalStreamConnection.sendAndReceiveInternal`) already has both the + `Span` and the connection description; thread the capability + traceparent into + `CommandMessage` encoding via the existing plumbing. + +### 6.4 Validation — phased + +- **Phase 1 (in-repo, automated — the real proof):** + - Positive: build the OP_MSG bytes for a command with an active sampled span on a + tracing-capable connection; re-parse the sections and assert section kind 3 is present + and its traceparent round-trips and matches the span's trace-id/span-id. + - Negative: capability absent ⇒ no section; no/unsampled span ⇒ no section. +- **Phase 2 (optional, manual runbook — not CI):** build the server POC branch (#49930) + locally with an OTel collector, point the sync driver at it, and confirm the server + emits a child span linked to the client trace. + +### 6.5 Scope guards (YAGNI) + +Explicitly **out of scope** for the POC: reactive/async send path, any public API, +tracestate handling beyond pass-through, sampling rework, and `mongos`/load-balanced +propagation. + +--- + +## 7. Testing summary + +- Unit tests for traceparent formatting (`MicrometerTraceContext.traceParent()`), including + unsampled and no-op cases. +- Unit test for `tracingSupport` parsing in `DescriptionHelper`. +- OP_MSG encode/parse round-trip tests in `CommandMessage` (positive + negative), per §6.4. +- All new code follows `driver-core` conventions (Java 8 baseline, SLF4J, copyright header, + `@Nullable`/`@NonNull`). No reduction in coverage. + +--- + +## 8. Deliverable order + +1. Write & socialize this spec proposal (basis for the DRIVERS spec PR + SERVER-107128 + coordination on the `hello` flag). +2. Build the Java `driver-sync` POC (§6) and run Phase 1 validation. +3. Feed POC findings back into the spec; optionally run Phase 2 end-to-end. + +--- + +## 9. Open questions + +- Server `hello` `tracingSupport` flag ownership/timing (SERVER-107128 is unassigned). +- Final `tracestate` size/truncation policy. +- Whether this section should be designed as a special case or as the first user of the + broader "client-side telemetry section" effort (retry metadata + OTel + client config). +- Confirmation that the W3C `traceparent` (vs. a BSON-structured payload) is the final + on-wire encoding the server team will commit to. diff --git a/docs/superpowers/specs/2026-06-02-otel-e2e-jaeger-design.md b/docs/superpowers/specs/2026-06-02-otel-e2e-jaeger-design.md new file mode 100644 index 00000000000..7bd19fe5562 --- /dev/null +++ b/docs/superpowers/specs/2026-06-02-otel-e2e-jaeger-design.md @@ -0,0 +1,218 @@ +# Design: End-to-end OTel trace visualization (driver toggle + Spring Boot + Jaeger) + +- **Ticket:** [DRIVERS-3454](https://jira.mongodb.org/browse/DRIVERS-3454) — trace context propagation to the server +- **Status:** Design +- **Author:** Nabil Hachicha +- **Date:** 2026-06-02 +- **Builds on:** `docs/superpowers/specs/2026-06-01-otel-opmsg-propagation-design.md` (the OP_MSG kind-3 POC, already staged) and the running server POC (`~/MongoDB/otel-poc-server/`, mongod `gitVersion b2cb2bf`). + +--- + +## 1. Goal + +Make a client→server trace **visible end to end in Jaeger**: a Spring Boot app issues a MongoDB +operation; the driver propagates its sampled W3C `traceparent` to the server via the OP_MSG kind-3 +section; the server starts a child span; both client and server spans export to one Jaeger instance +and appear under a single trace. + +The POC server does **not** advertise `tracingSupport` in `hello`, so the driver (which gates on +that capability) needs a temporary, removable switch to send the section anyway. + +## 2. Components + +1. **Driver test-only force toggle** — bypasses the server-capability gate only. +2. **Local Maven publish** — `5.9.0-SNAPSHOT` consumable by the app. +3. **Jaeger** (Docker) — single OTLP collector + UI. +4. **mongod re-config** — export server spans to Jaeger. +5. **Spring Boot app** (host JVM) — Spring Data MongoDB + Micrometer→OTel→Jaeger, wired to the + driver's internal tracing, with a REST trigger. + +--- + +## 3. Driver: force-propagation toggle + +New file `driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java`: + +```java +public final class OtelTracePropagationTestToggle { + // TEST-ONLY (DRIVERS-3454): force sending the OP_MSG OTel trace-context section even when the + // server did not advertise tracingSupport in hello. Remove before any production use. + public static volatile boolean FORCE_PROPAGATION = false; + + private OtelTracePropagationTestToggle() { + } +} +``` + +`CommandMessage.writeOtelTraceContextSection` gate changes from: + +```java +if (!getSettings().isTracingSupported()) { + return; +} +``` + +to: + +```java +if (!getSettings().isTracingSupported() && !OtelTracePropagationTestToggle.FORCE_PROPAGATION) { + return; +} +``` + +The remaining gates are unchanged: a `null` tracing span or a `null`/unsampled `traceParent()` still +omits the section. So the toggle only relaxes the capability check. + +**Test** (`CommandMessageOtelTraceContextTest`): new case — `tracingSupported=false` + +`FORCE_PROPAGATION=true` + sampled span ⇒ section IS written. Set/reset `FORCE_PROPAGATION` in a +`try/finally` so the static does not leak into other tests. + +**Removal path:** delete the toggle class and revert the one `&&` clause. + +## 4. Publish to Maven Local + +From the repo root: + +```bash +./gradlew publishToMavenLocal -PskipCryptVerify=true +``` + +Publishes all modules (`bson`, `bson-record-codec`, `driver-core`, `driver-sync`, …) as +`org.mongodb:*:5.9.0-SNAPSHOT`. The app consumes `5.9.0-SNAPSHOT` from `mavenLocal()`. + +## 5. Jaeger (Docker) + +`jaegertracing/all-in-one:1.62.0` on a dedicated network `otel-poc-net`, OTLP enabled, host ports: + +| Port | Purpose | +|------|---------| +| 16686 | Jaeger UI | +| 4317 | OTLP gRPC | +| 4318 | OTLP HTTP | + +Run: + +```bash +docker network create otel-poc-net 2>/dev/null || true +docker run -d --name jaeger --network otel-poc-net \ + -e COLLECTOR_OTLP_ENABLED=true \ + -p 16686:16686 -p 4317:4317 -p 4318:4318 \ + jaegertracing/all-in-one:1.62.0 +``` + +## 6. mongod: export server spans to Jaeger + +Restart the existing POC mongod container **on `otel-poc-net`** so it can reach Jaeger by name, and +point its OTLP exporter at Jaeger: + +```bash +docker rm -f otel-poc-mongod-run +docker run -d --name otel-poc-mongod-run --platform linux/arm64 --network otel-poc-net \ + -v "$HOME/MongoDB/otel-poc-server/dist-test:/opt/mongo:ro" \ + -v "$HOME/MongoDB/otel-poc-server/data:/data/db" \ + -p 27017:27017 \ + otel-poc-mongod \ + /opt/mongo/bin/mongod --dbpath /data/db --bind_ip_all --port 27017 \ + --setParameter opentelemetryHttpEndpoint=http://jaeger:4318/v1/traces \ + --setParameter openTelemetryExportIntervalMillis=1000 +``` + +`featureFlagTracing` is already enabled in this build. **Verification & fallback:** confirm via +server logs that the OTLP endpoint is accepted and exports succeed; if the exact endpoint form is +rejected, fall back to the already-working file exporter +(`opentelemetryTraceDirectory=/data/db/otel-traces`) and note that server spans are then inspected as +JSONL rather than in the Jaeger UI. + +## 7. Spring Boot app (host JVM) + +Maven project at `~/MongoDB/otel-poc-server/otel-poc-client/` — kept outside the driver repo so it is +not entangled with the driver build. + +**Stack:** Spring Boot 3.3.x, Java 17, Maven wrapper. + +**Dependencies:** `spring-boot-starter-web`, `spring-boot-starter-data-mongodb`, +`spring-boot-starter-actuator`, `micrometer-tracing-bridge-otel`, `opentelemetry-exporter-otlp`. +`pom.xml` adds `mavenLocal()` (via a ``) and overrides `5.9.0-SNAPSHOT`. + +**Critical wiring bean** — activates the driver's *internal* tracing (the path that sets +`operationContext.tracingSpan`, which `traceParent()` reads): + +```java +@Bean +MongoClientSettingsBuilderCustomizer tracingCustomizer(ObservationRegistry registry) { + return builder -> builder.observabilitySettings( + MicrometerObservabilitySettings.builder() + .observationRegistry(registry) + .build()); +} +``` + +**Toggle activation** — set before any operation runs: + +```java +@PostConstruct +void enableForcedPropagation() { + OtelTracePropagationTestToggle.FORCE_PROPAGATION = true; +} +``` + +**Trigger** — a `@RestController`: + +```java +@GetMapping("/ping") +String ping() { + mongoTemplate.getCollection("ping").insertOne(new Document("at", new Date())); + long n = mongoTemplate.getCollection("ping").countDocuments(); + return "ok, count=" + n; +} +``` + +Hitting `/ping` creates an HTTP server span (root, sampled), under which the driver creates a mongo +command span (exported to Jaeger; its context is the propagated `traceparent`), and the server +creates its child span (exported to Jaeger). + +**`application.properties`:** + +```properties +spring.application.name=otel-poc-client +spring.data.mongodb.uri=mongodb://localhost:27017/test +management.tracing.sampling.probability=1.0 +management.otlp.tracing.endpoint=http://localhost:4318/v1/traces +# Suppress Spring's auto Mongo command instrumentation so the driver's internal tracer is the +# ONLY source of Mongo command spans (avoids duplicate/competing spans). +management.metrics.enable.mongodb=false +spring.autoconfigure.exclude=org.springframework.boot.actuate.autoconfigure.metrics.mongo.MongoMetricsAutoConfiguration +``` + +**Suppressing Spring's auto Mongo instrumentation (firm decision).** Spring Boot's actuator +auto-registers a Mongo command listener via `MongoMetricsAutoConfiguration`. To guarantee the +driver's own internal tracer is the single source of Mongo command spans (and the one performing +propagation), we **exclude** `org.springframework.boot.actuate.autoconfigure.metrics.mongo.MongoMetricsAutoConfiguration` +and set `management.metrics.enable.mongodb=false`. Only our `MongoClientSettingsBuilderCustomizer` +(§7 wiring) then contributes Mongo observability. + +## 8. End-to-end run & verification + +1. Start Jaeger (§5). 2. Restart mongod on the network with OTLP (§6). 3. Publish the driver (§4). +4. `./mvnw spring-boot:run` (§7). 5. `curl localhost:8080/ping`. 6. Open **http://localhost:16686**, +select service `otel-poc-client`, open the latest trace. + +**Pass criteria:** one trace contains the HTTP span, the driver mongo command span, **and** a +`mongod` server span; the server span's `traceId` equals the client trace, and its parent is the +client mongo span's id. Negative check: with `FORCE_PROPAGATION=false`, the server span is absent +from the client trace (server starts its own unrelated trace, if any). + +## 9. Scope guards + +- Test-only toggle; not a public API; removed before any real merge. +- Single mongod (no replica set/sharding), `find`/`insert` only. +- No auth/TLS on the local mongod. +- The app lives outside the driver repo; it is not committed to the driver repo. + +## 10. Open questions / risks + +- Exact accepted form of `opentelemetryHttpEndpoint` (full `/v1/traces` URL vs base) — verify via + logs; file-exporter fallback documented (§6). +- (Resolved) Spring's auto Mongo instrumentation is suppressed via autoconfigure-exclude + metric disable (§7). +- Spring Boot 3.3.x pins an older driver; overriding `mongodb.version` to `5.9.0-SNAPSHOT` assumes API + compatibility (it is, the API is unchanged by this POC). diff --git a/driver-core/build.gradle.kts b/driver-core/build.gradle.kts index 047b3a43a63..5efda24e3a3 100644 --- a/driver-core/build.gradle.kts +++ b/driver-core/build.gradle.kts @@ -57,9 +57,15 @@ dependencies { optionalImplementation(platform(libs.micrometer.observation.bom)) optionalImplementation(libs.micrometer.observation) + optionalImplementation(libs.micrometer.tracing) testImplementation(project(path = ":bson", configuration = "testArtifacts")) testImplementation(libs.reflections) + + // Tracing testing + testImplementation(platform(libs.micrometer.tracing.integration.test.bom)) + testImplementation(libs.micrometer.tracing.integration.test) { exclude(group = "org.junit.jupiter") } + testImplementation(libs.netty.tcnative.boringssl.static) listOf("linux-x86_64", "linux-aarch_64", "osx-x86_64", "osx-aarch_64", "windows-x86_64").forEach { arch -> testImplementation("${libs.netty.tcnative.boringssl.static.get()}::$arch") diff --git a/driver-core/src/main/com/mongodb/connection/ConnectionDescription.java b/driver-core/src/main/com/mongodb/connection/ConnectionDescription.java index c8c213398b7..b62cc018595 100644 --- a/driver-core/src/main/com/mongodb/connection/ConnectionDescription.java +++ b/driver-core/src/main/com/mongodb/connection/ConnectionDescription.java @@ -48,6 +48,7 @@ public class ConnectionDescription { private final List compressors; private final BsonArray saslSupportedMechanisms; private final Integer logicalSessionTimeoutMinutes; + private final boolean tracingSupport; private static final int DEFAULT_MAX_MESSAGE_SIZE = 0x2000000; // 32MB private static final int DEFAULT_MAX_WRITE_BATCH_SIZE = 512; @@ -150,6 +151,15 @@ private ConnectionDescription(@Nullable final ObjectId serviceId, final Connecti final ServerType serverType, final int maxBatchCount, final int maxDocumentSize, final int maxMessageSize, final List compressors, @Nullable final BsonArray saslSupportedMechanisms, @Nullable final Integer logicalSessionTimeoutMinutes) { + this(serviceId, connectionId, maxWireVersion, serverType, maxBatchCount, maxDocumentSize, maxMessageSize, compressors, + saslSupportedMechanisms, logicalSessionTimeoutMinutes, false); + } + + private ConnectionDescription(@Nullable final ObjectId serviceId, final ConnectionId connectionId, final int maxWireVersion, + final ServerType serverType, final int maxBatchCount, final int maxDocumentSize, + final int maxMessageSize, final List compressors, + @Nullable final BsonArray saslSupportedMechanisms, @Nullable final Integer logicalSessionTimeoutMinutes, + final boolean tracingSupport) { this.serviceId = serviceId; this.connectionId = connectionId; this.serverType = serverType; @@ -160,6 +170,7 @@ private ConnectionDescription(@Nullable final ObjectId serviceId, final Connecti this.compressors = notNull("compressors", Collections.unmodifiableList(new ArrayList<>(compressors))); this.saslSupportedMechanisms = saslSupportedMechanisms; this.logicalSessionTimeoutMinutes = logicalSessionTimeoutMinutes; + this.tracingSupport = tracingSupport; } /** * Creates a new connection description with the set connection id @@ -171,7 +182,7 @@ private ConnectionDescription(@Nullable final ObjectId serviceId, final Connecti public ConnectionDescription withConnectionId(final ConnectionId connectionId) { notNull("connectionId", connectionId); return new ConnectionDescription(serviceId, connectionId, maxWireVersion, serverType, maxBatchCount, maxDocumentSize, - maxMessageSize, compressors, saslSupportedMechanisms, logicalSessionTimeoutMinutes); + maxMessageSize, compressors, saslSupportedMechanisms, logicalSessionTimeoutMinutes, tracingSupport); } /** @@ -184,7 +195,22 @@ public ConnectionDescription withConnectionId(final ConnectionId connectionId) { public ConnectionDescription withServiceId(final ObjectId serviceId) { notNull("serviceId", serviceId); return new ConnectionDescription(serviceId, connectionId, maxWireVersion, serverType, maxBatchCount, maxDocumentSize, - maxMessageSize, compressors, saslSupportedMechanisms, logicalSessionTimeoutMinutes); + maxMessageSize, compressors, saslSupportedMechanisms, logicalSessionTimeoutMinutes, tracingSupport); + } + + /** + * Creates a new connection description with the given tracing support flag. + * + *

A value of {@code true} indicates that the server advertised OpenTelemetry trace-context support in its {@code hello} response + * and that the driver may send trace context to the server over OP_MSG.

+ * + * @param tracingSupport whether the server supports OpenTelemetry trace-context propagation + * @return the new connection description + * @since 5.9 + */ + public ConnectionDescription withTracingSupport(final boolean tracingSupport) { + return new ConnectionDescription(serviceId, connectionId, maxWireVersion, serverType, maxBatchCount, maxDocumentSize, + maxMessageSize, compressors, saslSupportedMechanisms, logicalSessionTimeoutMinutes, tracingSupport); } /** @@ -293,6 +319,21 @@ public BsonArray getSaslSupportedMechanisms() { public Integer getLogicalSessionTimeoutMinutes() { return logicalSessionTimeoutMinutes; } + + /** + * Returns whether the server advertised OpenTelemetry trace-context support in its {@code hello} response. + * + *

When {@code true}, the driver may propagate trace context to the server over OP_MSG. + * When {@code false} (the default), the server did not advertise this capability and sending unknown OP_MSG sections would cause + * the server to reject the message.

+ * + * @return {@code true} if the server supports OpenTelemetry trace-context propagation + * @since 5.9 + */ + public boolean isTracingSupported() { + return tracingSupport; + } + /** * Get the default maximum message size. * @@ -350,6 +391,9 @@ public boolean equals(final Object o) { if (!Objects.equals(logicalSessionTimeoutMinutes, that.logicalSessionTimeoutMinutes)) { return false; } + if (tracingSupport != that.tracingSupport) { + return false; + } return Objects.equals(saslSupportedMechanisms, that.saslSupportedMechanisms); } @@ -365,6 +409,7 @@ public int hashCode() { result = 31 * result + (serviceId != null ? serviceId.hashCode() : 0); result = 31 * result + (saslSupportedMechanisms != null ? saslSupportedMechanisms.hashCode() : 0); result = 31 * result + (logicalSessionTimeoutMinutes != null ? logicalSessionTimeoutMinutes.hashCode() : 0); + result = 31 * result + (tracingSupport ? 1 : 0); return result; } @@ -380,6 +425,7 @@ public String toString() { + ", compressors=" + compressors + ", logicialSessionTimeoutMinutes=" + logicalSessionTimeoutMinutes + ", serviceId=" + serviceId + + ", tracingSupport=" + tracingSupport + '}'; } } diff --git a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java index 348349fd18c..d3d779ac984 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java +++ b/driver-core/src/main/com/mongodb/internal/connection/CommandMessage.java @@ -25,6 +25,8 @@ import com.mongodb.internal.MongoNamespaceHelper; import com.mongodb.internal.TimeoutContext; import com.mongodb.internal.connection.MessageSequences.EmptyMessageSequences; +import com.mongodb.internal.observability.micrometer.OtelTracePropagationTestToggle; +import com.mongodb.internal.observability.micrometer.Span; import com.mongodb.internal.session.SessionContext; import com.mongodb.lang.Nullable; import org.bson.BsonArray; @@ -80,6 +82,12 @@ public final class CommandMessage extends RequestMessage { * Specifies that the `OP_MSG` section payload is a sequence of BSON documents. */ private static final byte PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE = 1; + /** + * Specifies that the `OP_MSG` section payload is a W3C traceparent C-string (OpenTelemetry trace context). + * + *

Mirrors the server's {@code kOtelTelemetryContext = 3} section kind (see DRIVERS-3454).

+ */ + private static final byte PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT = 3; private static final int UNINITIALIZED_POSITION = -1; @@ -156,15 +164,20 @@ BsonDocument getCommandDocument(final ByteBufferBsonOutput bsonOutput) { byteBuf.position(firstDocumentPosition); ByteBufBsonDocument byteBufBsonDocument = createOne(byteBuf); - // If true, it means there is at least one `PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE` section in the OP_MSG + // If true, there are more sections after the `PAYLOAD_TYPE_0_DOCUMENT` section: either one or more + // `PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE` sections, and/or a trailing `PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT` section. if (byteBuf.hasRemaining()) { BsonDocument commandBsonDocument = byteBufBsonDocument.toBaseBsonDocument(); // Each loop iteration processes one Document Sequence // When there are no more bytes remaining, there are no more Document Sequences while (byteBuf.hasRemaining()) { - // skip reading the payload type, we know it is `PAYLOAD_TYPE_1` - byteBuf.position(byteBuf.position() + 1); + byte payloadType = byteBuf.get(); + // Document-sequence sections always precede any trailing non-sequence section (e.g. + // PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT), and such sections carry no command-document fields, so stop here. + if (payloadType != PAYLOAD_TYPE_1_DOCUMENT_SEQUENCE) { + break; + } int sequenceStart = byteBuf.position(); int sequenceSizeInBytes = byteBuf.getInt(); int sectionEnd = sequenceStart + sequenceSizeInBytes; @@ -277,11 +290,29 @@ private int writeOpMsg(final ByteBufferBsonOutput bsonOutput, final OperationCon fail(sequences.toString()); } + writeOtelTraceContextSection(bsonOutput, operationContext); + // Write the flag bits bsonOutput.writeInt32(flagPosition, getOpMsgFlagBits()); return commandStartPosition; } + private void writeOtelTraceContextSection(final ByteBufferBsonOutput bsonOutput, final OperationContext operationContext) { + if (!getSettings().isTracingSupported() && !OtelTracePropagationTestToggle.FORCE_PROPAGATION) { + return; + } + Span tracingSpan = operationContext.getTracingSpan(); + if (tracingSpan == null) { + return; + } + String traceParent = tracingSpan.context().traceParent(); + if (traceParent == null) { + return; + } + bsonOutput.writeByte(PAYLOAD_TYPE_3_OTEL_TRACE_CONTEXT); + bsonOutput.writeCString(traceParent); + } + private int writeOpQuery(final ByteBufferBsonOutput bsonOutput) { bsonOutput.writeInt32(0); bsonOutput.writeCString(new MongoNamespace(getDatabase(), MongoNamespaceHelper.COMMAND_COLLECTION_NAME).getFullName()); diff --git a/driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java b/driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java index 26f73bcee9c..a070d8bbdf4 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java +++ b/driver-core/src/main/com/mongodb/internal/connection/DescriptionHelper.java @@ -69,6 +69,7 @@ static ConnectionDescription createConnectionDescription(final ClusterConnection getMaxWireVersion(helloResult), getServerType(helloResult), getMaxWriteBatchSize(helloResult), getMaxBsonObjectSize(helloResult), getMaxMessageSizeBytes(helloResult), getCompressors(helloResult), helloResult.getArray("saslSupportedMechs", null), getLogicalSessionTimeoutMinutes(helloResult)); + connectionDescription = connectionDescription.withTracingSupport(getTracingSupport(helloResult)); if (helloResult.containsKey("connectionId")) { ConnectionId newConnectionId = connectionDescription.getConnectionId().withServerValue(helloResult.getNumber("connectionId").longValue()); @@ -170,6 +171,10 @@ private static Integer getLogicalSessionTimeoutMinutes(final BsonDocument helloR ? helloResult.getNumber("logicalSessionTimeoutMinutes").intValue() : null; } + private static boolean getTracingSupport(final BsonDocument helloResult) { + return helloResult.getBoolean("tracingSupport", BsonBoolean.FALSE).getValue(); + } + @Nullable private static String getString(final BsonDocument response, final String key) { if (response.containsKey(key)) { diff --git a/driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java b/driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java index 51587e8f91d..6eaad0ea379 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java +++ b/driver-core/src/main/com/mongodb/internal/connection/MessageSettings.java @@ -58,6 +58,7 @@ public final class MessageSettings { private final int maxWireVersion; private final ServerType serverType; private final boolean sessionSupported; + private final boolean tracingSupported; private final boolean cryptd; /** @@ -80,6 +81,7 @@ public static final class Builder { private int maxWireVersion = UNKNOWN_WIRE_VERSION; private ServerType serverType; private boolean sessionSupported; + private boolean tracingSupported; private boolean cryptd; /** @@ -139,6 +141,11 @@ public Builder sessionSupported(final boolean sessionSupported) { return this; } + public Builder tracingSupported(final boolean tracingSupported) { + this.tracingSupported = tracingSupported; + return this; + } + /** * Set whether the server is a mongocryptd. * @@ -193,6 +200,10 @@ public boolean isSessionSupported() { return sessionSupported; } + public boolean isTracingSupported() { + return tracingSupported; + } + private MessageSettings(final Builder builder) { this.maxDocumentSize = builder.maxDocumentSize; @@ -201,6 +212,7 @@ private MessageSettings(final Builder builder) { this.maxWireVersion = builder.maxWireVersion; this.serverType = builder.serverType; this.sessionSupported = builder.sessionSupported; + this.tracingSupported = builder.tracingSupported; this.cryptd = builder.cryptd; } } diff --git a/driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java b/driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java index c6ad5f451a0..a519252dfed 100644 --- a/driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java +++ b/driver-core/src/main/com/mongodb/internal/connection/ProtocolHelper.java @@ -235,6 +235,7 @@ static MessageSettings getMessageSettings(final ConnectionDescription connection .maxWireVersion(connectionDescription.getMaxWireVersion()) .serverType(connectionDescription.getServerType()) .sessionSupported(connectionDescription.getLogicalSessionTimeoutMinutes() != null) + .tracingSupported(connectionDescription.isTracingSupported()) .cryptd(serverDescription.isCryptd()) .build(); } diff --git a/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java b/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java index d0d306de2c4..a75bdaa0140 100644 --- a/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java +++ b/driver-core/src/main/com/mongodb/internal/observability/micrometer/MicrometerTracer.java @@ -24,6 +24,7 @@ import io.micrometer.observation.Observation; import io.micrometer.observation.ObservationConvention; import io.micrometer.observation.ObservationRegistry; +import io.micrometer.tracing.handler.TracingObservationHandler; import org.bson.BsonDocument; import org.bson.BsonReader; import org.bson.json.JsonMode; @@ -116,6 +117,29 @@ private static class MicrometerTraceContext implements TraceContext { MicrometerTraceContext(@Nullable final Observation observation) { this.observation = observation; } + + @Override + @Nullable + public String traceParent() { + if (observation == null) { + return null; + } + TracingObservationHandler.TracingContext tracingContext = + observation.getContextView().get(TracingObservationHandler.TracingContext.class); + if (tracingContext == null || tracingContext.getSpan() == null) { + return null; + } + // Fully qualified to avoid a name clash with this package's own TraceContext interface. + io.micrometer.tracing.TraceContext ctx = tracingContext.getSpan().context(); + if (ctx == null || ctx.traceId() == null || ctx.spanId() == null) { + return null; + } + Boolean sampled = ctx.sampled(); + if (sampled == null || !sampled) { + return null; + } + return "00-" + ctx.traceId() + "-" + ctx.spanId() + "-01"; + } } /** diff --git a/driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java b/driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java new file mode 100644 index 00000000000..96d5b767121 --- /dev/null +++ b/driver-core/src/main/com/mongodb/internal/observability/micrometer/OtelTracePropagationTestToggle.java @@ -0,0 +1,32 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.mongodb.internal.observability.micrometer; + +/** + * TEST-ONLY switch (DRIVERS-3454): when {@code true}, the driver writes the OP_MSG OpenTelemetry + * trace-context section even if the server did not advertise {@code tracingSupport} in its + * {@code hello} response. The sampled-{@code traceparent} requirement still applies. + * + *

This exists only to exercise end-to-end propagation against a server that does not yet advertise + * the capability. Remove before any production use.

+ */ +public final class OtelTracePropagationTestToggle { + public static volatile boolean FORCE_PROPAGATION = false; + + private OtelTracePropagationTestToggle() { + } +} diff --git a/driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java b/driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java index 5ca248db59d..c7f78b727e6 100644 --- a/driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java +++ b/driver-core/src/main/com/mongodb/internal/observability/micrometer/TraceContext.java @@ -16,8 +16,22 @@ package com.mongodb.internal.observability.micrometer; +import com.mongodb.lang.Nullable; + @SuppressWarnings("InterfaceIsType") public interface TraceContext { TraceContext EMPTY = new TraceContext() { + @Override + public String traceParent() { + return null; + } }; + + /** + * The W3C {@code traceparent} string for this context + * ({@code 00-<32hex traceId>-<16hex spanId>-<2hex flags>}), + * or {@code null} if unavailable or the span is not sampled. + */ + @Nullable + String traceParent(); } diff --git a/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java new file mode 100644 index 00000000000..5f941327699 --- /dev/null +++ b/driver-core/src/test/unit/com/mongodb/internal/connection/CommandMessageOtelTraceContextTest.java @@ -0,0 +1,207 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.mongodb.internal.connection; + +import com.mongodb.MongoNamespace; +import com.mongodb.ReadConcern; +import com.mongodb.ReadPreference; +import com.mongodb.connection.ClusterConnectionMode; +import com.mongodb.connection.ServerType; +import com.mongodb.internal.TimeoutContext; +import com.mongodb.internal.TimeoutSettings; +import com.mongodb.internal.connection.MessageSequences.EmptyMessageSequences; +import com.mongodb.internal.observability.micrometer.OtelTracePropagationTestToggle; +import com.mongodb.internal.observability.micrometer.Span; +import com.mongodb.internal.observability.micrometer.TraceContext; +import com.mongodb.internal.session.SessionContext; +import com.mongodb.internal.validator.NoOpFieldNameValidator; +import org.bson.BsonDocument; +import org.bson.BsonString; +import org.junit.jupiter.api.Test; + +import java.nio.charset.StandardCharsets; + +import static com.mongodb.internal.mockito.MongoMockito.mock; +import static com.mongodb.internal.operation.ServerVersionHelper.LATEST_WIRE_VERSION; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.mockito.Mockito.when; + +class CommandMessageOtelTraceContextTest { + + private static final String TRACEPARENT = "00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01"; + private static final MongoNamespace NAMESPACE = new MongoNamespace("db.test"); + private static final BsonDocument COMMAND = new BsonDocument("find", new BsonString(NAMESPACE.getCollectionName())); + + @Test + void writesSectionWhenSupportedAndSampledSpanPresent() { + CommandMessage message = buildCommandMessage(true); + TraceContext traceContext = () -> TRACEPARENT; + Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); + OperationContext operationContext = buildOperationContext(span); + + byte[] encoded = encodeToBytes(message, operationContext); + + assertTrue(containsOtelSection(encoded, TRACEPARENT), + "Encoded message should contain the OTel trace context section (kind byte 3 + traceparent C-string)"); + } + + @Test + void getCommandDocumentIgnoresOtelSection() { + // Regression guard: InternalStreamConnection calls getCommandDocument() on every send (logging/monitoring/ + // compression). The trailing kind-3 section must not corrupt command-document reconstruction. + CommandMessage message = buildCommandMessage(true); + TraceContext traceContext = () -> TRACEPARENT; + Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); + OperationContext operationContext = buildOperationContext(span); + + try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { + message.encode(output, operationContext); + BsonDocument commandDocument = message.getCommandDocument(output); + assertEquals("test", commandDocument.getString("find").getValue()); + assertEquals("db", commandDocument.getString("$db").getValue()); + // The reconstructed command is exactly the body section (find + $db); the trailing + // kind-3 section must not leak any extra fields into it. + assertEquals(2, commandDocument.size()); + } + } + + @Test + void omitsSectionWhenCapabilityAbsent() { + CommandMessage message = buildCommandMessage(false); + TraceContext traceContext = () -> TRACEPARENT; + Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); + OperationContext operationContext = buildOperationContext(span); + + byte[] encoded = encodeToBytes(message, operationContext); + + assertFalse(containsOtelSection(encoded, TRACEPARENT), + "Encoded message should NOT contain the OTel trace context section when tracingSupported=false"); + } + + @Test + void omitsSectionWhenSpanHasNoTraceParent() { + CommandMessage message = buildCommandMessage(true); + TraceContext traceContext = () -> null; + Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); + OperationContext operationContext = buildOperationContext(span); + + byte[] encoded = encodeToBytes(message, operationContext); + + assertFalse(containsOtelSection(encoded, TRACEPARENT), + "Encoded message should NOT contain the OTel trace context section when traceParent is null"); + } + + @Test + void omitsSectionWhenSpanIsNull() { + CommandMessage message = buildCommandMessage(true); + OperationContext operationContext = buildOperationContext(null); + + byte[] encoded = encodeToBytes(message, operationContext); + + assertFalse(containsOtelSection(encoded, TRACEPARENT), + "Encoded message should NOT contain the OTel trace context section when there is no tracing span"); + } + + @Test + void writesSectionWhenForcedEvenIfCapabilityAbsent() { + OtelTracePropagationTestToggle.FORCE_PROPAGATION = true; + try { + CommandMessage message = buildCommandMessage(false); // server did NOT advertise tracingSupport + TraceContext traceContext = () -> TRACEPARENT; + Span span = mock(Span.class, mock -> when(mock.context()).thenReturn(traceContext)); + OperationContext operationContext = buildOperationContext(span); + + byte[] encoded = encodeToBytes(message, operationContext); + + assertTrue(containsOtelSection(encoded, TRACEPARENT), + "With FORCE_PROPAGATION the section must be sent even when the server did not advertise support"); + } finally { + OtelTracePropagationTestToggle.FORCE_PROPAGATION = false; + } + } + + // --- helpers --- + + private static CommandMessage buildCommandMessage(final boolean tracingSupported) { + return new CommandMessage( + NAMESPACE.getDatabaseName(), + COMMAND, + NoOpFieldNameValidator.INSTANCE, + ReadPreference.primary(), + MessageSettings.builder() + .maxWireVersion(LATEST_WIRE_VERSION) + .serverType(ServerType.REPLICA_SET_PRIMARY) + .sessionSupported(true) + .tracingSupported(tracingSupported) + .build(), + true, + EmptyMessageSequences.INSTANCE, + ClusterConnectionMode.MULTIPLE, + null); + } + + private static OperationContext buildOperationContext(final Span span) { + SessionContext sessionContext = mock(SessionContext.class, mock -> { + when(mock.getClusterTime()).thenReturn(null); + when(mock.hasSession()).thenReturn(false); + when(mock.getReadConcern()).thenReturn(ReadConcern.DEFAULT); + when(mock.notifyMessageSent()).thenReturn(true); + when(mock.hasActiveTransaction()).thenReturn(false); + when(mock.isSnapshot()).thenReturn(false); + }); + TimeoutContext timeoutContext = new TimeoutContext(TimeoutSettings.DEFAULT); + return mock(OperationContext.class, mock -> { + when(mock.getSessionContext()).thenReturn(sessionContext); + when(mock.getTimeoutContext()).thenReturn(timeoutContext); + when(mock.getTracingSpan()).thenReturn(span); + }); + } + + private static byte[] encodeToBytes(final CommandMessage message, final OperationContext operationContext) { + try (ByteBufferBsonOutput output = new ByteBufferBsonOutput(new SimpleBufferProvider())) { + message.encode(output, operationContext); + return output.toByteArray(); + } + } + + /** + * Searches for the needle: kind byte {@code 3} immediately followed by the UTF-8 bytes of + * {@code traceparent} and a trailing null byte (C-string terminator). + */ + private static boolean containsOtelSection(final byte[] encoded, final String traceparent) { + byte[] traceparentBytes = traceparent.getBytes(StandardCharsets.UTF_8); + // needle = [0x03, tp[0], tp[1], ..., tp[n-1], 0x00] + int needleLen = 1 + traceparentBytes.length + 1; + outer: + for (int i = 0; i <= encoded.length - needleLen; i++) { + if (encoded[i] != 3) { + continue; + } + for (int j = 0; j < traceparentBytes.length; j++) { + if (encoded[i + 1 + j] != traceparentBytes[j]) { + continue outer; + } + } + if (encoded[i + 1 + traceparentBytes.length] == 0) { + return true; + } + } + return false; + } +} diff --git a/driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java b/driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java new file mode 100644 index 00000000000..1af0ca854fa --- /dev/null +++ b/driver-core/src/test/unit/com/mongodb/internal/connection/DescriptionHelperTracingTest.java @@ -0,0 +1,81 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.mongodb.internal.connection; + +import com.mongodb.ServerAddress; +import com.mongodb.connection.ClusterConnectionMode; +import com.mongodb.connection.ClusterId; +import com.mongodb.connection.ConnectionDescription; +import com.mongodb.connection.ConnectionId; +import com.mongodb.connection.ServerId; +import org.bson.BsonBoolean; +import org.bson.BsonDocument; +import org.junit.jupiter.api.Test; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class DescriptionHelperTracingTest { + private static final ConnectionId CONNECTION_ID = + new ConnectionId(new ServerId(new ClusterId(), new ServerAddress())); + + private static BsonDocument hello(final boolean tracing) { + BsonDocument doc = BsonDocument.parse( + "{ ok: 1, ismaster: true, maxWireVersion: 25, minWireVersion: 0," + + " maxBsonObjectSize: 16777216, maxMessageSizeBytes: 48000000, maxWriteBatchSize: 100000 }"); + if (tracing) { + doc.put("tracingSupport", BsonBoolean.TRUE); + } + return doc; + } + + @Test + void parsesTracingSupportTrue() { + ConnectionDescription description = DescriptionHelper.createConnectionDescription( + ClusterConnectionMode.SINGLE, CONNECTION_ID, hello(true)); + assertTrue(description.isTracingSupported()); + } + + @Test + void defaultsTracingSupportFalseWhenAbsent() { + ConnectionDescription description = DescriptionHelper.createConnectionDescription( + ClusterConnectionMode.SINGLE, CONNECTION_ID, hello(false)); + assertFalse(description.isTracingSupported()); + } + + @Test + void parsesTracingSupportFalseWhenExplicitlyFalse() { + BsonDocument helloResult = hello(false); + helloResult.put("tracingSupport", BsonBoolean.FALSE); + ConnectionDescription description = DescriptionHelper.createConnectionDescription( + ClusterConnectionMode.SINGLE, CONNECTION_ID, helloResult); + assertFalse(description.isTracingSupported()); + } + + @Test + void withTracingSupportPreservesOtherFields() { + ConnectionDescription original = DescriptionHelper.createConnectionDescription( + ClusterConnectionMode.SINGLE, CONNECTION_ID, hello(false)); + ConnectionDescription updated = original.withTracingSupport(true); + + assertTrue(updated.isTracingSupported()); + assertEquals(original.getConnectionId(), updated.getConnectionId()); + assertEquals(original.getMaxWireVersion(), updated.getMaxWireVersion()); + assertEquals(original.getServerType(), updated.getServerType()); + } +} diff --git a/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java b/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java new file mode 100644 index 00000000000..99324750344 --- /dev/null +++ b/driver-core/src/test/unit/com/mongodb/internal/observability/micrometer/MicrometerTraceParentTest.java @@ -0,0 +1,73 @@ +/* + * Copyright 2008-present MongoDB, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package com.mongodb.internal.observability.micrometer; + +import io.micrometer.observation.ObservationRegistry; +import io.micrometer.tracing.test.simple.SimpleTracer; +import io.micrometer.tracing.test.simple.SimpleTraceContext; +import io.micrometer.tracing.handler.DefaultTracingObservationHandler; +import com.mongodb.observability.micrometer.MongodbObservation; +import org.junit.jupiter.api.Test; + +import java.util.regex.Pattern; + +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertNull; +import static org.junit.jupiter.api.Assertions.assertTrue; + +class MicrometerTraceParentTest { + // SimpleTracer generates 8-byte (16-hex-char) trace IDs; real OTel uses 16-byte (32-hex-char). + // Accept either length in the unit test — the format check is what matters here. + private static final Pattern TRACEPARENT = + Pattern.compile("00-[0-9a-f]{16,32}-[0-9a-f]{16}-[0-9a-f]{2}"); + + @Test + void returnsTraceParentForSampledSpan() { + ObservationRegistry registry = ObservationRegistry.create(); + SimpleTracer tracer = new SimpleTracer(); + registry.observationConfig().observationHandler(new DefaultTracingObservationHandler(tracer)); + + MicrometerTracer micrometerTracer = new MicrometerTracer(registry, false, 1000, null); + Span span = micrometerTracer.nextSpan(MongodbObservation.MONGODB_COMMAND, "find", null, null); + span.openScope(); + // SimpleTracer creates spans with sampled=false by default; mark as sampled so + // traceParent() emits the header (flags=01). + ((SimpleTraceContext) tracer.lastSpan().context()).setSampled(true); + try { + String traceParent = span.context().traceParent(); + assertNotNull(traceParent); + assertTrue(TRACEPARENT.matcher(traceParent).matches(), traceParent); + } finally { + span.closeScope(); + span.end(); + } + } + + @Test + void returnsNullWhenNoTracingBridgeConfigured() { + ObservationRegistry registry = ObservationRegistry.create(); + MicrometerTracer micrometerTracer = new MicrometerTracer(registry, false, 1000, null); + Span span = micrometerTracer.nextSpan(MongodbObservation.MONGODB_COMMAND, "find", null, null); + span.openScope(); + try { + assertNull(span.context().traceParent()); + } finally { + span.closeScope(); + span.end(); + } + } +} diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index 7686cc15c41..2e67d80b43b 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -186,6 +186,7 @@ project-reactor-test = { module = "io.projectreactor:reactor-test" } reactive-streams-tck = { module = " org.reactivestreams:reactive-streams-tck", version.ref = "reactive-streams" } reflections = { module = "org.reflections:reflections", version.ref = "reflections" } +micrometer-tracing = { module = "io.micrometer:micrometer-tracing", version.ref = "micrometer-tracing" } micrometer-tracing-integration-test-bom = { module = " io.micrometer:micrometer-tracing-bom", version.ref = "micrometer-tracing" } micrometer-tracing-integration-test = { module = " io.micrometer:micrometer-tracing-integration-test" }