Skip to content

[SDK] Support non-exporting pipeline log record processors (needed for EventToSpanEventBridgeProcessor) #4454

Description

@om7057

Summary

EventToSpanEventBridgeProcessor (proposed in #4309) bridges log-record events onto the live Span referenced by the log record's resolved context. It needs a LogRecordProcessor that can:

  • read the resolved context (Context/live Span) at OnEmit() time
  • read the log record's own data (body, attributes) without taking exclusive ownership away from downstream processors/exporters

The current LogRecordProcessor interface doesn't support this shape of "non-exporting", read/mutate pipeline processor, so the bridge processor can't be implemented correctly without new SDK architecture.

Background

From review on #4309:

Processors like the bridge processor will require some new architectural components and changes to existing components to be spec compliant and maintain reasonable performance. This is in addition to the Logger API level EmitLogRecordWithContext change, which is needed.

The architecture components / changes that seem to be required include:

  1. Standardize the SDK built-in exporters (OTLP, ostream) on a recordable with a read-write interface. Pipeline processors need to read data from the record and not duplicate it.
  2. Create a new processor interface for composable pipeline processors that can read and mutate data but do not own the recordable life-cycle.
  3. Create a new root level processor that supports pipeline processors
  4. Update the LoggerContext to support setting a new root processor.

The concrete problem: with today's LogRecordProcessor interface, a configuration like

logger_provider:
  processors:
    my_logs_filter:
    my_log_record_sanitizer:
    event_to_span_bridge:
    batch:
      exporter:
        otlp_http:

fans out via MultiLogRecordProcessor instead of forming a pipeline: each processor gets an independent copy of the record via OnEmit(), which transfers ownership. That means N copies of the record (N times the recording cost) and the exporter at the end receives the raw, unfiltered/unsanitized record rather than the output of the upstream stages, since mutations made by an earlier stage are made on a copy that is then discarded.

See the full discussion: #4309 (review) and #4309 (comment)

Scope

  • Standardize SDK built-in exporters (OTLP, ostream) on a ReadWriteLogRecord-style recordable.
  • New processor interface for composable pipeline processors that mutate data in place without owning the recordable lifecycle.
  • New root-level processor implementing a pipeline of these composable processors.
  • LoggerContext support for installing this new root processor.
  • Once the above lands, implement EventToSpanEventBridgeProcessor against the new interface and reintroduce it into declarative configuration (currently the config model/parser accepts the event_to_span_bridge processor block, but SdkBuilder only logs a warning that it is not yet supported; see the FIXME-SDK comment in sdk_builder.cc).

Status

Pending design and implementation of the components above. #4309 was scoped down to just the configuration model and YAML parser per this discussion, with the actual processor implementation deferred until this issue is resolved.

Metadata

Metadata

Assignees

No one assigned

    Labels

    discussTo discuss in SIG meetingtriage/acceptedIndicates an issue or PR is ready to be actively worked on.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions