opentelemetry-process-context: implement Resource publishing (OTEP-4719)

[herin049]

Description

External profilers and observability tools (such as the OpenTelemetry eBPF Profiler) operate outside the instrumented process and have no access to resource attributes configured inside OpenTelemetry SDKs. OTEP-4719 (Process Context Sharing) introduces a standard mechanism for OpenTelemetry SDKs to publish resource attributes for access by out-of-process readers

This PR introduces the opentelemetry-process-context package as a Rust backed Python extension (built with maturin/pyo3) that implements the process outlined in OTEP-4719. It publishes the SDK's resource attributes to
a memory region that any external reader with access to /proc/<pid>/maps and /proc/<pid>/mem can discover.

Native Code

While a large portion of this logic in this package could be implemented without native code, practically it makes more sense to keep it as native code for the following reasons:

• Some of the syscalls used aren't exposed natively by Python (e.g. madvise), (although, ctypes could be used)
• Synchronization with atomics/memory barriers is required during publication which is best done natively
• By using the prost Rust crate, we eliminate the runtime dependency on protobuf
• Keeping the implementation native reduces runtime overhead (although updates are rare)
• Future proposals (e.g. thread local context, which will very likely be accepted as an OTEP) will required native code anyways

Python API

The introduced Python API has a very light surface area, consisting of two methods to publish and unpublish the process context.

def publish_context(resource: Resource, attributes: Attributes) -> None: ...
def unpublish_context() -> None: ...

Compatibility

The original OTEP explicitly mentions that this functionality is Linux only. However, I have tried to implement this package in such a way that limited testing/development can be performed natively on any Posix compatible system (e.g. MacOS). However, the full functionality of OTEP-4719 is only available on Linux and Windows is explicitly not supported.

Feature	Linux	macOS
Backing region	memfd_create("OTEL_CTX") -> mmap(MAP_PRIVATE)	Anonymous mmap
Reader discoverability	/proc/<pid>/maps shows memfd:OTEL_CTX	Not discoverable (no /proc)
Fork protection	madvise(MADV_DONTFORK)	Skipped
Mapping name	prctl(PR_SET_VMA_ANON_NAME, "OTEL_CTX")	Skipped
Publish timestamp clock	CLOCK_BOOTTIME	CLOCK_MONOTONIC
Production use	Fully supported	Development/testing only

Fixes #5291

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

uv run tox -e py314-test-opentelemetry-process-context

Does This PR Require a Contrib Repo Change?

• Yes. - Link to PR:
• No.

Checklist:

• Followed the style guidelines of this project
• Changelogs have been updated
• Unit tests have been added
• Documentation has been updated

[ocelotl]

Key components of this PR are written in Rust, reviewers of this repo may not be experts in this language. I suggest adding comments to all the rust files for ease of understanding:

publish.rs.txt
proto.rs.txt
lib.rs.txt
convert.rs.txt
build.rs.txt

[ocelotl]

I found a few things that may be required in the OTEP but missing here. If so, they are all very small changes to this PR.

A few other things are minor but also worth of considering.

[ocelotl]

Some more automatic review comments ✌️

[aabmass]

Very cool! My hunch is a lot of this rust code which implements the exposition format from the OTEP is not python specific, and maybe the OTEP authors have already written it or would be open to maintaining it elsewhere since it could apply across languages.

I'm not super up to date on this though

[herin049]

Very cool! My hunch is a lot of this rust code which implements the exposition format from the OTEP is not python specific, and maybe the OTEP authors have already written it or would be open to maintaining it elsewhere since it could apply across languages.

I'm not super up to date on this though

There is a PR for this package in the Rust repo: https://github.com/open-telemetry/opentelemetry-rust/pull/3460/changes#diff-a475efbb673d2bddd008746034e46ce4acd67d12f3629d7a2fd57120659276d7

But, it introduces a lot of heavy dependencies that we should try to avoid. Maybe there is a discussion to be had about making this lighter weight? I'm also unsure of whether that implementation is complete since it doesn't appear to handle forking properly from what I can tell.

[aabmass]

Python API

The introduced Python API has a very light surface area, consisting of two methods to publish and unpublish the process context.

def publish_context(resource: Resource) -> None: ...
def unpublish_context() -> None: ...

Are there use cases where it would need to be called from python code?

[herin049]

Very cool! My hunch is a lot of this rust code which implements the exposition format from the OTEP is not python specific, and maybe the OTEP authors have already written it or would be open to maintaining it elsewhere since it could apply across languages.
I'm not super up to date on this though

There is a PR for this package in the Rust repo: https://github.com/open-telemetry/opentelemetry-rust/pull/3460/changes#diff-a475efbb673d2bddd008746034e46ce4acd67d12f3629d7a2fd57120659276d7

But, it introduces a lot of heavy dependencies that we should try to avoid. Maybe there is a discussion to be had about making this lighter weight? I'm also unsure of whether that implementation is complete since it doesn't appear to handle forking properly from what I can tell.

Python API

The introduced Python API has a very light surface area, consisting of two methods to publish and unpublish the process context.

def publish_context(resource: Resource) -> None: ...
def unpublish_context() -> None: ...

Are there use cases where it would need to be called from python code?

This PR only contains the logic to publish the Resource (not the thread context information yet). From my understanding, the idea is that on application startup (e.g. via auto-instrumentation or manual), publish_context() will be called with the newly created Resource object created by the SDK. Likewise, we'll want to register a post-fork handler to publish the newly created Resource since the behavior is that the process context is not inherited by forked processes. Ultimately, there needs to be a way for the Python SDK to associate its Resource object with the one that is published which is why these methods exist.

There will be a follow-up PR which will call publish_context() during auto-instrumentation if the opentelemetry-process-context package is available.

[aabmass]

This PR only contains the logic to publish the Resource (not the thread context information yet).

Ahh thanks I missed that makes sense. Do you mind updating the PR title with "Sharing Resource Attributes". If it's OK, I'd like to wait for @lzchen to take a pass at the Rust code and approve the approach.

[xrmx]

I think one question should be answered in the description of this PR: why does this need native code? My understanding is that we need it for the thread local stuff that is not an OTEP yet.

When I prototyped an ancient version of this (the protocol was different, a socket was involved and the the profile read to well-known locations instead of memfd) I implemented a small wrapper on a thread_local variable in C and handled everything in Python:

#define PY_SSIZE_T_CLEAN
#define Py_LIMITED_API 0x03080000
#include <stddef.h>
// compilers that do not support C23 expose thread_local as a macro
#include <threads.h>
#include <Python.h>

void *apm_profiling_correlation_process_storage_v1 = NULL;

thread_local void* apm_profiling_correlation_tls_v1 = NULL;

static PyObject *
up_set_correlation_process_storage(PyObject *self, PyObject *args)
{
    if (!PyArg_ParseTuple(args, "y*", &apm_profiling_correlation_process_storage_v1))
        return NULL;
    Py_RETURN_TRUE;
}

static PyObject *
up_set_correlation_tls_storage(PyObject *self, PyObject *args)
{
    if (!PyArg_ParseTuple(args, "y*", &apm_profiling_correlation_tls_v1))
        return NULL;
    Py_RETURN_TRUE;
}

static PyMethodDef UniversalProfilingMethods[] = {
    {"set_correlation_process_storage",  up_set_correlation_process_storage, METH_VARARGS,
     "Set correlation process storage for Universal Profiling integration."},
    {"set_correlation_tls_storage",  up_set_correlation_tls_storage, METH_VARARGS,
     "Set correlation TLS storage for Universal Profiling integration."},
    {NULL, NULL, 0, NULL}        /* Sentinel */
};

PyDoc_STRVAR(universal_profiling_doc, "Helper functions for integrations with Universal Profiling.");

static struct PyModuleDef UniversalProfilingModule = {
    PyModuleDef_HEAD_INIT,
    "universal_profiling",
    universal_profiling_doc,
    -1,       /* size of per-interpreter state of the module,
                 or -1 if the module keeps state in global variables. */
    UniversalProfilingMethods
};

PyMODINIT_FUNC
PyInit_universal_profiling(void)
{
    return PyModule_Create(&UniversalProfilingModule);
}

[xrmx]

Does the epbf profiler works on these many platforms? I think it runs on arm64 and amd64 only.

[herin049]

I think one question should be answered in the description of this PR: why does this need native code? My understanding is that we need it for the thread local stuff that is not an OTEP yet.

I did mention this in the SIG that a large portion of this logic could be implemented without native code (maybe all), but practically it made more sense to keep it in native code for the following reasons:

• Some of the syscalls used aren't exposed natively by Python (e.g. madvise), which would require the use of ctypes. Not a huge deal, but makes the implementation a little messy.
• On the correctness side, the synchronization portions require atomics/memory barriers which is best done natively
• By using the prost crate, we eliminate the runtime dependency on protobuf

Also, (as you mentioned), the thread context proposal will very likely be accepted which requires native code to be introduced anyways.

[github-actions]

This PR has been automatically marked as stale because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 days of this comment.
If you're still working on this, please add a comment or push new commits.

[ocelotl]

Don't close this PR