feat!: emit per-chunk streaming span events on backend spans

[ajbozarth]

Issue

Fixes #1051

Description

Adds streaming span events on the backend generation span (chat / text_completion), so per-chunk streaming cadence is visible on individual traces rather than only as aggregate metrics.

This introduces a reusable event path for backend spans:

• A generic generation_event hook (GenerationEventPayload carrying event_name + a data dict), fired via ModelOutputThunk._emit_event.
• A BackendTracingPlugin.on_generation_event handler that dispatches on event_name and records the event on the in-flight backend span via the existing add_span_event. Future backend-span events reuse this path.

Its first consumer is an opt-in chunk_processed event, emitted once per streamed chunk during astream() when MELLEA_EMIT_CHUNK_EVENTS=true (off by default, since a long response produces one event per chunk). It carries mellea.generation.chunk_index and mellea.generation.chunk_text_length.

Design notes:

• Core stays telemetry-agnostic: the emit site passes raw domain values by keyword; the OpenTelemetry attribute vocabulary lives only in the plugin.
• The gate is a core-side hook gate (MELLEA_EMIT_CHUNK_EVENTS), checked before the hook fires so the disabled path costs only an env read.
• The handler runs SEQUENTIAL so the event is appended before on_post_call ends the span.

Span event attribute keys across the tracing plugins are also namespaced (previously emitted bare, e.g. chunk_index), following mellea.<origin>.<leaf> where the origin names the subsystem the value comes from. Note: the streaming-span event attributes were previously released, so renaming them is a breaking change for existing trace queries.

Also fixes a latent import-layering bug this surfaced: mellea/core/utils.py imported mellea.telemetry at module scope, forming a core → telemetry → core.base cycle. Those imports are now lazy (function-local), which is behavior-preserving.

Open question for review

GenerationEventPayload carries an untyped data: dict rather than a typed event class (contrast StreamEvent, which the streaming hook uses). This keeps core telemetry-agnostic and avoids a payload class per event type. The known event_name → data key contracts are documented in the GenerationEventPayload docstring. Still worth confirming before other subscribers depend on it: is the generic data dict the right call, or should backend events get a typed hierarchy?

Testing

• Tests added to the respective file if code was changed
• New code has 100% coverage if code was added
• Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

Attribution

• AI coding assistants used

---

Adding a new component, requirement, sampling strategy, or tool?

• Component
• Requirement
• Sampling Strategy
• Tool

[jakelorocco]

The implementation here looks good. I am slightly confused about the end goal of this event / hook (especially given the issue it fixes). Is the end goal of this hook to ultimately replace the StreamEvent type? Otherwise, I'm not quite understanding what the different data dicts might eventually be here.

[jakelorocco]

Is there a performance hit to doing the os.getenv multiple times here? Should we move that specific check into some code that runs once per import?

[ajbozarth]

I can check, this was in a helper originally so I may have messed that up when I moved this inline

[ajbozarth]

I looked into this and this is trivial as a performance hit, but I can cache the env var if you're still concerned. It's worth noting this will only run when streaming and only once per astream() call.

[jakelorocco]

I feel that at least for the data types we define; we should have typed implementations even if those types don't necessarily get propagated to the function processing the data.

[ajbozarth]

My reason for not making data classes is that these events are not intended to be user consumed like Streaming APOI events are. The hooks are available for devs, but their content is essentially telemetry data. I also didn't want to be making a bunch of new individual event hooks either. If this is an issue I'm ok either making event data classes or separate event hooks, I just felt this wants worth it as I saw this as a dev interface compared to streamings user interface

[ajbozarth]

Is the end goal of this hook to ultimately replace the StreamEvent type

this work is actually agnostic to the new streaming API and is directly on the backend telemetry, though this would nest a few layers under that streaming telemetry

Otherwise, I'm not quite understanding what the different data dicts might eventually be here.

I made this generic because #1073 intends to add more events and would re-use the same hook helper

[planetf1]

Went through Bob's review findings against cfca37cf and ran the test suite (66 telemetry + 376 plugin tests pass, ruff/mypy clean). None of the three WARNINGs hold up as blocking, but there's one real issue worth fixing before merge.

WARNING 1 (namespace) — not a bug, don't rename. mellea.generation.chunk_index (backend span, raw provider chunks from astream()) and mellea.streaming.chunk_index (application span, chunking-strategy output units) count different things — one astream() call can span zero or many strategy chunks. Merging them onto one key would conflate incompatible units. Separate namespaces are correct here.

That said, the PR does introduce a real split worth a follow-up: it namespaces span event attributes but leaves existing span attributes bare, so the same span now carries both mellea.full_text_length (attribute) and mellea.streaming.full_text_length (event on that span) for the same value. Not blocking — pre-existing pattern extended, not introduced.

WARNING 2 (negative chunk_text_length) — the mechanism doesn't exist. The emit path is gated on self.generation.streaming, and every streaming backend only ever appends to a str accumulator — there's no mid-stream Pydantic validation that could shrink it (parsing happens once, after the stream ends, under do_set_computed). The suggested guard would also introduce a bug: the else branch sets chunk_text_length = None, which add_span_event filters, so the event would fire with zero attributes.

WARNING 3 (docs) — comparing against the wrong table. The tracing doc documents span events by name only, never by attribute key (see the existing iteration/repair/chunk entries) — the attribute-key table is a different section covering span attributes. The new paragraph matches the existing convention.

What I'd actually change before merge:

1. MELLEA_EMIT_CHUNK_EVENTS breaks the established prefix convention — see inline suggestion on mellea/core/base.py.
2. It's also missing from the env var reference table in docs/docs/observability/telemetry.md (that file isn't touched by this PR, so no inline suggestion possible there) — add a row once the rename above lands:

   | `MELLEA_TRACES_CHUNK_EVENTS` | Emit a `chunk_processed` span event per streamed chunk on backend spans | `false` |
   

Everything else (the ! on commit messages given the acknowledged breaking rename, GenerationEventPayload export, the docstring wording) is a nit — happy to leave those to your judgement.

[planetf1]

A few small follow-ups from re-reviewing the chunk-events changes — none blocking.

[ajbozarth]

@jakelorocco @planetf1 I believe I have addressed or responded to all your feedback so far if you could re-review and resolve the threads.

[planetf1]

All 4 threads addressed.

• Env var rename — is a better fit than since it gates a hook, not telemetry directly. Proper subsystem prefix.
• Test assertion — Parametrised for both emit on/off, covering both branches. Attribute values pinned in .
• Doc: requires tracing — Pushback correct. The table keeps all vars consistent without annotating tracing dependencies.
• Doc: terminal chunk — Reasonable to omit as implementation detail.

The namespace refactor across all span-event attributes to mellea.* prefixes is solid — prevents OTel reserved key collisions and is architecturally consistent. Tests pass.

[planetf1]

<deleted. reposted in error after api failure>