What is missing
MetricsLayer labels app_log_warn_total / app_log_error_total with a topic taken from the nearest enclosing span, falling back to an empty string: event_topic.
Pluto sets that span field in exactly two places — health/checker.rs#L60 and stacksnipe.rs#L100. Everything else, including all of crates/core, is counted under topic="".
Charon derives the same label from a context.Context value (WithTopic, metricsTopicFromCtx) and sets ~20 topics covering every component, with app-start as a catch-all over app.Run and the lifecycle hooks. Its unlabelled bucket is effectively empty.
Effects
- A warn/error rate spike carries no signal about which component produced it, since 209 of pluto's 211
warn!/error! sites share one label value.
- PromQL that selects or excludes topics behaves differently against pluto and charon. Alert rules and Grafana panels written against charon's labels do not transfer.
Suggested fix
Span context does not cross tokio::spawn. Measured against MetricsLayer: a warn from a bare spawn inside a topic span lands on topic="", while the same future wrapped in .instrument(Span::current()) keeps the label. Hand-instrumenting each of the ~130 spawn sites is the wrong shape.
- Add a spawn helper that attaches
Span::current() to the future, restoring context-like propagation once. Components then inherit their topic by default instead of silently losing it.
- Set
topic on one root span per long-running component (~20), reusing charon's names: tracker, vmock, sched, vapi, qbft, bcast, sigagg, parsigex, p2p, peerinfo, dkg, relay.
Keep topic values &'static str so the label stays bounded.
What is missing
MetricsLayerlabelsapp_log_warn_total/app_log_error_totalwith atopictaken from the nearest enclosing span, falling back to an empty string:event_topic.Pluto sets that span field in exactly two places —
health/checker.rs#L60andstacksnipe.rs#L100. Everything else, including all ofcrates/core, is counted undertopic="".Charon derives the same label from a
context.Contextvalue (WithTopic,metricsTopicFromCtx) and sets ~20 topics covering every component, withapp-startas a catch-all overapp.Runand the lifecycle hooks. Its unlabelled bucket is effectively empty.Effects
warn!/error!sites share one label value.Suggested fix
Span context does not cross
tokio::spawn. Measured againstMetricsLayer: a warn from a bare spawn inside atopicspan lands ontopic="", while the same future wrapped in.instrument(Span::current())keeps the label. Hand-instrumenting each of the ~130 spawn sites is the wrong shape.Span::current()to the future, restoring context-like propagation once. Components then inherit their topic by default instead of silently losing it.topicon one root span per long-running component (~20), reusing charon's names:tracker,vmock,sched,vapi,qbft,bcast,sigagg,parsigex,p2p,peerinfo,dkg,relay.Keep topic values
&'static strso the label stays bounded.