Skip to content

Log topics: app_log_{warn,error}_total is unlabelled across almost all of pluto #588

Description

@emlautarom1-agent

What is missing

MetricsLayer labels app_log_warn_total / app_log_error_total with a topic taken from the nearest enclosing span, falling back to an empty string: event_topic.

Pluto sets that span field in exactly two places — health/checker.rs#L60 and stacksnipe.rs#L100. Everything else, including all of crates/core, is counted under topic="".

Charon derives the same label from a context.Context value (WithTopic, metricsTopicFromCtx) and sets ~20 topics covering every component, with app-start as a catch-all over app.Run and the lifecycle hooks. Its unlabelled bucket is effectively empty.

Effects

  • A warn/error rate spike carries no signal about which component produced it, since 209 of pluto's 211 warn!/error! sites share one label value.
  • PromQL that selects or excludes topics behaves differently against pluto and charon. Alert rules and Grafana panels written against charon's labels do not transfer.

Suggested fix

Span context does not cross tokio::spawn. Measured against MetricsLayer: a warn from a bare spawn inside a topic span lands on topic="", while the same future wrapped in .instrument(Span::current()) keeps the label. Hand-instrumenting each of the ~130 spawn sites is the wrong shape.

  1. Add a spawn helper that attaches Span::current() to the future, restoring context-like propagation once. Components then inherit their topic by default instead of silently losing it.
  2. Set topic on one root span per long-running component (~20), reusing charon's names: tracker, vmock, sched, vapi, qbft, bcast, sigagg, parsigex, p2p, peerinfo, dkg, relay.

Keep topic values &'static str so the label stays bounded.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingrustPull requests that update rust code

    Type

    No type

    Projects

    Status
    Todo

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions