test(compose): gate the sched log topic in mixed clusters - #586
test(compose): gate the sched log topic in mixed clusters#586emlautarom1-agent[bot] wants to merge 2 commits into
Conversation
![emlautarom1-agent[bot]](https://avatars.githubusercontent.com/u/31224949?s=60&v=4){kind=small}
| @@ -236,12 +236,6 @@ func smokeScenarios() []smokeScenario { | |||
| } | |||
| // `pluto run` fails fast on --synthetic-block-proposals. | |||
| conf.SyntheticBlockProposals = false | |||
There was a problem hiding this comment.
Heads up if you run this locally. The scenario fails today, before and after this change, on #585 — the inclusion checker rejects charon beaconmock's block responses and re-warns every tick, so both pluto nodes trip Warn Log Rate. To reproduce the charon-side verdict this change is actually about, temporarily add conf.AlertExcludeJobs = []string{"node2", "node3"} here; it passes with "No alerts detected". That exemption is deliberately not committed.
| // TestWriteAlertRulesWarnTopics asserts the Warn Log Rate gate excludes exactly | ||
| // the two charon mock-noise topics, so every other topic — including the empty | ||
| // one pluto emits — stays gated. | ||
| func TestWriteAlertRulesWarnTopics(t *testing.T) { | ||
| content := writeRules(t, NewDefaultConfig()) | ||
| require.Contains(t, content, `increase(app_log_warn_total{topic!~"vmock|tracker"}[30s]) > 2`) | ||
| } |
There was a problem hiding this comment.
Deliberately a stricter test than the one it replaces. The previous case asserted the knob's append behaviour; with the knob gone this pins the exclusion list exactly, so adding a third topic has to be an explicit edit here rather than a scenario-local escape hatch. Given that #583 exists because a topic was excluded on a rationale nobody re-checked, making the list hard to extend quietly is the point.
Charon's infosync exchange with pluto peers completes cleanly over
/charon/priority/2.0.0, so no warning is logged under topic=sched and the
Warn Log Rate gate has nothing to suppress. Measured across three
mixed_2_charon_2_pluto runs: app_log_warn_total{topic="sched"} is never
created on the charon nodes, no sched-topic WARN or ERROR line appears in
their logs, and charon never logs "P2P sending failing" — the warning the
exclusion named.
mixed_2_charon_2_pluto was the exclusion's only user, so
AlertWarnExcludeTopics goes with it. The two remaining warn-topic
exclusions are pinned by test instead, since both name charon components
and neither occurs on pluto — pluto sets a log topic on almost nothing, so
its warnings carry no topic label and stay gated.
Closes #583
7287d4f to
addc64e
Compare
| // Both are mock artifacts, not node behavior; every other warn topic stays | ||
| // gated. Both name charon components and only ever occur on charon nodes: | ||
| // pluto sets a log topic on almost nothing, so its warnings carry no topic | ||
| // label and this exclusion never applies to them. | ||
| const warnTopics = "vmock|tracker" |
There was a problem hiding this comment.
Why removing sched is safe, and the trap to avoid. Pluto attaches a log topic in exactly one live code path (the health checker's health span). Everything else it warns about is recorded with no topic label, and Prometheus drops empty label values at ingest, so those series carry no topic key at all. A negative regex matcher selects a series whose label is absent, which means topic!~"vmock|tracker" and topic!~"vmock|tracker|sched" gate pluto's warnings identically — verified with promtool test rules on the pinned Prometheus version. That is why dropping sched cannot alter the pluto side of the verdict, and why the control run failed the same way as the un-excluded one.
The trap: both remaining topics are named after charon components, and the matcher itself is not scoped by job. If pluto ever adopts charon's topic convention, a pluto node emitting topic=tracker would start being excluded silently. Worth scoping this to charon jobs at that point rather than discovering it the way #583 was discovered.
(Charon labels its own untopiced warnings topic="unknown" rather than leaving the label empty — a divergence that happens not to matter here, since both forms are gated.)
Closes #583
Summary
mixed_2_charon_2_plutoexempted theschedlog topic from theWarn Log Rategate on the grounds that pluto did not serve/charon/priority/2.0.0but this is no longer true with #574.This PR removes this suppression and verifies that test continue to stay green. We also drop the additional config code that was introduced only to deal with this suppression.
Out of scope
Currently, the tests do not pass cleanly with only this fix since #585 is required:
inclusion checker rejects its own simnet mock's block responses on every due slot, flooding each pluto node's warn log and resulting in failing tests.