Skip to content

test(compose): gate the sched log topic in mixed clusters #586

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
emlautarom1-agent wants to merge 3 commits into
base: main
Choose a base branch
from
Open

test(compose): gate the sched log topic in mixed clusters #586

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a3ece2b
test(compose): gate the sched log topic in mixed clusters
emlautarom1-agent[bot] Aug 4, 2026
addc64e
chore(nix): add go, gopls and delve to the dev shell
emlautarom1-agent[bot] Aug 4, 2026
5007a3c
refactor: simplify comments in alert rules and tests for clarity
emlautarom1 Aug 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions flake.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@
cargo-machete
protobuf
oas3-gen
go
gopls
delve
];

shellHook = ''
Expand Down
1 change: 0 additions & 1 deletion test-infra/compose/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,6 @@ Scenarios that intentionally degrade the cluster tune the gate via config, not t
| Config knob | Effect | Used by |
|-------------|--------|---------|
| `AlertExcludeJobs` | exempt a node from the per-node rules (never from `Pluto Down`) | `1_of_4_down`, `1_of_3_down` |
| `AlertWarnExcludeTopics` | extra warn-topic exclusions | `mixed_2_charon_2_pluto` (excludes `sched` until pluto serves infosync) |
| `AlertDisableRules` | drop an entire rule | `1_of_3_down` (disables the error-rate gates — a downed round-1 leader makes every third proposer duty unrecoverable on the mock) |

## Versioning
Expand Down
7 changes: 0 additions & 7 deletions test-infra/compose/config.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,13 +163,6 @@ type Config struct {
// clean.
AlertExcludeJobs []string `json:"alert_exclude_jobs,omitempty"`

// AlertWarnExcludeTopics appends log topics to the Warn Log Rate
// exclusion list on top of the built-in charon mock noise (see
// writeAlertRules). Scenario-scoped escape hatch: e.g. mixed
// charon/pluto clusters exclude "sched" because charon's infosync warns
// each epoch until pluto serves /charon/priority/2.0.0 (#402B).
AlertWarnExcludeTopics []string `json:"alert_warn_exclude_topics,omitempty"`

// AlertDisableRules drops entire alert rules (by name, see
// alertRuleNames) from the generated rules. Last-resort scenario knob
// for cluster-wide degradation that per-job exclusion cannot express:
Expand Down
11 changes: 5 additions & 6 deletions test-infra/compose/define.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -432,9 +432,8 @@ var alertRuleNames = map[string]bool{
// writeAlertRules writes the prometheus alert rules evaluated by the smoke
// tests. Rules are generated (not static) because the expressions depend on
// config: scenarios that deliberately degrade a node exempt its job via
// conf.AlertExcludeJobs, mixed-impl scenarios extend the warn-topic
// exclusions via conf.AlertWarnExcludeTopics, and cluster-wide degradations
// drop whole rules via conf.AlertDisableRules.
// conf.AlertExcludeJobs, and cluster-wide degradations drop whole rules via
// conf.AlertDisableRules.
//
// Charon's "Outstanding Duty Rate" rule (core_bcast_broadcast_total -
// core_scheduler_duty_total > 50) is deliberately not ported: a node cannot
Expand Down Expand Up @@ -477,9 +476,9 @@ func writeAlertRules(dir string, conf Config) error {
// every successful proposal epoch warns "Broadcasted block/attestation
// never included on-chain" (the better the cluster works, the more it
// warns).
// Both are mock artifacts, not node behavior; all other warn topics stay
// gated unless a scenario opts out via AlertWarnExcludeTopics.
warnTopics := strings.Join(append([]string{"vmock", "tracker"}, conf.AlertWarnExcludeTopics...), "|")
// Both are mock artifacts, not node behavior; every other warn topic stays
// gated.
const warnTopics = "vmock|tracker"
Comment on lines +479 to +481

@emlautarom1-agent emlautarom1-agent Bot Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why removing sched is safe, and the trap to avoid. Pluto attaches a log topic in exactly one live code path (the health checker's health span). Everything else it warns about is recorded with no topic label, and Prometheus drops empty label values at ingest, so those series carry no topic key at all. A negative regex matcher selects a series whose label is absent, which means topic!~"vmock|tracker" and topic!~"vmock|tracker|sched" gate pluto's warnings identically — verified with promtool test rules on the pinned Prometheus version. That is why dropping sched cannot alter the pluto side of the verdict, and why the control run failed the same way as the un-excluded one.

The trap: both remaining topics are named after charon components, and the matcher itself is not scoped by job. If pluto ever adopts charon's topic convention, a pluto node emitting topic=tracker would start being excluded silently. Worth scoping this to charon jobs at that point rather than discovering it the way #583 was discovered.

(Charon labels its own untopiced warnings topic="unknown" rather than leaving the label empty — a divergence that happens not to matter here, since both forms are gated.)


// The broadcast-liveness expression must fail when a node exposes NO
// core_bcast_broadcast_total series at all: the counter is created on
Expand Down
13 changes: 5 additions & 8 deletions test-infra/compose/rules_internal_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,14 +67,11 @@ func TestWriteAlertRulesExcludesDegradedJobs(t *testing.T) {
require.Contains(t, content, "expr: up == 0")
}

// TestWriteAlertRulesWarnTopicExtension asserts scenario-scoped warn-topic
// exclusions append to the built-in mock-noise list.
func TestWriteAlertRulesWarnTopicExtension(t *testing.T) {
conf := NewDefaultConfig()
conf.AlertWarnExcludeTopics = []string{"sched"}

content := writeRules(t, conf)
require.Contains(t, content, `increase(app_log_warn_total{topic!~"vmock|tracker|sched"}[30s]) > 2`)
// TestWriteAlertRulesWarnTopics asserts the Warn Log Rate gate excludes exactly
// the two charon mock-noise topics.
func TestWriteAlertRulesWarnTopics(t *testing.T) {
content := writeRules(t, NewDefaultConfig())
require.Contains(t, content, `increase(app_log_warn_total{topic!~"vmock|tracker"}[30s]) > 2`)
}
Comment on lines +70 to 75

@emlautarom1-agent emlautarom1-agent Bot Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deliberately a stricter test than the one it replaces. The previous case asserted the knob's append behaviour; with the knob gone this pins the exclusion list exactly, so adding a third topic has to be an explicit edit here rather than a scenario-local escape hatch. Given that #583 exists because a topic was excluded on a rationale nobody re-checked, making the list hard to extend quietly is the point.


// TestWriteAlertRulesDropsOutstandingDuty pins the removal of charon's dead
Expand Down
6 changes: 0 additions & 6 deletions test-infra/compose/smoke/smoke_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -236,12 +236,6 @@ func smokeScenarios() []smokeScenario {
}
// `pluto run` fails fast on --synthetic-block-proposals.
conf.SyntheticBlockProposals = false

@emlautarom1-agent emlautarom1-agent Bot Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heads up if you run this locally. The scenario fails today, before and after this change, on #585 — the inclusion checker rejects charon beaconmock's block responses and re-warns every tick, so both pluto nodes trip Warn Log Rate. To reproduce the charon-side verdict this change is actually about, temporarily add conf.AlertExcludeJobs = []string{"node2", "node3"} here; it passes with "No alerts detected". That exemption is deliberately not committed.

// Charon triggers infosync (/charon/priority/2.0.0) every
// epoch; pluto does not serve the protocol yet (#402B), so
// charon nodes warn "P2P sending failing" under topic=sched
// twice per epoch. Exempt that topic in mixed clusters until
// the protocol lands; drop this with #402B.
conf.AlertWarnExcludeTopics = []string{"sched"}
},
},
{
Expand Down
Loading