Flaky test: TestGetRules/No_Sharding_with_Alert_state_filter_for_inactive_alerts — asserts the transient pre-evaluation 'unknown' alert state

[CharlieTLe]

Summary

TestGetRules/No_Sharding_with_Alert_state_filter_for_inactive_alerts fails intermittently with expected: 2, actual: 1 at pkg/ruler/ruler_test.go:1444.

Seen on master run 35636592621, job test (arm64), on commit c9000f7 — a commit that changes only CHANGELOG.md, so it is unrelated to the code under test.

This is the same parent test as #6106 (closed, 2024), but a different subtest and a different root cause, so filing separately.

Failure output

--- FAIL: TestGetRules (5.55s)
    --- FAIL: TestGetRules/No_Sharding_with_Alert_state_filter_for_inactive_alerts (0.07s)
        ruler_test.go:1444:
            	Error Trace:	/__w/cortex/cortex/pkg/ruler/ruler_test.go:1444
            	            				/__w/cortex/cortex/pkg/ruler/ruler_test.go:1403
            	            				/__w/cortex/cortex/pkg/ruler/ruler_test.go:1432
            	Error:      	Not equal:
            	            	expected: 2
            	            	actual  : 1
            	Test:       	TestGetRules/No_Sharding_with_Alert_state_filter_for_inactive_alerts
FAIL	github.com/cortexproject/cortex/pkg/ruler	33.670s

Root cause

Despite the subtest name, the filter under test is unknownStateFilter, not inactive:

https://github.com/cortexproject/cortex/blob/master/pkg/ruler/ruler_test.go#L920-L932

"No Sharding with Alert state filter for inactive alerts": {
	sharding: false,
	rulesRequest: RulesRequest{
		State:         unknownStateFilter, // Prometheus v2.x uses "unknown" for unevaluated alerts
		MaxRuleGroups: -1,
	},

unknown is Prometheus' state for an alerting rule that has not yet been evaluated (vendor/github.com/prometheus/prometheus/rules/alerting.go:57-58), and returnByState is an exact string match against rule.State().String(). So the subtest asserts that all of user1's alerting rules are still in their pre-evaluation state at the moment GetRules is called. That is a transient startup condition, and nothing in the test holds it still.

Once syncRules loads the groups, the rule manager starts each group's evaluation goroutine. The first evaluation is scheduled by Group.run:

// Wait an initial amount to have consistently slotted intervals.
evalTimestamp := g.EvalTimestamp(time.Now().UnixNano()).Add(g.interval)
select {
case <-time.After(time.Until(evalTimestamp)):

EvalTimestamp aligns to the interval grid with a per-group offset of g.hash() % interval, so the first evaluation lands at a point in (0, interval] from now — the exact delay is a function of the group's hash and the wall-clock time the test happens to start. The groups here use Interval: 10 * time.Minute, so the delay is usually minutes and the test passes. But when the current time sits just past a group's aligned slot, the delay collapses toward zero, that group evaluates within the assertion window, its alerting rules leave StateUnknown, and they stop matching the unknown filter. One of user1's two rules drops out and the count comes back 1.

This is why the failure is rare and not tied to a code change. It is also not really arm64-specific — the arm64 runners are just slower, which widens the window between syncRules and the assertion.

Proposed fix

The durable fix is to stop asserting on a state that the production code is actively racing to change. Options, roughly in order of preference:

1. Prevent evaluation from completing for the duration of the subtest. Give the test ruler a QueryFunc that blocks on a channel closed in t.Cleanup. An alerting rule whose evaluation has not returned stays in StateUnknown, making the subtest deterministic regardless of timing. This keeps the subtest testing what it means to test — the unknown filter — without depending on scheduling.
2. Assert the filter, not the startup window. Drive the rules to a known state (evaluate them deliberately, then assert on inactive/pending/firing), and cover unknown with a unit test over returnByState rather than a live manager.

Worth also renaming the subtest while in there — it says "inactive alerts" but filters on unknown, which is what made this failure read as a genuine regression at first glance.