Skip to content

Fix flaky compactor tests: ignore userIndexUpdateLoop error logs in exact log-set assertions - #7859

Open
CharlieTLe with Copilot wants to merge 2 commits into
masterfrom
copilot/fix-flaky-compactor-test
Open

CharlieTLe with Copilot wants to merge 2 commits into
masterfrom
copilot/fix-flaky-compactor-test

Conversation

Copilot AI commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

What this PR does:

TestCompactor_ShouldCompactAllUsersOnShardingEnabledButOnlyOneInstanceRunning fails intermittently on the arm64 -race CI job: the background userIndexUpdateLoop ticks every ~100ms in tests (via prepare()'s user-index strategy) and logs a ring error whenever no healthy instance exists — before the lifecycler is ACTIVE or during LEAVING at shutdown. Any tick landing in the test window leaks an extra line into the exact log-set assertion, failing with elements differ.

removeIgnoredLogs already suppresses two of the loop's log lines via map entries; the two error lines were missed because their variable err= payloads require regexes. This adds them:

ignoredLogStringsRegexList := []*regexp.Regexp{
	regexp.MustCompile(`^level=(info|debug|warn) component=cleaner .+$`),
	regexp.MustCompile(`^level=info component=compactor msg="set state" .+$`),
	regexp.MustCompile(`^level=error component=compactor msg="failed to check if compactor owns updating user index" err=.+$`),
	regexp.MustCompile(`^level=error component=compactor msg="failed to update user index" err=.+$`),
}

Since all 14 exact log-set assertions in compactor_test.go and compactor_paritioning_test.go funnel through removeIgnoredLogs, this single change covers every sharding-enabled test exposed to the same race. Regexes were verified against real go-kit logfmt output, including the exact err="at least 1 live replicas required, could only find 0" line from the failed CI jobs. Test-only change; no production behavior modified.

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags

Co-authored-by: CharlieTLe <3375195+CharlieTLe@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix flaky test for compactor log output assertion Fix flaky compactor tests: ignore userIndexUpdateLoop error logs in exact log-set assertions Sep 22, 2026
Copilot AI requested a review from CharlieTLe September 22, 2026 18:56
@CharlieTLe
CharlieTLe marked this pull request as ready for review September 22, 2026 21:52
@CharlieTLe
CharlieTLe requested a review from a team as a code owner September 22, 2026 21:52

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

2 participants