What happened
PR #3295, a Renovate security bump of in-toto-golang, was blocked for 106 days partly due to flaky CI failures. The Checks workflow failed on 6 of 9 runs. Two failures (workflow runs 30586521570 on Jul 30 and 30907557496 on Aug 4) were caused by TestFilteringWithMixedRules in internal/evaluator/conftest_evaluator_unit_core_test.go panicking with a test timeout during OPA capabilities JSON parsing. The test performs I/O-heavy operations (creating temp directories, building tar.gz archives, compiling Rego policies via NewConftestEvaluatorWithNamespace()) that can exceed the timeout under CI load. The same test passed on subsequent runs with no code changes, confirming it is a flake rather than a real failure.
What could go better
The unit test timeout (set in the Makefile at line 110) is tight relative to the work TestFilteringWithMixedRules performs. Under CI load, OPA policy compilation occasionally takes longer than the budget allows, causing a panic that fails the entire test suite. This is a test-fixture resilience problem — the production code path (conftest evaluator initialization) is inherently variable in execution time depending on system load, and the test harness does not account for that variability. Confidence is high: the same test failed and then passed on identical commits with no intervening code change, and the failure is in a known-expensive code path (OPA compilation).
Proposed change
In the Makefile, increase the -timeout flag for unit tests from its current value to at least 30s to provide headroom for I/O-heavy tests like TestFilteringWithMixedRules that compile OPA policies. Alternatively, if the tight timeout is intentional to catch slow tests, refactor TestFilteringWithMixedRules to extract the archive creation and policy loading into a TestMain or shared fixture so that expensive setup runs once rather than per-test-case, reducing the time pressure on the overall test budget.
Validation criteria
After the change, TestFilteringWithMixedRules should not produce timeout panics. Validate by running make test-unit at least 10 consecutive times (or checking the next 10 CI runs on the release-v0.7 branch) and confirming zero timeout-related failures in the conftest evaluator tests.
Generated by retro agent from #3295
What happened
PR #3295, a Renovate security bump of
in-toto-golang, was blocked for 106 days partly due to flaky CI failures. TheChecksworkflow failed on 6 of 9 runs. Two failures (workflow runs 30586521570 on Jul 30 and 30907557496 on Aug 4) were caused byTestFilteringWithMixedRulesininternal/evaluator/conftest_evaluator_unit_core_test.gopanicking with a test timeout during OPA capabilities JSON parsing. The test performs I/O-heavy operations (creating temp directories, building tar.gz archives, compiling Rego policies viaNewConftestEvaluatorWithNamespace()) that can exceed the timeout under CI load. The same test passed on subsequent runs with no code changes, confirming it is a flake rather than a real failure.What could go better
The unit test timeout (set in the Makefile at line 110) is tight relative to the work
TestFilteringWithMixedRulesperforms. Under CI load, OPA policy compilation occasionally takes longer than the budget allows, causing a panic that fails the entire test suite. This is a test-fixture resilience problem — the production code path (conftest evaluator initialization) is inherently variable in execution time depending on system load, and the test harness does not account for that variability. Confidence is high: the same test failed and then passed on identical commits with no intervening code change, and the failure is in a known-expensive code path (OPA compilation).Proposed change
In the Makefile, increase the
-timeoutflag for unit tests from its current value to at least30sto provide headroom for I/O-heavy tests likeTestFilteringWithMixedRulesthat compile OPA policies. Alternatively, if the tight timeout is intentional to catch slow tests, refactorTestFilteringWithMixedRulesto extract the archive creation and policy loading into aTestMainor shared fixture so that expensive setup runs once rather than per-test-case, reducing the time pressure on the overall test budget.Validation criteria
After the change,
TestFilteringWithMixedRulesshould not produce timeout panics. Validate by runningmake test-unitat least 10 consecutive times (or checking the next 10 CI runs on therelease-v0.7branch) and confirming zero timeout-related failures in the conftest evaluator tests.Generated by retro agent from #3295