validate: a skipped check is an environment gap, not a failed entry - #1305
Merged
Conversation
#1304 made a run with no assertions fail, to stop h3-only entries showing a green check while nothing had been verified. That was right for the case it was aimed at and wrong for the one CI actually hits. The per-framework validate job runs on ubuntu-latest. Load-generator images are built by benchmark.sh on the self-hosted runner, so h2load-h3 does not exist there and never will under the current workflow. The h3 checks therefore skip, the run ends 0 passed / 0 failed, and #1304 turned that into a hard failure -- zix-http3 now fails CI for a missing image on the runner rather than for anything about the entry: SKIP [h3]: no h2load-h3 image ... === Results: 0 passed, 0 failed, 1 skipped === FAIL: no checks ran for zix-http3 ... Split the two causes. If coverage exists and its tool was absent, say plainly that the entry is UNVALIDATED and pass -- nothing was proven, but nothing is wrong with the entry either. Fail only when validate.sh genuinely has no checks for anything the entry subscribes to. Verified both reachable paths against zix-http3: h2load-h3 present -> 2 passed, 0 failed, exit 0 h2load-h3 absent -> 0 passed, 0 failed, 1 skipped, WARNING, exit 0 Getting real h3 coverage on CI needs the image to exist there -- either built and cached in the workflow, or pulled from a registry. Neither is in this change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
My #1304 broke CI. Reported on the #1303 run — zix-http3 job:
What I got wrong
#1304 made a run with no assertions fail, so an h3-only entry could not show a green check while nothing had been verified. That reasoning holds for genuine missing coverage. It does not hold for the case CI actually hits.
The per-framework validate job runs on
ubuntu-latest. Load-generator images are built bybenchmark.shon the self-hosted runner, soh2load-h3does not exist on the validate runner and never will under the current workflow. The h3 checks skip, the run ends 0/0, and #1304 turned that into a hard failure — zix-http3 fails for a missing image on the runner, not for anything about the entry.Fix
Split the two causes:
WARNING: … is UNVALIDATED, exit 0FAIL, exit 1Nothing is proven in the first case, but nothing is wrong with the entry either — so it says so plainly rather than failing it.
Verification
Both reachable paths, against
zix-http3:h2load-h3presenth2load-h3absent (CI's situation)Still open
Real h3 coverage on CI needs the image to exist there — built and cached in the workflow, or pulled from a registry. Neither is in this change; the h3 checks stay fully effective locally and on the self-hosted runner, where the image is present.
🤖 Generated with Claude Code