validate: close the h3 and grpc-tls coverage gaps - #1304
Merged
Conversation
Three holes, all found while working out why zix publishes nothing. 1. The h3 profiles had no checks at all. baseline-h3 and static-h3 ran zero assertions, so an entry subscribing to h3 alone finished "0 passed, 0 failed" and exited 0 -- indistinguishable on CI from a clean pass. zix-http3 has been passing that way. 2. -p publishes tcp unless told otherwise, so an h3 entry on the bridge network got tcp/8443 and no datagram path whatsoever. Nothing could reach a QUIC listener even if a check had existed. 3. The readiness probe assumed every h3 server also advertises h2 on the same TLS listener. sark-h3 does not: it serves QUIC on udp/8443 and nothing on tcp/8443, so it failed "Server did not start within 30s" on every run it has ever had, while being perfectly healthy. curl here has no QUIC support and openssl cannot speak it either, so both the readiness fallback and the new checks use the same ngtcp2-built h2load the benchmark already uses. When that image is absent the result is an explicit SKIP counted separately from passes -- a skip must never read as coverage. unary-grpc-tls also had no TLS posture or quality probe, though it terminates a real TLS 1.3 handshake per connection exactly like baseline-h2. That is how zix-grpc passed 4/4 while serving a self-generated Ed25519 certificate on :8443. It now gets the same two probes the other TLS profiles get. Finally, a run that makes no assertions at all now fails with a message saying so, instead of exiting 0. Verified: zix-http3 silent pass, 0 checks -> 2 pass / 0 fail sark-h3 never started within 30s -> 2 pass / 0 fail cardigan-grpc-tls -> 7 pass / 0 fail (5 are new) zix, zix-http2, zix-ws, zix-grpc -> unchanged, no regression Blast radius: 15 enabled entries subscribe to an h3 profile and 11 to unary-grpc-tls. The four above are the ones actually exercised here; the rest will show up on the next full sweep. An entry that claims a profile it does not really serve is meant to fail this -- that is the point of the checks.
MDA2AV
added a commit
that referenced
this pull request
Aug 24, 2026
…1305) #1304 made a run with no assertions fail, to stop h3-only entries showing a green check while nothing had been verified. That was right for the case it was aimed at and wrong for the one CI actually hits. The per-framework validate job runs on ubuntu-latest. Load-generator images are built by benchmark.sh on the self-hosted runner, so h2load-h3 does not exist there and never will under the current workflow. The h3 checks therefore skip, the run ends 0 passed / 0 failed, and #1304 turned that into a hard failure -- zix-http3 now fails CI for a missing image on the runner rather than for anything about the entry: SKIP [h3]: no h2load-h3 image ... === Results: 0 passed, 0 failed, 1 skipped === FAIL: no checks ran for zix-http3 ... Split the two causes. If coverage exists and its tool was absent, say plainly that the entry is UNVALIDATED and pass -- nothing was proven, but nothing is wrong with the entry either. Fail only when validate.sh genuinely has no checks for anything the entry subscribes to. Verified both reachable paths against zix-http3: h2load-h3 present -> 2 passed, 0 failed, exit 0 h2load-h3 absent -> 0 passed, 0 failed, 1 skipped, WARNING, exit 0 Getting real h3 coverage on CI needs the image to exist there -- either built and cached in the workflow, or pulled from a registry. Neither is in this change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while working out why zix publishes nothing. Three separate holes, all in the same area.
1. The h3 profiles had no checks at all
baseline-h3andstatic-h3ran zero assertions. An entry subscribing to h3 alone finished:and exited 0 — indistinguishable on CI from a clean pass.
zix-http3has been passing that way. The docs said so out loud, and pointed atoha, which has been retired since the switch toh2load-h3.2. QUIC had no datagram path
-ppublishes tcp unless told otherwise, so an h3 entry on the bridge network gottcp/8443and nothing on udp. No check could have reached a QUIC listener even if one had existed. Now-p $H2PORT:8443/udpis added for h3 subscribers.3. Readiness assumed every h3 server also speaks h2
The comment claimed "H/3 servers still advertise h2 on the same TLS listener via ALPN, so this single fallback covers both cases".
sark-h3does not — it serves QUIC onudp/8443and nothing ontcp/8443. It has failedServer did not start within 30son every run it has ever had, while being perfectly healthy: h2load reaches it over QUIC and gets4 2xximmediately.Plus: unary-grpc-tls had no TLS probe
It terminates a real TLS 1.3 handshake per connection exactly like
baseline-h2, but got neither the posture nor the quality probe. That is howzix-grpcpassed 4/4 while serving a self-generated Ed25519 certificate on :8443 (fixed in #1303). It now gets the same two probes every other TLS profile gets.And: a run with no assertions now fails
PASS == 0 && FAIL == 0means unvalidated, not validated-clean. It now says so and exits 1.curlhere has no QUIC support and openssl cannot speak it either, so both the readiness fallback and the new checks shell out to the same ngtcp2-builth2loadthe benchmark already uses. When that image is absent the result is an explicitSKIP, counted separately from passes — a skip must never read as coverage.Verification
zix-http3sark-h3Server did not start within 30scardigan-grpc-tlszix,zix-http2,zix-ws,zix-grpcBlast radius
15 enabled entries subscribe to an h3 profile, 11 to
unary-grpc-tls. The four above are the ones actually exercised here; the rest surface on the next full sweep. An entry that claims a profile it does not really serve is meant to fail this.Also adds
VALIDATE_SKIP_BUILD=1to reuse an existing image.validate.shdoesdocker build --no-cacheinside the timeout window, so a slow-building entry reportsFAIL: Validation timed outfollowed byno such container: it was never created— which reads as a server failure when it is really build duration. I hit exactly that on zix: one run spent the whole 800s window insidezig buildand reported a validation timeout withno such container: it was never created. Later no-cache builds of the same entry finished well inside 180s, so this is an intermittent-slow-build hazard (toolchain/dependency fetch, or this box misbehaving) rather than a fixed property of the entry — the point is only that a slow build is reported as a server failure.🤖 Generated with Claude Code