Skip to content

validate: close the h3 and grpc-tls coverage gaps - #1304

Merged
MDA2AV merged 1 commit into
mainfrom
harden/validate-coverage-gaps
Aug 24, 2026
Merged

validate: close the h3 and grpc-tls coverage gaps#1304
MDA2AV merged 1 commit into
mainfrom
harden/validate-coverage-gaps

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Aug 24, 2026

Copy link
Copy Markdown
Owner

Found while working out why zix publishes nothing. Three separate holes, all in the same area.

1. The h3 profiles had no checks at all

baseline-h3 and static-h3 ran zero assertions. An entry subscribing to h3 alone finished:

=== Results: 0 passed, 0 failed ===

and exited 0 — indistinguishable on CI from a clean pass. zix-http3 has been passing that way. The docs said so out loud, and pointed at oha, which has been retired since the switch to h2load-h3.

2. QUIC had no datagram path

-p publishes tcp unless told otherwise, so an h3 entry on the bridge network got tcp/8443 and nothing on udp. No check could have reached a QUIC listener even if one had existed. Now -p $H2PORT:8443/udp is added for h3 subscribers.

3. Readiness assumed every h3 server also speaks h2

The comment claimed "H/3 servers still advertise h2 on the same TLS listener via ALPN, so this single fallback covers both cases". sark-h3 does not — it serves QUIC on udp/8443 and nothing on tcp/8443. It has failed Server did not start within 30s on every run it has ever had, while being perfectly healthy: h2load reaches it over QUIC and gets 4 2xx immediately.

Plus: unary-grpc-tls had no TLS probe

It terminates a real TLS 1.3 handshake per connection exactly like baseline-h2, but got neither the posture nor the quality probe. That is how zix-grpc passed 4/4 while serving a self-generated Ed25519 certificate on :8443 (fixed in #1303). It now gets the same two probes every other TLS profile gets.

And: a run with no assertions now fails

PASS == 0 && FAIL == 0 means unvalidated, not validated-clean. It now says so and exits 1.

curl here has no QUIC support and openssl cannot speak it either, so both the readiness fallback and the new checks shell out to the same ngtcp2-built h2load the benchmark already uses. When that image is absent the result is an explicit SKIP, counted separately from passes — a skip must never read as coverage.

Verification

entry before after
zix-http3 silent pass, 0 checks, exit 0 2 pass / 0 fail
sark-h3 Server did not start within 30s 2 pass / 0 fail
cardigan-grpc-tls 2 pass 7 pass / 0 fail (5 new)
zix, zix-http2, zix-ws, zix-grpc unchanged, no regression

Blast radius

15 enabled entries subscribe to an h3 profile, 11 to unary-grpc-tls. The four above are the ones actually exercised here; the rest surface on the next full sweep. An entry that claims a profile it does not really serve is meant to fail this.

Also adds VALIDATE_SKIP_BUILD=1 to reuse an existing image. validate.sh does docker build --no-cache inside the timeout window, so a slow-building entry reports FAIL: Validation timed out followed by no such container: it was never created — which reads as a server failure when it is really build duration. I hit exactly that on zix: one run spent the whole 800s window inside zig build and reported a validation timeout with no such container: it was never created. Later no-cache builds of the same entry finished well inside 180s, so this is an intermittent-slow-build hazard (toolchain/dependency fetch, or this box misbehaving) rather than a fixed property of the entry — the point is only that a slow build is reported as a server failure.

🤖 Generated with Claude Code

Three holes, all found while working out why zix publishes nothing.

1. The h3 profiles had no checks at all. baseline-h3 and static-h3 ran zero
   assertions, so an entry subscribing to h3 alone finished "0 passed, 0
   failed" and exited 0 -- indistinguishable on CI from a clean pass.
   zix-http3 has been passing that way.

2. -p publishes tcp unless told otherwise, so an h3 entry on the bridge
   network got tcp/8443 and no datagram path whatsoever. Nothing could reach
   a QUIC listener even if a check had existed.

3. The readiness probe assumed every h3 server also advertises h2 on the same
   TLS listener. sark-h3 does not: it serves QUIC on udp/8443 and nothing on
   tcp/8443, so it failed "Server did not start within 30s" on every run it
   has ever had, while being perfectly healthy.

curl here has no QUIC support and openssl cannot speak it either, so both the
readiness fallback and the new checks use the same ngtcp2-built h2load the
benchmark already uses. When that image is absent the result is an explicit
SKIP counted separately from passes -- a skip must never read as coverage.

unary-grpc-tls also had no TLS posture or quality probe, though it terminates
a real TLS 1.3 handshake per connection exactly like baseline-h2. That is how
zix-grpc passed 4/4 while serving a self-generated Ed25519 certificate on
:8443. It now gets the same two probes the other TLS profiles get.

Finally, a run that makes no assertions at all now fails with a message
saying so, instead of exiting 0.

Verified:

  zix-http3   silent pass, 0 checks      -> 2 pass / 0 fail
  sark-h3     never started within 30s   -> 2 pass / 0 fail
  cardigan-grpc-tls                      -> 7 pass / 0 fail (5 are new)
  zix, zix-http2, zix-ws, zix-grpc       -> unchanged, no regression

Blast radius: 15 enabled entries subscribe to an h3 profile and 11 to
unary-grpc-tls. The four above are the ones actually exercised here; the rest
will show up on the next full sweep. An entry that claims a profile it does
not really serve is meant to fail this -- that is the point of the checks.
@MDA2AV
MDA2AV merged commit f90981e into main Aug 24, 2026
124 of 136 checks passed
MDA2AV added a commit that referenced this pull request Aug 24, 2026
…1305)

#1304 made a run with no assertions fail, to stop h3-only entries showing a
green check while nothing had been verified. That was right for the case it
was aimed at and wrong for the one CI actually hits.

The per-framework validate job runs on ubuntu-latest. Load-generator images
are built by benchmark.sh on the self-hosted runner, so h2load-h3 does not
exist there and never will under the current workflow. The h3 checks therefore
skip, the run ends 0 passed / 0 failed, and #1304 turned that into a hard
failure -- zix-http3 now fails CI for a missing image on the runner rather
than for anything about the entry:

  SKIP [h3]: no h2load-h3 image ...
  === Results: 0 passed, 0 failed, 1 skipped ===
  FAIL: no checks ran for zix-http3 ...

Split the two causes. If coverage exists and its tool was absent, say plainly
that the entry is UNVALIDATED and pass -- nothing was proven, but nothing is
wrong with the entry either. Fail only when validate.sh genuinely has no
checks for anything the entry subscribes to.

Verified both reachable paths against zix-http3:

  h2load-h3 present  -> 2 passed, 0 failed, exit 0
  h2load-h3 absent   -> 0 passed, 0 failed, 1 skipped, WARNING, exit 0

Getting real h3 coverage on CI needs the image to exist there -- either built
and cached in the workflow, or pulled from a registry. Neither is in this
change.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant