ci: validate every entry when the validator itself changes - #1285
Merged
Conversation
validate.yml triggers on scripts/validate.sh, then builds its framework list from changed frameworks/ paths -- so a pull request that only touches the validator matches the trigger and then validates nothing. That is what happened when the static staleness probe landed in #1267 and again when it was hardened in #1284: both changed what every entry is checked against, and neither ran a single entry. A validator change now covers every enabled entry (185: 177 on the matrix, 8 compose entries sequential on the self-hosted box). The workflow file itself joins the trigger paths and the same rule, since editing the workflow can change what validation means just as much as editing the script. Also adds workflow_dispatch, taking an entry name, a comma-separated list, or "all", so a full sweep can be run on demand without inventing a commit. Timeouts raised to match what a full sweep reaches: 60 minutes on the matrix, because entries that build from source on a four-core hosted runner (h2o, the LTO Rust entries, Swift) do not finish in 30; and 300 on the self-hosted job, which runs all eight compose entries in one sequential loop.
This was referenced Aug 24, 2026
Merged
MDA2AV
added a commit
that referenced
this pull request
Aug 24, 2026
From the full validation sweep in #1285. Eleven entries fail `docker build` and cannot be validated or benchmarked at all, so they are disabled and their published results removed. araara, araara-standard opam: no package matching hcs.0.16.1 fasthttp, pico, veb, the V compiler no longer builds from source: vanilla-epoll, "C function C.open was already declared with a vanilla-h2c, different signature". All seven pin different V vanilla-io_uring, commits and fail identically, so it is the vanilla-ws toolchain rather than any one pin. symfony-spawn-franken composer exits 127, not present in its base image workerman-websocket composer cannot unpack: "the zip extension and unzip/7z commands are both missing" Every one was reproduced locally, not taken from CI alone. laravel and hyperf are deliberately not here. Both failed the sweep on composer errors and both build clean on the benchmark box -- laravel first try, hyperf earlier the same day at 53/0 -- and their three failures carried three different signatures (CI exit 100, local exit 139, local heap corruption inside composer). That is the flaky toolchain, not the entries, and disabling them would have dropped 33 result keys of good data.
MDA2AV
added a commit
that referenced
this pull request
Aug 24, 2026
…1289) From the full validation sweep in #1285. Nineteen entries fail the exhaustive fragmentation check, which splits nine request shapes at every byte offset and requires the server to answer correctly regardless of where the request is cut across TCP segments. Twelve fail 100% of offsets in the shapes they fail -- fishcake, helidon-production, helidon-tuned, ioxide, libreactorng, may-minihttp, minima, nestjs, ringzero, rust-epoll, vinyl-cache, web-framework-csharp -- and slim and dart-io fail 91% and 88%. Those cannot parse a request that arrives in more than one read. The remaining five fail a narrow slice rather than the whole shape: hical 9%, effinitive 3%, and bjoern, dart-zig and blackbull about 1% (blackbull is a single offset out of 140). They are included because the instruction was every entry that fails the check, but they are a different kind of defect from the twelve above -- an edge case at one split point rather than a parser that cannot reassemble at all -- and are the ones to revisit first if this looks too broad.
MDA2AV
added a commit
that referenced
this pull request
Aug 24, 2026
… entries that fail them (#1290) Two changes from the sweep in #1285. The profiles stop feeding the framework composite. static and static-tls go to scored=False, which is the shape Pipelined already has: still measured, still shown, faded on the board, out of the sum. infraScored stays True -- serving files fast from a tuned cache is what a proxy is for, and that league is unaffected. engineScored was already False. CATALOG is the single source: the board reads it through data.js and the badge port reads it through _scored_for(), so the two scoring implementations cannot disagree about this and check_badge_parity.js stays green. Twelve entries fail validation on static and on nothing else, and are flagship, emerging or experimental. They lose the two profiles from their tests array and the 69 result keys those profiles produced: django, fletch, genhttp-11, genhttp-11-kestrel, genhttp-kestrel, mq-bridge, mq-bridge-py, pyronova, rage, rails, true-async-server, userver Nothing else is touched: every other profile keeps its results, and static-h2 and static-h3 are left alone -- the failures were on the H1 pair. The entries stay enabled, since static was their only failure.
MDA2AV
added a commit
that referenced
this pull request
Aug 24, 2026
) The last group from the sweep in #1285: twelve entries that fail checks other than the static probe, so neither #1288 (build), #1289 (fragmentation) nor #1290 (static reference-only) covers them. deno-websocket, fleck reject non-upgrade GET /ws node-websocket text echo, multi-message, dies during the run dogrider binary echo sark upload chunked, upload truncated body, StreamSum over h2+TLS swerver HTTP/2 negotiation and h2c prior-knowledge wtx-grpc, wtx-grpc-tls GetSum over h2c / h2+TLS iris, typev upload chunked, plus static zio-http, zix upload chunked Each fails something the entry claims to implement, so the numbers it publishes for those profiles do not describe a working server. zio-http is the only flagship here and the only one whose failure is a single profile -- upload chunked. It is included because the instruction was the remaining failures, but it is the one worth a second look before this merges: it loses 23 result keys over one check.
Contributor
|
I am coming from #1289 and am surprised. Could you please help me understand which validation is failing and where I can find more information about it? |
Owner
Author
Of course, you can find details here https://www.http-arena.com/docs/test-profiles/h1/isolated/baseline/validation/ last point "Exhaustive TCP fragmentation" this is a added validation step taken from https://github.com/uNetworking/uWebSockets/blob/master/tests/fragment_test.ts modified to also include query parameters. Run details on Vinyl Cache here https://github.com/MDA2AV/HttpArena/actions/runs/32719610781/job/97408097735 from 1904 onwards |
Contributor
Excellent, thank you for your help! |
Owner
Author
Sorry for the trouble, we are still not 100% sure if some validation failures are false positives hence the "temporary", as we keep adding more validation points. Would be great if you could look into it. |
Contributor
This is a great attitude, but in the case of Vinyl Cache I can already confirm that it is real and a regression: https://code.vinyl-cache.org/vinyl-cache/vinyl-cache/issues/4575 I love your project already. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opens a full validation sweep over every enabled entry, and fixes the gap that made one impossible.
The gap
validate.ymltriggers onscripts/validate.sh, then builds its framework list from changedframeworks/**paths:A PR that only touches the validator matches the trigger and then validates nothing — the list is empty and both jobs are skipped by
if: ... != '[]'.That is not hypothetical. It happened when the static staleness probe landed in #1267, and again when it was hardened in #1284. Both changed what every entry is checked against; neither ran a single entry.
The change
scripts/validate.sh,scripts/validate-ws.py, or this workflow) now covers every enabled entry.workflow_dispatchtaking an entry name, a comma-separated list, orall, so a sweep can be run on demand without inventing a commit.Scale
185 enabled entries — 177 on the matrix (GitHub's cap is 256) and 8 compose entries running sequentially on the self-hosted box.
Timeouts were sized for the handful of entries a normal PR touches, so both are raised:
What to expect from this run
This is the first sweep against the hardened static probe, so it is also the blast-radius measurement for #1284. From a 12-entry local sample, no entry that passed the old probe fails the new one — every failure below already fails on
main:Two things that sample already surfaced, both independent of the hardening:
genhttp-11andgenhttp-11-kestrelserve from GenHTTP'sAssets.From, which never revalidates — 35s after the file changed on disk they still serve the original bytes. Their siblinggenhttp-11-ioxidepasses becauseIoxideFiles.Fromre-stats per request. Same framework, different handler.ioxideis stale on both the identity and variant paths out to 35s, despiteProgram.cs:100describing the plain path as read-per-request.Expect some failures in this sweep to be infrastructure rather than entries — a four-core hosted runner is not the benchmark box, and a few builds are slow or flaky there. Worth reading the first run as a survey, not a verdict.
🤖 Generated with Claude Code