disable entries that fail to build, and drop their results - #1288
Merged
Conversation
From the full validation sweep in #1285. Eleven entries fail `docker build` and cannot be validated or benchmarked at all, so they are disabled and their published results removed. araara, araara-standard opam: no package matching hcs.0.16.1 fasthttp, pico, veb, the V compiler no longer builds from source: vanilla-epoll, "C function C.open was already declared with a vanilla-h2c, different signature". All seven pin different V vanilla-io_uring, commits and fail identically, so it is the vanilla-ws toolchain rather than any one pin. symfony-spawn-franken composer exits 127, not present in its base image workerman-websocket composer cannot unpack: "the zip extension and unzip/7z commands are both missing" Every one was reproduced locally, not taken from CI alone. laravel and hyperf are deliberately not here. Both failed the sweep on composer errors and both build clean on the benchmark box -- laravel first try, hyperf earlier the same day at 53/0 -- and their three failures carried three different signatures (CI exit 100, local exit 139, local heap corruption inside composer). That is the flaky toolchain, not the entries, and disabling them would have dropped 33 result keys of good data.
Contributor
|
👋 Heads up! This PR modifies the following frameworks:
|
MDA2AV
added a commit
that referenced
this pull request
Aug 24, 2026
) The last group from the sweep in #1285: twelve entries that fail checks other than the static probe, so neither #1288 (build), #1289 (fragmentation) nor #1290 (static reference-only) covers them. deno-websocket, fleck reject non-upgrade GET /ws node-websocket text echo, multi-message, dies during the run dogrider binary echo sark upload chunked, upload truncated body, StreamSum over h2+TLS swerver HTTP/2 negotiation and h2c prior-knowledge wtx-grpc, wtx-grpc-tls GetSum over h2c / h2+TLS iris, typev upload chunked, plus static zio-http, zix upload chunked Each fails something the entry claims to implement, so the numbers it publishes for those profiles do not describe a working server. zio-http is the only flagship here and the only one whose failure is a single profile -- upload chunked. It is included because the instruction was the remaining failures, but it is the one worth a second look before this merges: it loses 23 result keys over one check.
MDA2AV
added a commit
that referenced
this pull request
Aug 24, 2026
all_entries() ends on a jq that exits non-zero for a disabled entry, so the function inherited that status and `changed=$(all_entries)` aborted the whole step under `bash -e` -- before a single line was echoed, which is why the failure looked like nothing at all had run. Whether it triggers depends on which entry sorts last and whether it happens to be enabled, so it passed on the run that introduced it and failed on the next one, after #1288, #1289 and #1291 disabled a batch of entries. An explicit `return 0` on the function.
MDA2AV
added a commit
that referenced
this pull request
Aug 24, 2026
all_entries() ends on a jq that exits non-zero for a disabled entry, so the function inherited that status and `changed=$(all_entries)` aborted the whole step under `bash -e` -- before a single line was echoed, which is why the failure looked like nothing at all had run. Whether it triggers depends on which entry sorts last and whether it happens to be enabled, so it passed on the run that introduced it and failed on the next one, after #1288, #1289 and #1291 disabled a batch of entries. An explicit `return 0` on the function.
MDA2AV
added a commit
that referenced
this pull request
Aug 24, 2026
* validate: check TLS quality and TLS fairness, not just ALPN
Until now the only thing checked about TLS was which protocol ALPN settled
on. Two questions went unasked.
What did this connection negotiate -- the fairness half, tls_posture_probe:
* the certificate must be the one mounted at /certs. Every handshake costs
the server one signature with it, and on this box RSA-2048 signs 3,052/s
against 77,124/s for ECDSA P-256. An entry that quietly generates its own
EC certificate gets 25x cheaper handshakes and nothing would have
noticed. Verified against a synthetic entry serving its own EC pair.
* TLS 1.3, against a client offering it. The 1.2 handshake costs an extra
round trip.
* one of the three TLS 1.3 AEAD suites, which rules out NULL, anon, export
and RC4 making "TLS" free.
* ALPN must never name a protocol the client did not offer. Selecting
nothing is fine and common -- bun does it, and the client falls back.
The cipher is reported rather than required. The field is already split:
some entries pick AES-128-GCM, some AES-256-GCM, and those measure ~17%
apart on bulk encryption at static-file block sizes (11.1 vs 9.5 GB/s at
16 KB). Not every framework exposes cipher preference, so this is on the
record instead of invisible.
What will the server negotiate at all -- the quality half, tls_quality_scan.
A server can hand a modern client TLS 1.3 and still accept TLS 1.0 or a NULL
cipher from anything that asks. Fails on SSLv2, SSLv3, TLS 1.0, TLS 1.1 and
on NULL, anonymous, export, 64-bit/DES/RC4/MD5 and 3DES cipher lists; notes
obsolete CBC and AEAD-without-forward-secrecy, which are weak but shipped as
defaults by much of the field.
That half is testssl.sh rather than a hand-rolled sweep: it already knows
every suite, it is the reference tool, and -p -s answers in ~5s. Its full run
also produces an SSL Labs grade, but costs ~48s per port and caps every entry
at B on the self-signed certificate the harness mounts, so the graded run is
left for audits. Skippable with HTTPARENA_SKIP_TLS_SCAN=1, and it skips
itself rather than failing an entry when the scanner image is unavailable.
validate: bun 70 -> 79 passed 0 failed, caddy 71/0. bun draws the note --
it accepts CBC and non-FS suites; caddy accepts neither.
* docs: document the TLS checks on the json-tls validation page
The checks added alongside this were not written down anywhere. The json-tls
page now carries them, since that is the profile whose whole point is TLS:
* the certificate must be the one mounted at /certs, with the numbers that
make it matter -- RSA-2048 signs 3,052/s against ECDSA P-256's 77,124/s,
and every handshake is one signature, so a self-generated EC certificate
is a 25x discount on that step
* TLS 1.3, because the 1.2 handshake costs an extra round trip
* one of the three TLS 1.3 AEAD suites, and why the choice among them is
reported rather than required: the server picks, the field is split, and
AES-128 against AES-256 is ~17% on bulk encryption
* ALPN may select nothing, but never something the client did not offer
* the testssl.sh scan: what fails, what is only noted, how to skip it, and
why the graded run is left for audits rather than the gate
static-tls, baseline-h2 and static-h2 get a short pointer instead of a copy
-- the checks run there too, and one description is easier to keep true than
four.
* validate: ask about obsolete protocols with openssl rather than a scanner
Same question, asked directly: offer SSLv3, TLS 1.0 and TLS 1.1, and the
NULL, anonymous, export, RC4, DES and 3DES cipher families, and fail when a
handshake actually completes. An alert, a reset or a timeout is a refusal.
OpenSSL will not make those offers at its default security level, so the
probes carry @SECLEVEL=0; a protocol this openssl cannot offer at all is
reported as unprobed rather than counted as refused.
testssl.sh agrees with these results and is what found the first real failure
here -- humming-bird completes a TLS 1.0 handshake with
ECDHE-RSA-AES256-SHA. What decides it for the gate is cost: ~70ms for the
whole set against ~5s, and nothing outside the base image to pull or hang.
testssl.sh stays the better tool for an audit, where its wider suite coverage
and SSL Labs style grade earn the time, and the docs say so.
Verified on a clean host: humming-bird accepts tls1 and tls1_1, bun and caddy
accept neither.
* board: a TLS shield beside entries whose TLS was checked and clean
A small green shield after the framework name, on the table rows and in the
detail modal, for an entry that passed the TLS checks: serves the certificate
the harness mounts, negotiates TLS 1.3 with an AEAD cipher, and refuses
SSLv3, TLS 1.0/1.1 and the NULL, anonymous, export, RC4, DES and 3DES cipher
families.
The verdict is earned rather than declared. validate.sh writes
site/data/tls/<entry>.json when it actually runs those probes, the generator
folds it into meta, and the board reads meta[fw].tls. It is deliberately not
a meta.json field: that would let an entry claim the shield without being
checked, which is the opposite of the point.
Three states, and only one of them shows anything. "pass" gets the shield;
"fail" and absent both get nothing, because most entries have no TLS profile
at all and a red mark against them would say something untrue.
Seeded with the eight entries verified on a clean host this session -- bun,
caddy, elysia, genhttp-11-ioxide, h2o-mruby, hono-bun and slimeweb pass,
humming-bird fails on accepting TLS 1.0 and 1.1. Every other entry stays
unmarked until it is checked.
Hidden below 640px, where the name column is tightest.
check_badge_parity.js: 533 ranks match.
* ci: keep the TLS verdicts a validation run produces
validate.sh writes site/data/tls/<entry>.json when it runs the TLS probes,
and the board turns that into the shield -- but nothing collected the files,
so a full sweep computed 177 verdicts and threw all of them away.
One artifact per entry rather than a commit from inside the job: 177 matrix
jobs writing one directory would race, and a validation run committing to the
PR it is validating is a loop nobody wants. Collected after the run and
committed deliberately.
Uploaded with if: always(), because an entry that fails its TLS checks is
exactly the verdict worth keeping.
* ci: stop the detect step aborting on a disabled entry
all_entries() ends on a jq that exits non-zero for a disabled entry, so the
function inherited that status and `changed=$(all_entries)` aborted the whole
step under `bash -e` -- before a single line was echoed, which is why the
failure looked like nothing at all had run.
Whether it triggers depends on which entry sorts last and whether it happens
to be enabled, so it passed on the run that introduced it and failed on the
next one, after #1288, #1289 and #1291 disabled a batch of entries.
An explicit `return 0` on the function.
* validate: always mount the static directory, and raise the watchdog to 800s
Two things the full sweep on this PR exposed.
/data/static was mounted only when an entry subscribed to a static profile.
benchmark.sh mounts it unconditionally (scripts/lib/framework.sh), and the
entries that read the directory at startup cannot boot without it: rage and
rails copy it with FileUtils.cp_r, userver builds an fs-cache from it. So
when #1290 dropped the static profiles from those three, "does not serve
static" quietly became "does not start" -- all three failed the sweep with
the container dead before the first check.
It is a read-only bind of a small directory and there is nothing to be gained
by leaving it out, so validate.sh now matches benchmark.sh and always mounts
it. rage goes from a dead container to 44 passed, 0 failed.
The watchdog goes 300s -> 800s. The entries with the widest profile sets --
humming-bird and the three web-framework-* entries -- were killed mid-run
rather than failing anything, so their results said nothing at all.
* board: drop the TLS shield; the tls_check badge replaces it
Two badges for one subject was one too many, and the green shield was the
weaker of the two: it marked TLS basics an entry got right by default, while
the gold badge marks a hardening bar an entry opted into and wrote code for.
Removes the renderer, its CSS, the three render sites, the meta field the
board read and the loader that filled it. validate.sh still writes
site/data/tls/<entry>.json -- the verdict is what the opt-in badge is built
on -- but the files themselves are no longer committed, since nothing here
consumes them any more.
It also repairs a real break. The modal insertion had produced
'</h3>'+tlsShield(fw)<span class="badge b-'+t+'"
-- the +' was lost, which is a syntax error in the board's only script block,
so the page would not have rendered at all. The earlier check evaluated
data.js and the shield function in isolation and never parsed index.html,
which is exactly the gap that let it through. Both are parsed now:
all script blocks parse
badge parity ok — 533 ranks match
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
From the full validation sweep in #1285. Eleven entries fail
docker build, so they cannot be validated or benchmarked at all — disabled, and their published results removed.araara,araara-standardNo package matching hcs.0.16.1 foundfasthttp,pico,veb,vanilla-epoll,vanilla-h2c,vanilla-io_uring,vanilla-wsC function C.open was already declared with a different signaturesymfony-spawn-frankencomposerexits 127 — not present in its base imageworkerman-websocket9 result files deleted (
araaraandaraara-standardhad none).Every failure was reproduced locally, not taken from CI alone.
Two exclusions
laravelandhyperffailed the sweep but are not disabled here. Both build clean on the benchmark box — laravel first try, hyperf earlier the same day at 53/0 — and their failures carried three different signatures across attempts (CI exit 100, local exit 139, local heap corruption inside composer). That is the flaky toolchain, not the entries. Disabling them would have dropped 33 result keys of good data.Worth a separate look
The seven V entries pin different V commits (
84a76d791,cef604a26,02015a7ce) and fail identically, so this is one toolchain problem rather than seven broken entries. They also useFROM debian:stable-slim, a moving tag that has rolled to trixie. Pinning back tobookworm-slimalone did not fix it, so it needs someone who knows V — but if it is fixed, all seven come back together.