static, static-tls: reference-only for frameworks; drop them from the entries that fail them - #1290
Merged
Merged
Conversation
… entries that fail them Two changes from the sweep in #1285. The profiles stop feeding the framework composite. static and static-tls go to scored=False, which is the shape Pipelined already has: still measured, still shown, faded on the board, out of the sum. infraScored stays True -- serving files fast from a tuned cache is what a proxy is for, and that league is unaffected. engineScored was already False. CATALOG is the single source: the board reads it through data.js and the badge port reads it through _scored_for(), so the two scoring implementations cannot disagree about this and check_badge_parity.js stays green. Twelve entries fail validation on static and on nothing else, and are flagship, emerging or experimental. They lose the two profiles from their tests array and the 69 result keys those profiles produced: django, fletch, genhttp-11, genhttp-11-kestrel, genhttp-kestrel, mq-bridge, mq-bridge-py, pyronova, rage, rails, true-async-server, userver Nothing else is touched: every other profile keeps its results, and static-h2 and static-h3 are left alone -- the failures were on the H1 pair. The entries stay enabled, since static was their only failure.
Contributor
|
👋 Heads up! This PR modifies the following frameworks:
|
MDA2AV
added a commit
that referenced
this pull request
Aug 24, 2026
) The last group from the sweep in #1285: twelve entries that fail checks other than the static probe, so neither #1288 (build), #1289 (fragmentation) nor #1290 (static reference-only) covers them. deno-websocket, fleck reject non-upgrade GET /ws node-websocket text echo, multi-message, dies during the run dogrider binary echo sark upload chunked, upload truncated body, StreamSum over h2+TLS swerver HTTP/2 negotiation and h2c prior-knowledge wtx-grpc, wtx-grpc-tls GetSum over h2c / h2+TLS iris, typev upload chunked, plus static zio-http, zix upload chunked Each fails something the entry claims to implement, so the numbers it publishes for those profiles do not describe a working server. zio-http is the only flagship here and the only one whose failure is a single profile -- upload chunked. It is included because the instruction was the remaining failures, but it is the one worth a second look before this merges: it loses 23 result keys over one check.
MDA2AV
added a commit
that referenced
this pull request
Aug 24, 2026
…o 800s Two things the full sweep on this PR exposed. /data/static was mounted only when an entry subscribed to a static profile. benchmark.sh mounts it unconditionally (scripts/lib/framework.sh), and the entries that read the directory at startup cannot boot without it: rage and rails copy it with FileUtils.cp_r, userver builds an fs-cache from it. So when #1290 dropped the static profiles from those three, "does not serve static" quietly became "does not start" -- all three failed the sweep with the container dead before the first check. It is a read-only bind of a small directory and there is nothing to be gained by leaving it out, so validate.sh now matches benchmark.sh and always mounts it. rage goes from a dead container to 44 passed, 0 failed. The watchdog goes 300s -> 800s. The entries with the widest profile sets -- humming-bird and the three web-framework-* entries -- were killed mid-run rather than failing anything, so their results said nothing at all.
This was referenced Aug 24, 2026
MDA2AV
added a commit
that referenced
this pull request
Aug 24, 2026
…1294) * drop the static profiles from the four entries that fail only static From the sweep on #1292. These four fail validation on the static staleness probe and on nothing else, so the entries stay enabled and keep every other profile they publish; they lose static, static-tls and the 21 result keys those produced. fastpysgi-wsgi, uvicorn engine h2o, varnish infrastructure Same treatment #1290 gave the flagship, emerging and experimental entries in this position, applied to the tiers that were left out of it -- with one difference worth stating: for infrastructure this is a real result rather than a formality. h2o and varnish are proxies whose rule explicitly allows open_file_cache and mmap, and they were given a 30s window against the framework tiers' 2s. They still serve the old bytes after it. * also drop the static profiles from fastpysgi-asgi It fails static and only static against main -- 53 passed, 4 failed, every failure a staleness probe -- so it belongs here rather than in #1295, where it was first put. The sweep that classified it ran on the #1292 branch and showed a fragmentation failure that main does not reproduce.
MDA2AV
added a commit
that referenced
this pull request
Aug 24, 2026
…o 800s Two things the full sweep on this PR exposed. /data/static was mounted only when an entry subscribed to a static profile. benchmark.sh mounts it unconditionally (scripts/lib/framework.sh), and the entries that read the directory at startup cannot boot without it: rage and rails copy it with FileUtils.cp_r, userver builds an fs-cache from it. So when #1290 dropped the static profiles from those three, "does not serve static" quietly became "does not start" -- all three failed the sweep with the container dead before the first check. It is a read-only bind of a small directory and there is nothing to be gained by leaving it out, so validate.sh now matches benchmark.sh and always mounts it. rage goes from a dead container to 44 passed, 0 failed. The watchdog goes 300s -> 800s. The entries with the widest profile sets -- humming-bird and the three web-framework-* entries -- were killed mid-run rather than failing anything, so their results said nothing at all.
MDA2AV
added a commit
that referenced
this pull request
Aug 24, 2026
* validate: check TLS quality and TLS fairness, not just ALPN
Until now the only thing checked about TLS was which protocol ALPN settled
on. Two questions went unasked.
What did this connection negotiate -- the fairness half, tls_posture_probe:
* the certificate must be the one mounted at /certs. Every handshake costs
the server one signature with it, and on this box RSA-2048 signs 3,052/s
against 77,124/s for ECDSA P-256. An entry that quietly generates its own
EC certificate gets 25x cheaper handshakes and nothing would have
noticed. Verified against a synthetic entry serving its own EC pair.
* TLS 1.3, against a client offering it. The 1.2 handshake costs an extra
round trip.
* one of the three TLS 1.3 AEAD suites, which rules out NULL, anon, export
and RC4 making "TLS" free.
* ALPN must never name a protocol the client did not offer. Selecting
nothing is fine and common -- bun does it, and the client falls back.
The cipher is reported rather than required. The field is already split:
some entries pick AES-128-GCM, some AES-256-GCM, and those measure ~17%
apart on bulk encryption at static-file block sizes (11.1 vs 9.5 GB/s at
16 KB). Not every framework exposes cipher preference, so this is on the
record instead of invisible.
What will the server negotiate at all -- the quality half, tls_quality_scan.
A server can hand a modern client TLS 1.3 and still accept TLS 1.0 or a NULL
cipher from anything that asks. Fails on SSLv2, SSLv3, TLS 1.0, TLS 1.1 and
on NULL, anonymous, export, 64-bit/DES/RC4/MD5 and 3DES cipher lists; notes
obsolete CBC and AEAD-without-forward-secrecy, which are weak but shipped as
defaults by much of the field.
That half is testssl.sh rather than a hand-rolled sweep: it already knows
every suite, it is the reference tool, and -p -s answers in ~5s. Its full run
also produces an SSL Labs grade, but costs ~48s per port and caps every entry
at B on the self-signed certificate the harness mounts, so the graded run is
left for audits. Skippable with HTTPARENA_SKIP_TLS_SCAN=1, and it skips
itself rather than failing an entry when the scanner image is unavailable.
validate: bun 70 -> 79 passed 0 failed, caddy 71/0. bun draws the note --
it accepts CBC and non-FS suites; caddy accepts neither.
* docs: document the TLS checks on the json-tls validation page
The checks added alongside this were not written down anywhere. The json-tls
page now carries them, since that is the profile whose whole point is TLS:
* the certificate must be the one mounted at /certs, with the numbers that
make it matter -- RSA-2048 signs 3,052/s against ECDSA P-256's 77,124/s,
and every handshake is one signature, so a self-generated EC certificate
is a 25x discount on that step
* TLS 1.3, because the 1.2 handshake costs an extra round trip
* one of the three TLS 1.3 AEAD suites, and why the choice among them is
reported rather than required: the server picks, the field is split, and
AES-128 against AES-256 is ~17% on bulk encryption
* ALPN may select nothing, but never something the client did not offer
* the testssl.sh scan: what fails, what is only noted, how to skip it, and
why the graded run is left for audits rather than the gate
static-tls, baseline-h2 and static-h2 get a short pointer instead of a copy
-- the checks run there too, and one description is easier to keep true than
four.
* validate: ask about obsolete protocols with openssl rather than a scanner
Same question, asked directly: offer SSLv3, TLS 1.0 and TLS 1.1, and the
NULL, anonymous, export, RC4, DES and 3DES cipher families, and fail when a
handshake actually completes. An alert, a reset or a timeout is a refusal.
OpenSSL will not make those offers at its default security level, so the
probes carry @SECLEVEL=0; a protocol this openssl cannot offer at all is
reported as unprobed rather than counted as refused.
testssl.sh agrees with these results and is what found the first real failure
here -- humming-bird completes a TLS 1.0 handshake with
ECDHE-RSA-AES256-SHA. What decides it for the gate is cost: ~70ms for the
whole set against ~5s, and nothing outside the base image to pull or hang.
testssl.sh stays the better tool for an audit, where its wider suite coverage
and SSL Labs style grade earn the time, and the docs say so.
Verified on a clean host: humming-bird accepts tls1 and tls1_1, bun and caddy
accept neither.
* board: a TLS shield beside entries whose TLS was checked and clean
A small green shield after the framework name, on the table rows and in the
detail modal, for an entry that passed the TLS checks: serves the certificate
the harness mounts, negotiates TLS 1.3 with an AEAD cipher, and refuses
SSLv3, TLS 1.0/1.1 and the NULL, anonymous, export, RC4, DES and 3DES cipher
families.
The verdict is earned rather than declared. validate.sh writes
site/data/tls/<entry>.json when it actually runs those probes, the generator
folds it into meta, and the board reads meta[fw].tls. It is deliberately not
a meta.json field: that would let an entry claim the shield without being
checked, which is the opposite of the point.
Three states, and only one of them shows anything. "pass" gets the shield;
"fail" and absent both get nothing, because most entries have no TLS profile
at all and a red mark against them would say something untrue.
Seeded with the eight entries verified on a clean host this session -- bun,
caddy, elysia, genhttp-11-ioxide, h2o-mruby, hono-bun and slimeweb pass,
humming-bird fails on accepting TLS 1.0 and 1.1. Every other entry stays
unmarked until it is checked.
Hidden below 640px, where the name column is tightest.
check_badge_parity.js: 533 ranks match.
* ci: keep the TLS verdicts a validation run produces
validate.sh writes site/data/tls/<entry>.json when it runs the TLS probes,
and the board turns that into the shield -- but nothing collected the files,
so a full sweep computed 177 verdicts and threw all of them away.
One artifact per entry rather than a commit from inside the job: 177 matrix
jobs writing one directory would race, and a validation run committing to the
PR it is validating is a loop nobody wants. Collected after the run and
committed deliberately.
Uploaded with if: always(), because an entry that fails its TLS checks is
exactly the verdict worth keeping.
* ci: stop the detect step aborting on a disabled entry
all_entries() ends on a jq that exits non-zero for a disabled entry, so the
function inherited that status and `changed=$(all_entries)` aborted the whole
step under `bash -e` -- before a single line was echoed, which is why the
failure looked like nothing at all had run.
Whether it triggers depends on which entry sorts last and whether it happens
to be enabled, so it passed on the run that introduced it and failed on the
next one, after #1288, #1289 and #1291 disabled a batch of entries.
An explicit `return 0` on the function.
* validate: always mount the static directory, and raise the watchdog to 800s
Two things the full sweep on this PR exposed.
/data/static was mounted only when an entry subscribed to a static profile.
benchmark.sh mounts it unconditionally (scripts/lib/framework.sh), and the
entries that read the directory at startup cannot boot without it: rage and
rails copy it with FileUtils.cp_r, userver builds an fs-cache from it. So
when #1290 dropped the static profiles from those three, "does not serve
static" quietly became "does not start" -- all three failed the sweep with
the container dead before the first check.
It is a read-only bind of a small directory and there is nothing to be gained
by leaving it out, so validate.sh now matches benchmark.sh and always mounts
it. rage goes from a dead container to 44 passed, 0 failed.
The watchdog goes 300s -> 800s. The entries with the widest profile sets --
humming-bird and the three web-framework-* entries -- were killed mid-run
rather than failing anything, so their results said nothing at all.
* board: drop the TLS shield; the tls_check badge replaces it
Two badges for one subject was one too many, and the green shield was the
weaker of the two: it marked TLS basics an entry got right by default, while
the gold badge marks a hardening bar an entry opted into and wrote code for.
Removes the renderer, its CSS, the three render sites, the meta field the
board read and the loader that filled it. validate.sh still writes
site/data/tls/<entry>.json -- the verdict is what the opt-in badge is built
on -- but the files themselves are no longer committed, since nothing here
consumes them any more.
It also repairs a real break. The modal insertion had produced
'</h3>'+tlsShield(fw)<span class="badge b-'+t+'"
-- the +' was lost, which is a syntax error in the board's only script block,
so the page would not have rendered at all. The earlier check evaluated
data.js and the shield function in isolation and never parsed index.html,
which is exactly the gap that let it through. Both are parsed now:
all script blocks parse
badge parity ok — 533 ranks match
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes, both from the full sweep in #1285.
1.
staticandstatic-tlsstop feeding the framework compositeBoth go to
scored=False— the shapepipelinedalready has. Still measured, still shown, faded on the board, out of the sum.infraScoredstaysTrue: serving files fast from a tuned cache is what a proxy is for, and that league is unaffected.engineScoredwas alreadyFalse.CATALOGis the single source of truth here — the board reads it throughdata.js(PROF←D.profiles) and the badge port reads it through_scored_for(). The two scoring implementations cannot disagree about this, socheck_badge_parity.jsstays green.2. Twelve entries drop the two profiles
These fail validation on static and nothing else, and are flagship / emerging / experimental:
Each loses
static/static-tlsfrom itstestsarray and the result keys those profiles produced — 69 keys across the twelve.They stay enabled: static was their only failure, and everything else they publish is unaffected.
Scope
static-h2andstatic-h3are left alone — the failures were on the H1 pair.fastpysgi-asgi,fastpysgi-wsgi,uvicorn(engine) andh2o,varnish(infrastructure).frankenphp-trueasyncandh2o-mrubyare not here either. They failed the sweep because their containers never started on missing shared libraries, which json-tls coverage for flagship/emerging entries on the H1 table #1273 fixes — a different problem.