docs: the ClickBench shape labels describe the queries they are attached to (#533) - #534
Conversation
…hed to (#533) Six of the ten query shape labels in the 2026-08-05 section described a different query. The timings were fine; only the prose beside them was wrong, which is the awkward shape of it, because the numbers look checkable and the labels did not invite checking. The worst two: the losses table put "SELECT * of every column" on q23 when it belongs to q24, and the paragraph under the table inherited it. And q28 was labelled "GROUP BY a normalised URL", which is q29. In the wins table, q20 was called "COUNT(*) with a LIKE on a short column" when it is a point lookup, SELECT UserID WHERE UserID = <constant>, with no COUNT and no LIKE. It is not an off-by-one, which was the first hypothesis, because upstream ClickBench numbers from q0 while the harness prints from q1. Two labels fit the shift exactly and three fit the 1-based numbering instead, and one fits neither. The magnitudes confirm the numbering: the doc's largest loss is q24 at 11.6x and the file's q24 is the SELECT * query, which is also the largest loss on the 2026-08-09 run. So the labels were written from memory rather than read off the definition. The shapes are now read off the definition as recorded on 2026-08-06, queries.sql a7d6673357348ee9. That run predates the digest being recorded, so the section now says the labels are matched against the recorded definition rather than proved against the file that run fetched. Claiming more would be claiming what cannot be checked. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WmQJqcXdwyuoAiHHt2znBr
|
Note on authorship: this PR is under my account but another session opened it. I did not write it, so this is a review and not a self-approval. GitHub will not let me formally approve or request changes on a PR attributed to me, so this is a plain comment; treat it as a review. I checked every relabelled query against the definition file rather than against the prose. Provenance first, because it is what makes the rest checkable. I recomputed the digest of the file I verified against: The first 16 hex characters are Every relabelled query is right
The old One label to settle: is it five predicates or six?
Six Either say "six predicates", or say "five, counting the Not blocking here, but do not merge yet
The content of this PR is good and the finding behind it is a real one: timings that look checkable sitting beside prose that was not inviting a check is exactly the failure mode worth fixing. |
Closes #533. Stacked on #526 (base is
docs/clickbench-2026-08-08); mergethat one first.
Six of the ten query shape labels in the 2026-08-05 ClickBench section described
a different query. The timings are fine. Only the shape column is wrong, which is
the awkward part: the numbers look checkable and the prose beside them was not
inviting a check.
Losses table
SearchPhrase LIKE,ORDER BY EventTime LIMIT 10SELECT * FROM hits WHERE URL LIKE '%google%' ORDER BY EventTime LIMIT 10SELECT *of every columnGROUP BY SearchPhrasewithTitle LIKE '%Google%'andCOUNT(DISTINCT UserID)GROUP BYa normalised URLGROUP BY CounterIDwithAVG(length(URL)). The normalised URL is q29SearchPhrase LIKE,ORDER BY EventTime LIMIT 10GROUP BY SearchPhrasewithURL LIKE '%google%', ordered by the countThe
SELECT *label sat on q23 when it belongs to q24, and the paragraph underthe table inherited it word for word.
Wins table
GROUP BYa URL prefixURLHash, EventDate, q42 byWindowClientWidth, WindowClientHeightCOUNT(*)with aLIKEon a short columnSELECT UserID FROM hits WHERE UserID = 435090932899640449, a point lookup with noCOUNTand noLIKEIt is not an off-by-one
That was the first hypothesis, and it is a good one: upstream ClickBench numbers
from q0 and the harness prints from q1. Two labels fit the shift exactly, the
doc's q23 being the file's q24 and the doc's q28 being the file's q29. But q3, q7
and q21 fit the 1-based numbering and not the shifted one, and "GROUP BY a URL
prefix" fits neither q41/q42 nor q42/q43. So it is not a systematic shift.
The magnitudes settle which numbering the table uses. The doc's largest loss is
q24 at 11.6x, and the file's q24 is the
SELECT *query, which is also thelargest loss on the 2026-08-09 run at 6.19x. The doc's q41 and q42 are 25x wins,
and those are 17.8x and 16.8x wins on 2026-08-09. The numbering is 1-based and
only the prose was wrong: the labels were written from memory rather than read
off the definition.
What this does not claim
The 2026-08-05 run predates the definition digest being recorded, so there is no
proof it fetched this
queries.sql. The section now says the labels are matchedagainst the definition as recorded on 2026-08-06,
queries.sql a7d6673357348ee9, rather than implying the run itself was pinned.Claiming more would be claiming something that cannot be checked.
test/docs_style.shpasses 6/6.