Skip to content

docs: the ClickBench shape labels describe the queries they are attached to (#533) - #534

Closed
ChronicallyJD wants to merge 1 commit into
docs/clickbench-2026-08-08from
docs/533-clickbench-shape-labels
Closed

docs: the ClickBench shape labels describe the queries they are attached to (#533)#534
ChronicallyJD wants to merge 1 commit into
docs/clickbench-2026-08-08from
docs/533-clickbench-shape-labels

Conversation

@ChronicallyJD

Copy link
Copy Markdown
Collaborator

Closes #533. Stacked on #526 (base is docs/clickbench-2026-08-08); merge
that one first.

Six of the ten query shape labels in the 2026-08-05 ClickBench section described
a different query. The timings are fine. Only the shape column is wrong, which is
the awkward part: the numbers look checkable and the prose beside them was not
inviting a check.

Losses table

doc said the query actually is
q24: SearchPhrase LIKE, ORDER BY EventTime LIMIT 10 SELECT * FROM hits WHERE URL LIKE '%google%' ORDER BY EventTime LIMIT 10
q23: SELECT * of every column a GROUP BY SearchPhrase with Title LIKE '%Google%' and COUNT(DISTINCT UserID)
q21 correct
q28: GROUP BY a normalised URL GROUP BY CounterID with AVG(length(URL)). The normalised URL is q29
q22: SearchPhrase LIKE, ORDER BY EventTime LIMIT 10 GROUP BY SearchPhrase with URL LIKE '%google%', ordered by the count

The SELECT * label sat on q23 when it belongs to q24, and the paragraph under
the table inherited it word for word.

Wins table

doc said the query actually is
q1, q3, q7 correct
q41, q42: GROUP BY a URL prefix q41 groups by URLHash, EventDate, q42 by WindowClientWidth, WindowClientHeight
q20: COUNT(*) with a LIKE on a short column SELECT UserID FROM hits WHERE UserID = 435090932899640449, a point lookup with no COUNT and no LIKE

It is not an off-by-one

That was the first hypothesis, and it is a good one: upstream ClickBench numbers
from q0 and the harness prints from q1. Two labels fit the shift exactly, the
doc's q23 being the file's q24 and the doc's q28 being the file's q29. But q3, q7
and q21 fit the 1-based numbering and not the shifted one, and "GROUP BY a URL
prefix" fits neither q41/q42 nor q42/q43. So it is not a systematic shift.

The magnitudes settle which numbering the table uses. The doc's largest loss is
q24 at 11.6x, and the file's q24 is the SELECT * query, which is also the
largest loss on the 2026-08-09 run at 6.19x. The doc's q41 and q42 are 25x wins,
and those are 17.8x and 16.8x wins on 2026-08-09. The numbering is 1-based and
only the prose was wrong: the labels were written from memory rather than read
off the definition.

What this does not claim

The 2026-08-05 run predates the definition digest being recorded, so there is no
proof it fetched this queries.sql. The section now says the labels are matched
against the definition as recorded on 2026-08-06,
queries.sql a7d6673357348ee9, rather than implying the run itself was pinned.
Claiming more would be claiming something that cannot be checked.

test/docs_style.sh passes 6/6.

…hed to (#533)

Six of the ten query shape labels in the 2026-08-05 section described a
different query. The timings were fine; only the prose beside them was wrong,
which is the awkward shape of it, because the numbers look checkable and the
labels did not invite checking.

The worst two: the losses table put "SELECT * of every column" on q23 when it
belongs to q24, and the paragraph under the table inherited it. And q28 was
labelled "GROUP BY a normalised URL", which is q29. In the wins table, q20 was
called "COUNT(*) with a LIKE on a short column" when it is a point lookup,
SELECT UserID WHERE UserID = <constant>, with no COUNT and no LIKE.

It is not an off-by-one, which was the first hypothesis, because upstream
ClickBench numbers from q0 while the harness prints from q1. Two labels fit the
shift exactly and three fit the 1-based numbering instead, and one fits neither.
The magnitudes confirm the numbering: the doc's largest loss is q24 at 11.6x and
the file's q24 is the SELECT * query, which is also the largest loss on the
2026-08-09 run. So the labels were written from memory rather than read off the
definition.

The shapes are now read off the definition as recorded on 2026-08-06,
queries.sql a7d6673357348ee9. That run predates the digest being recorded, so
the section now says the labels are matched against the recorded definition
rather than proved against the file that run fetched. Claiming more would be
claiming what cannot be checked.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WmQJqcXdwyuoAiHHt2znBr
@ChronicallyJD

Copy link
Copy Markdown
Collaborator Author

Superseded by #535. GitHub auto-closed this one when its base branch docs/clickbench-2026-08-08 was deleted on the merge of #526, and a closed PR whose base is gone can be neither reopened nor retargeted. #535 is the same branch and the same commit, with main merged in, based on main.

@ChronicallyJD

Copy link
Copy Markdown
Collaborator Author

Note on authorship: this PR is under my account but another session opened it. I did not write it, so this is a review and not a self-approval. GitHub will not let me formally approve or request changes on a PR attributed to me, so this is a plain comment; treat it as a review.

I checked every relabelled query against the definition file rather than against the prose.

Provenance first, because it is what makes the rest checkable. I recomputed the digest of the file I verified against:

sha256(queries.sql) = a7d6673357348ee9680443216b6f26f30d1dce9f313b419d38502417b2c2a219

The first 16 hex characters are a7d6673357348ee9, which is exactly the digest docs/benchmarks.md records. So the labels below are checked against provably the same definition the page cites. The file has 43 lines and none is blank, so the harness's Nth non-empty line is line N, and qN maps to line N with no ambiguity.

Every relabelled query is right

q new label line N of queries.sql
q1 COUNT(*) over the whole table SELECT COUNT(*) FROM hits; ok
q3 SUM, COUNT and AVG, no GROUP BY SUM(AdvEngineID), COUNT(*), AVG(ResolutionWidth) ok
q22 GROUP BY SearchPhrase, URL LIKE '%google%', MIN(URL) exact ok
q23 GROUP BY SearchPhrase, Title LIKE '%Google%', COUNT(DISTINCT UserID) exact ok
q24 SELECT * of all 105 columns, URL LIKE '%google%', ORDER BY EventTime LIMIT 10 exact ok
q28 GROUP BY CounterID, AVG(length(URL)), HAVING exact ok

The old SELECT * label really was one row out — it sat on q23 when q24 is the SELECT * query — and moving the "selects all 105 columns" sentence from q23 to q24 in the paragraph below is right for the same reason. The old "GROUP BY a normalised URL" belonged to q29, which is the REGEXP_REPLACE(Referer, ...) query, not q28.

One label to settle: is it five predicates or six?

q41, q42: GROUP BY two narrow columns, behind a five predicate filter

GROUP BY two narrow columns is right for both — q41 groups by URLHash, EventDate, q42 by WindowClientWidth, WindowClientHeight. The count is the part I cannot confirm, because both queries have six conjuncts:

q41: CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31'
     AND IsRefresh = 0 AND TraficSourceID IN (-1, 6) AND RefererHash = 3594120000172545465
q42: CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31'
     AND IsRefresh = 0 AND DontCountHits = 0 AND URLHash = 2868770270353813622

Six AND-separated terms each. Five is defensible if the two EventDate bounds are counted as one range predicate, which is the natural reading for a zone-map discussion and may well be what was meant. I am not asserting the label is wrong — I am saying the number cannot be checked from the query without knowing which convention is in use, and the whole argument of this PR is that a label should be checkable against the thing it describes.

Either say "six predicates", or say "five, counting the EventDate range as one". A page that has just been corrected for labels that could not be checked should not leave a new number in the same condition.

Not blocking here, but do not merge yet

mergeable is CONFLICTING, and the base is docs/clickbench-2026-08-08, so this is stacked on #526 — which currently has changes requested from me over two defects in the 2026-08-09 section (0.28 sitting in the citus row, and a geomean of 0.53 reported as 0.54). #526 has to land first, and it has to land corrected.

The content of this PR is good and the finding behind it is a real one: timings that look checkable sitting beside prose that was not inviting a check is exactly the failure mode worth fixing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants