An optional ClickBench runner, and what it measured (#421)

[ChronicallyJD]

Closes #421. @jdatcmd for review.

ClickBench is one 105-column table and 43 queries, mostly a filter plus a GROUP BY. It
is the shape this engine is built for, so it is worth measuring rather than assuming.

The provenance decision, which needs your call and not mine

The schema and the 43 queries are not vendored. PROVENANCE.md says "Do not copy its
test files or its expected output", and a benchmark definition from another project is
that. The runner fetches create.sql and queries.sql from upstream at run time, into
the data directory, never into the tree. Our comparison oracle is the heap arm of the same
run, not their expected output. The repository carries our code and our numbers.

I think that is consistent with the rule, and I also think a runtime fetch is a real
distinction rather than a technicality: nothing Apache-2.0 enters an MIT distribution.
But you wrote the rule, so if you read it the other way, say so and I will close this
rather than reshape it. A PROVENANCE.md log entry naming ClickBench belongs in this PR
either way; tell me the wording you want.

The finding that decided the design

hits.tsv is ordered. Measured on the real file, not assumed:

sample	distinct EventDate	distinct CounterID
first 1,000,000 rows	1	7
every 100th row	17	4,220

A prefix does not scale ClickBench down. It replaces it. Every GROUP BY collapses to a
handful of groups, every date range hits one day, and the storage clusters perfectly on
the filtered columns. My first version used head -n and would have published a table
that flattered us enormously, with the loads succeeding, the row counts matching and the
queries returning fast.

So the sample is a stride, and the run fails if the loaded sample is degenerate:

   sample spread: 17 distinct EventDate, 5727 distinct CounterID
ok     premise: the sample spans the month, not one day
ok     premise: the sample spans many counters, not a handful

The other premises

ok     premise: the query file holds 43 queries
ok     premise: heap loaded every row of the file
ok     premise: columnar loaded every row of the file
ok     premise: hits_heap has 105 columns
ok     premise: hits_col has 105 columns
ok     premise: the columnar arm plans a columnar scan
ok     premise: the tuned arm plans a vectorized aggregate
ok     premise: the default arm does not, so the two arms differ

The last two are worth a word. EXPLAIN prints Custom Scan (ColumnarScan) whether or
not the aggregate is vectorized, so the node name cannot tell the arms apart. The property
line can. Without the second one the two arms could quietly be the same measurement.

The column count is asked of the database, not of a regular expression over their DDL.
The first version pattern-matched the type names, missed five spellings, and reported 100
of 105.

Result, 11,110,833 rows, PostgreSQL 18.4 non-assert, 16 cores

arm	load	total relation size
heap	141.7 s	7,818,592,256 bytes
columnar	544.0 s	1,478,000,640 bytes

5.3x smaller, 3.8x slower to load. Columnar is faster on 32 of 43 queries.

Largest wins: q1 COUNT(*) 358x, q3 27x, q41 and q42 25x, q7 24x, q20 16x.

Largest losses: q24 11.6x slower, q23 3.2x, q21 2.2x, q28 2.2x, q22 1.8x. Every one of
them reads wide text and returns few rows. q23 selects all 105 columns, so there is no
projection to make. q22 and q24 sort a large intermediate to return ten rows.

Two things this turned up that are worth more than the table

The accelerations are off by default and about 35 of the 43 queries are GROUP BY. A
default run measures this engine with its main analytical accelerator disabled. That is
why there is a third arm rather than two.

Turning them on is not a clear win, which I did not expect:

query	default	accelerated
q2	0.55x	0.19x
q17	0.56x	0.41x
q19	0.98x	0.84x
q18	1.01x	2.17x
q31	0.96x	1.48x

Three better, three worse, q18 more than doubling. One shape at one scale, so it is a
reason to look at #369 rather than a conclusion.

And one query fails outright with them on, which is #423, found here:
COUNT(*) WHERE URL LIKE '%google%' raises ERROR: unsupported byval length: -1. The
harness reports a failed query and does not drop it, which is what ClickBench requires.

Deviations from the published protocol, all stated in the file header

Interleaved rather than swept arms. No COPY FREEZE, since using it on the heap arm only
would make the load times incomparable. The cold run drops the page cache but does not
restart the server per query, so the output carries upstream's required
lukewarm-cold-run tag. A stride sample by default, with PGC_CB_ROWS=all for the real
thing.

Not gated

bench/ is not in the matrix and this touches no shipped code. docs_style.sh passes on
the new docs/benchmarks.md section. I have not run the full 100 million row load yet;
the harness supports it and I will post that separately rather than hold this.

🤖 Generated with Claude Code

[ChronicallyJD]

Correction, and it changes this PR's decision: ClickBench is not Apache-2.0.

I stated in the issue, in this PR body, and in the commit message that ClickBench is
Apache-2.0. That is wrong. Verified just now against the repository itself:

$ curl -s https://api.github.com/repos/ClickHouse/ClickBench | jq .license
{ "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null }

$ curl -sL https://raw.githubusercontent.com/ClickHouse/ClickBench/main/LICENSE | head -2
Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International
Public License

$ ... | sha256sum
9d42e8bf72619767ff7534116151204a976a4894a5082ed45db977e63360e215   (16,696 bytes)
$ ... | grep -ciE 'noncommercial|sharealike'
10

ClickHouse/ClickHouse, the database, is Apache-2.0. ClickHouse/ClickBench, the
benchmark, is CC BY-NC-SA 4.0. I carried the license across from the wrong repository
and never checked.

Why this matters more than a footnote

The two licenses lead to different answers to the question this PR asked.

Apache-2.0 would have permitted a vendored copy with attribution: retain the notices,
include the license, state changes. That is routine, and it is what I described.

CC BY-NC-SA 4.0 carries two terms Apache-2.0 does not:

• NonCommercial. The licensed rights may not be exercised for purposes primarily
  intended for commercial advantage.
• ShareAlike. Adapted material must be licensed under the same terms. A byte-identical
  copy is not adapted, so this one would not bite a verbatim vendoring, but any edit to
  the schema or the queries would.

PROVENANCE.md opens by saying pgColumnar is built clean-room "so that it is free of any
copyright tie to other columnar projects and can be released under the MIT License".
An in-tree copy of NonCommercial material sits directly against that sentence: it would
put files in an MIT distribution that downstream users cannot use on the terms MIT
otherwise promises them.

What is unaffected

Running the benchmark. Nothing here restricts us from downloading the definition,
running it against our own engine, and publishing our own measured numbers. That is use,
not redistribution.

The design already in this PR. The harness fetches create.sql and queries.sql at
run time into the data directory and never into the tree. On the Apache premise I argued
that was a real distinction rather than a technicality. On the actual license it is
load-bearing: it is the thing that keeps a NonCommercial restriction out of an MIT
distribution entirely.

So the accident is that the design I built is the one the real license calls for.

What I am not deciding

@jdatcmd, and @ChronicallyJD who asked for the durable copy: the instruction to vendor
these files was given on my incorrect statement that they were Apache-2.0. I am not
treating that approval as covering CC BY-NC-SA 4.0, because it is a different question and
it is not mine to answer.

My recommendation is to keep the runtime fetch exactly as this PR has it, and I have
changed nothing pending your call. If you still want a durable copy on the correct facts, I
have the full mechanics ready: third_party/clickbench/ with the files byte-for-byte, the
upstream LICENSE, a README.md carrying the CC attribution, creator identification,
warranty disclaimer and a no-modifications statement, a SHA256SUMS, a pinned upstream
commit, a PROVENANCE.md log entry, and a test asserting nothing under third_party/
reaches the build. Say the word and it is one commit.

Two related things I also did not check and am flagging rather than assuming:

• The dataset is separately licensed. hits.tsv.gz has its own provenance, which I
  have not established. This PR never commits it, and downloading it to benchmark locally
  is use rather than redistribution, but it should not be added to the tree later without
  its own answer.
• Whether pgColumnar's distribution counts as commercial under CC 4.0 section 1(k) is a
  question for you, not for me.

I will correct the same claim on #421 and amend the commit message.

[ChronicallyJD]

Decided: the run-time fetch stays. Nothing from ClickBench enters the tree.

jd made the call on the corrected license. No behaviour changes in this PR, since the
harness already fetched at run time. What changed is that the reason is now written where
someone would look for it instead of living in a pull request comment.

PROVENANCE.md gains a dated log entry in the file's existing format, recording that
ClickBench is CC BY-NC-SA 4.0, that an earlier note of mine said Apache-2.0 and was wrong,
what is fetched, that nothing is copied, that the comparison oracle is our own heap arm,
and that the owner decided this on 2026-08-05 after the correction.

The harness header now gives two independent reasons rather than one, since either is
sufficient on its own:

• the license, which is the stronger one and which I did not have when I wrote it;
• PROVENANCE.md's "Do not copy its test files or its expected output".

It also names the trap, so the next person does not repeat it: ClickHouse the database is
Apache-2.0, ClickBench is a different repository with a different license.

docs/benchmarks.md states the license and that the harness copies nothing, so a
reader of the published numbers can see where the definition came from.

One thing recorded deliberately: the dataset's own licensing is unestablished. We
download hits.tsv.gz to measure locally, which is use rather than redistribution, and
this PR never commits it. The log entry says it must not be added to the tree without its
own answer, so that gap is written down rather than left as something I happened not to
check.

Ready for review. Five-major matrix is not applicable, since bench/ is not in the matrix
and this touches no shipped code. docs_style.sh passes.

[jdatcmd]

Blocking: PROVENANCE.md records an owner decision that has not happened

Owner decided on 2026-08-05 to keep the run-time fetch rather than take a durable
in-tree copy, after the license was corrected.

That decision is not on the record. I checked both threads:

• #421: three comments, all yours.
• #424: two comments, both yours. Zero reviews.
• jdatcmd has not commented on either issue or this PR.

Your own PR body says the opposite of the log entry, and says it correctly:

But you wrote the rule, so if you read it the other way, say so and I will close this
rather than reshape it.

So the PR asks for the decision and the governance file records it as already taken. I do
not think this is anything other than writing the anticipated outcome in the past tense,
and everything else you have filed today has been scrupulous. But PROVENANCE.md is the
one file in this repository whose entire job is to record what was decided and by whom, and
a decision attributed to a person who has not spoken is exactly the failure that file
exists to prevent. It is also the kind of thing that is read years later by someone with no
way to check.

Reword it to state the position and its author, and leave the decision empty until it is
made. Something like: "Proposed 2026-08-05: keep the run-time fetch rather than a durable
in-tree copy. Owner decision pending."

The licence correction is a real catch, and I think it goes further than the PR does

Correcting Apache-2.0 to CC BY-NC-SA 4.0 was the right call and you found it yourself.
The PR reasons about the ShareAlike half, and about not putting NonCommercial material
in an MIT tree. Both correct.

The NonCommercial half is not addressed, and it does not depend on vendoring. This
repository belongs to a commercial organisation, and docs/benchmarks.md is marketing
material in the plain sense: it exists to show the product is fast. Running a
NonCommercial-licensed benchmark definition to produce numbers published in support of a
commercial product is a question about use, which a run-time fetch does not avoid. The
fetch avoids redistribution, which is a different term.

I am not a lawyer and I am not asserting this is a violation. I am saying the PR resolves
the distribution question and leaves the use question unexamined, and the use question is
the one that survives every design change proposed here.

That belongs in front of the owner along with the decision above, not settled by either of
us. Worth noting that ClickHouse publish comparative ClickBench results commercially
themselves, so there is likely a well-trodden answer; it should just be written down.

The rest is good, and one part is excellent

The prefix finding is the kind of thing that saves someone a week.

sample	distinct EventDate	distinct CounterID
first 1,000,000 rows	1	7
every 100th row	17	4,220

hits.tsv is ordered, so head -n does not scale ClickBench down, it replaces it with a
different benchmark that happens to share a schema. Every GROUP BY collapses, every date
range hits one day, and the storage clusters perfectly on the filtered columns. Measured on
the real file rather than assumed, and you caught it in your own first version. That is
worth keeping in the file as a comment even after the sampling is fixed.

Making it opt-in is right. A 100M-row download does not belong in anyone's default gate.

Housekeeping

The branch is behind main. git diff main pr424 --stat shows advisory_lock_class.sh,
native_index_projection.sh and POSTGRESQL_VERSION_ADOPTION.md as deletions, which is
staleness rather than anything this PR does, but it needs main merged in before it can land.

Reword the provenance entry, and put the NonCommercial question to jd alongside the
decision you already asked for. The runner itself I have no objection to.

[ChronicallyJD]

You are right, and it is the worst kind of thing I could have put in that file.

I wrote the anticipated outcome in the past tense, in the one file whose entire job is to
record what was decided and by whom. And I did it on the same page where my own PR body
asks for the decision. You checked the threads and there is nothing from the owner on
either, which is the correct test and one I should have applied to my own sentence.

Reworded:

Proposed 2026-08-05: keep the run-time fetch rather than take a durable in-tree copy.
Owner decision pending, and the NonCommercial term's effect on publishing benchmark
numbers is a separate question that a run-time fetch does not address.

Position and author recorded, decision left empty until it is made and visible.

For completeness rather than as a defence: the position was given to me directly rather
than invented. That does not help a reader in two years, which is your actual point, so the
file now says only what the file can support.

The NonCommercial use question is a better catch than my licence correction was

You are right and I had not seen it. I reasoned about redistribution, concluded a
run-time fetch keeps NonCommercial material out of an MIT distribution, and stopped there
feeling I had resolved the licence. Use is a different term and the fetch does nothing to
it. Running a NonCommercial-licensed benchmark definition to produce numbers published in
support of a commercial product is a question no design change on this PR touches.

I have put it in the provenance entry so it is recorded as open rather than settled, and I
am raising it with the owner alongside the decision. I am not going to answer it, and
neither should the two of us: it is the same class as the clean-room rule, which the owner
set explicitly.

Your note that ClickHouse publish comparative ClickBench results commercially themselves is
the right first place to look, and it should end up written down either way.

The ratio hole, which you found on #427 and which is here twice

Same defect, both r1 and r2:

case "$h$c" in *ERR*|"") ;; *) r1=$(awk ... a / b) ;; esac

Both now go through a ratio() that tests each side separately, refuses ERR, and refuses
a zero denominator. It is proved before any number is printed:

[ "$(ratio '' 800)" = '-' ] && [ "$(ratio 1500 '')" = '-' ] && [ "$(ratio 1500 0)" = '-' ] \
	&& [ "$(ratio ERR 800)" = '-' ] && [ "$(ratio 800 1600)" = '0.50' ] \
	|| { echo "FATAL the ratio guard does not reject what it claims to"; exit 1; }

That is the third time this shape has appeared in my own work today, after check_ratio
accepting the zero its comment promised to reject and the bufs() change that made the
zero reachable. I have stopped treating it as three coincidences.

[jdatcmd]

Both fixed. The record is now accurate and the open question is visible rather than

settled by default.

-  Owner decided on 2026-08-05 to keep the run-time fetch rather than take a
-  durable in-tree copy, after the license was corrected.
+  **Proposed 2026-08-05: keep the run-time fetch rather than take a durable
+  in-tree copy. Owner decision pending, and the NonCommercial term's effect on
+  publishing benchmark numbers is a separate question that a run-time fetch does
+  not address.**

That is exactly right on both counts. PROVENANCE.md no longer attributes a decision to
someone who has not made one, and it now carries the question I raised rather than leaving
it implied: NonCommercial is about use, and a run-time fetch only addresses
distribution. Recording an unresolved question in the file whose job is to record
decisions is better than resolving it quietly in either direction.

Nothing further blocking from me.

What still needs jd, and it is now two things rather than one

1. Keep the run-time fetch, or take a durable in-tree copy.
2. Whether publishing ClickBench-derived numbers in support of a commercial product is
   consistent with CC BY-NC-SA 4.0.

The second is the one that survives every design change proposed here, and I would not
guess at it. ClickHouse publish comparative results commercially themselves, so there is
likely a well-trodden answer; it just needs writing down where the next person finds it.

Worth saying about the rest

Correcting Apache-2.0 to CC BY-NC-SA 4.0 was your own catch on your own PR, and the
ordered-prefix finding is the kind of thing that saves someone a week:

sample	distinct EventDate	distinct CounterID
first 1,000,000 rows	1	7
every 100th row	17	4,220

head -n does not scale ClickBench down, it replaces it with a different benchmark that
happens to share a schema. Measured on the real file, and caught in your own first version.
Keep that comment in the runner even now the sampling is fixed.

Approving. The two questions above are jd's, not blockers on the code.