bench: a bulk Citus arm, so the bulk row compares like with like (#445)

[ChronicallyJD]

The ClickBench section compared pgcolumnar's 16-worker parallel_copy against Citus's single-connection COPY, because the harness had no bulk Citus arm. That is our best path against their non-best one, on the exact question #445 was opened about.

Citus does have a bulk path

Measured on the bench, its own cluster on a free port, citus_columnar 14.1-1 and pgcolumnar co-loaded, 4,000,000 rows, one split shared by every arm, three runs:

arm	seconds
citus, serial COPY	3.06, 3.07, 3.06
citus, 8 concurrent COPY into one table	0.509, 0.518, 0.513
pgcolumnar, serial COPY	4.80, 4.80, 4.79
pgcolumnar, parallel_copy 8	1.137, 1.138, 1.135

All four loaded 4,000,000 rows. Scaling from concurrency: citus 5.9x, pgcolumnar 4.2x. So the premise the published ratio rested on does not hold.

This PR does not publish a corrected ratio. My fixture is three narrow columns; ClickBench hits is 105, and one fixture is not the claim. The number comes from a run of this arm on the real data, which is @jdatcmd's — the run and the record are his.

The arm

N concurrent connections, each fed a byte range from pgcolumnar.file_split_offsets through COPY ... FROM PROGRAM. That is our splitter, the same newline-aligned boundaries parallel_copy gives its own loaders, so the two bulk arms differ in the engine and not in how the file was divided — and nothing copies a 15 GB file N times.

Same row assertion as the pgcolumnar bulk arm, for the same reason: an arm that errored leaves an empty table and returns fast, which reads as a win. #465's failure with a different cause.

The preflight that would have caught my own mistake

max_worker_processes is now preflighted beside max_prepared_transactions. An N-worker parallel_copy needs N + 2: one background worker per loader, one coordinator, and one already held by the logical replication launcher. The stock default is 8, so an 8-worker arm fails at could not register pgcolumnar parallel_copy loader 7 of 8 and leaves an empty table.

N + 2 is measured, not reasoned. Sweeping the setting against three worker counts, the smallest value that loaded every row was:

workers	minimum max_worker_processes
2	4
4	6
8	10

One below each failed on the last loader with the table left empty. Three counts rather than one, because a rule extrapolated from a single observation is a guess wearing a measurement's clothes.

I hit this exactly while measuring the Citus arms. The only reason I caught it is that the row count was asserted: the arm returned in about no time with nothing in it.

Tests

bench_guards.sh covers the new guard the way it already covers max_prepared_transactions: both thresholds either side of the measured minimum, the rule at a second worker count, a serial arm needing none, and non-numeric input refused rather than compared. 17 checks to 28.

The arm's mechanism was verified end to end against real citus_columnar on a 200,000-row fixture before shipping, with content checks rather than cardinality:

offsets: 5 (want 5)     rows: 200000/200000 OK
sum(a):  exact OK       duplicate keys: 0 OK
per-worker COPY counts: 49305 + 49306 + 50534 + 50855 = 200000

A boundary that loses one row and gains another passes a count check and fails that one.

Gate

bench_guards 28/0, docs_style 6/0, harness_selftest 54/0, full matrix on PG18 and PG19 ALL VERSIONS PASSED.

Refs #445, #465, #526.

[jdatcmd]

Approved. This is the arm I asked for and it is fairer by construction than the one I would have written.

The design decision that matters

The split is ours. Both bulk arms divide the file with pgcolumnar.file_split_offsets — the same newline-aligned byte boundaries parallel_copy gives its own loaders — and run the same PGC_CB_PCOPY_WORKERS. So the two arms differ in the engine and not in how the file was divided, which is the only way this comparison can be trusted. Handing Citus a different split, or a different worker count, would have reproduced the original defect with the sign flipped.

FROM PROGRAM 'tail -c +N | head -c LEN' feeding each connection its range, rather than materialising a 15 GB file N times, is the right mechanism, and the 1-based +$((start + 1)) is commented where someone would otherwise ask.

Verified rather than read

run_clickbench.sh / cb_guards.sh / bench_guards.sh   syntax OK
bench_guards.sh                                       28 checks, 0 fails

And the slot guard at its boundaries, which is where an off-by-one would live:

workers=2  needs=4   at 3: REFUSED  at 4: ok
workers=4  needs=6   at 5: REFUSED  at 6: ok
workers=8  needs=10  at 9: REFUSED  at 10: ok
workers=16 needs=18  at 17: REFUSED at 18: ok
serial arm (0 workers) on stock 8: ok
empty / non-numeric inputs: REFUSED / REFUSED

Exact at every boundary, the serial arm is not blocked by a setting it does not use, and both sides refuse garbage rather than comparing it — the #418 shape, handled.

The two failures this closes are the same failure

max_worker_processes at the stock 8 fails an 8-worker arm at "could not register loader 7 of 8" and leaves an empty table, which returns fast and reads as excellent scaling. That is #465 with a different cause, and the row-count assertion on the new arm is what stops the Citus side producing it too. Preflighting a PGC_POSTMASTER setting before any arm loads, rather than discovering it in a load log, is the same construction as max_prepared_transactions and for the same reason.

N + 2 measured rather than reasoned is the part I would have got wrong. I would have written N + 1 from the coordinator and never accounted for the logical replication launcher already holding a slot. Sweeping three worker counts and recording the smallest value that loaded every row is how that number becomes a fact instead of an argument.

What I will do with it

I will re-run ClickBench on the bench with this arm and update #526 and #445 with the result. The docs currently say the bulk comparison is unmeasured, which was the only honest position available; this is what replaces it with a number.

Merging.