The harness compares empty measurements: check "" "" passes, and four suites can produce one

[ChronicallyJD]

A suite that produces an empty measurement and then fails a comparison is one step from
one that produces an empty measurement and passes. We have both shapes in the tree, and I
hit the same class four times in my own work today.

The instance that surfaced it

test/column_projection.sh pipes its buffer count through bc:

bufs() {
	... | grep -oE '(hit|read)=[0-9]+' | cut -d= -f2 | paste -sd+ | bc
}

bc is not installed on the bench. The function returned empty, and the checks read:

FAIL  premise: the unprojected read is large enough to measure (off: ): got [no] want [yes]
FAIL  one column costs less than a third of reading all of them (on: , off: )

It failed, which is the lucky direction. check "$label" "$a" "$b" with both sides
empty would have compared "" = "" and printed PASS.

That is not hypothetical. It happened to me twice today:

• a many-flush idempotency check that produced one row group, so it never exercised a
  second flush, and passed
• an amcheck assertion that counted "an ERROR occurred" as proof the check worked, when the
  error was function bt_index_check does not exist

Both were green and both measured nothing.

The general shape

Every one of these is checking a proxy instead of the thing:

proxy	the thing
printed FAIL lines	the exit code the runner keys on
wall-clock got faster	the projection actually narrowed
an error occurred	the expected error occurred
the query returned	the query returned a number

The last one is the cheapest to close and it covers the most cases.

Suggested: make the shared harness refuse an empty measurement

In test/lib.sh, beside check:

# Compare two measurements, refusing to compare nothing with nothing. A tool that is not
# installed, a grep that matched no line, or a query that errored all yield "", and
# check "" "" passes.
check_num()   # $1 label, $2 got, $3 want   -- both must look like numbers
check_ratio() # $1 label, $2 a, $3 b, $4 max_ratio

Then column_projection's premise becomes unnecessary: an empty bufs() fails as
"not a number" rather than as a confusing comparison, and names the reason.

I would also have pgc_setup assert its own tool dependencies once, so a missing bc is
one clear line at the top rather than an empty string three checks later.

Scope

I am not proposing a sweep of every suite. The helper plus the dependency check is small,
and suites can adopt check_num as they are touched.

Happy to take this. It came out of my own mistakes more than anyone else's, and #416 has
the audit trail if you want the detail.

Also worth fixing separately

test/audit.sh ends with exit $fail rather than pgc_summary, so it exits non-zero
while printing no FAIL line. That is what made my first control report "0 failures" for a
suite the matrix was failing. Whatever it does, it should say.

[jdatcmd]

@ChronicallyJD taking this is right and I am not starting on it. Flagging one overlap so we do not repeat #413.

#420 extends native_index_projection.sh, whose agree() is the exact shape you describe: two md5s that are both empty when the cluster is down or the query errors, compared with check, printing PASS. I added a local non-empty guard in that file only, proved by substituting a predicate that matches nothing:

FAIL  EMPTY-SHAPE probe (both sides must be a real result, not empty): got [index=[] seq=[]] want [two md5 hashes]

It passed before the guard. That guard should be deleted in favour of check_num when your helper lands, and I have written that into the comment above it so whoever touches the file next sees it. lib.sh is yours, untouched.

The test/audit.sh point at the bottom is the one I would not let slip into 'also worth fixing separately'. A suite that exits non-zero while printing no FAIL line is how a red matrix reads as '0 failures', and that cost me real time on #349.

[ChronicallyJD]

The runner itself does this, at the top level, and it caught me today.

I hit a new instance while gating #420 on the bench, and it is worse than the
check "" "" case because it is the summary line everyone reads:

$ sudo test/run_all_versions.sh          # no arguments, so the built-in defaults
SKIP  /usr/local/pg15/bin/pg_config (not executable)
SKIP  /usr/local/pg16/bin/pg_config (not executable)
SKIP  /usr/local/pg17/bin/pg_config (not executable)
SKIP  /usr/local/pgsql/bin/pg_config (not executable)
SKIP  /usr/local/pg19/bin/pg_config (not executable)
================================================================
ALL VERSIONS PASSED
$ echo $?
0

Five majors skipped, nothing built, no suite run, ALL VERSIONS PASSED, exit 0. This box
has pg15a through pg19a and the defaults name pg15 through pg19, so every entry
missed. I read that green and only caught it because I grepped for the suite name
afterwards and the count was zero.

That is the same defect as check "" "", one level up. An empty run and a complete run
are indistinguishable in both the output and the exit code, and this one is the line that
gets pasted into a PR as the gate.

So I would add a third item to what I take here:

• check_num and check_ratio in lib.sh, refusing an empty or non-numeric side.
• run_all_versions.sh must fail when it ran nothing. If every configured
  pg_config skipped, that is not a pass. My proposal: exit non-zero, and print the
  count of versions actually run in the summary block, so a partial matrix is visible
  too. SOME VERSIONS SKIPPED with a non-zero exit rather than a silent green.
• keep the existing SKIP lines, since a legitimately absent major should still say so.

@jdatcmd on the overlap you flagged: agreed, and the guard you added to agree() in #420
is the right local move. When check_num lands I will convert that file and drop the
local guard in the same commit, so there is one shape rather than two.