From 23c96c753e990a6c197fe6948fe4a9b039bd44bc Mon Sep 17 00:00:00 2001 From: "Joshua D. Drake" Date: Fri, 7 Aug 2026 12:40:42 -0600 Subject: [PATCH] fix: a suite must not decide a verdict from a pipeline's exit status (#486) `echo "$s" | grep -q PATTERN` under `set -o pipefail` answers "not found" whenever the WRITER fails, whatever the string contained. The reader exits as soon as it has its answer, the writer takes EPIPE, and pipefail calls the whole pipeline failed, so the `&&` arm never runs and the helper reports absence. Found when it reddened #484's PG18 CI on `native_agg`: native_agg.sh: line 41: echo: write error: Broken pipe FAIL count(*) uses the metadata agg node: got [no] want [yes] PASS sum/min/max uses the metadata agg node The failing check and the passing one below it call the SAME helper on plan text from the same node. The same job re-run on the identical commit passed. The failure direction is what makes this worth a rule rather than a fix in one file: it always reports the thing you were looking for as ABSENT, which reads as a real regression in whatever area the check covers. That one read as a planner regression in the area #133 and #140 live in. Measured, deterministically, before writing anything: form 300 KB string, match on line 1 echo "$s" | grep -q no <- wrong case "$s" in *pat*) yes grep -q pat <<<"$s" yes WHAT THE MECHANISM IS NOT. The issue first said grep's early exit causes this. At a kilobyte it cannot: an EXPLAIN plan fits entirely in the 64 KB pipe buffer, so the writer's single write() completes before grep can have matched anything. For the reader to be gone at that size something else has to have killed it, and under a six-way parallel matrix a grep killed under memory pressure is the plausible candidate. That is not proved and is not claimed. What is proved is that the helper's answer depends on the reader process surviving, and that pipefail turns anything happening to that process into a wrong answer. Rewritten as a herestring rather than a `case`, in 60 places across 19 files. `case` is what #473 used for the fixed-string membership test and is still right there; most of these carry regex, alternation and `-i`, which a glob cannot express, and a herestring keeps the pattern and the flags exactly as they were while removing the pipeline entirely. Its status is grep's alone. Verified equivalent on the positive, negative, regex, whole-line and unterminated-last-line cases. The rule is in `harness_selftest.sh`, with the control above it, because a rule with no demonstrated failure is a style preference and this one is not. Scope is deliberate and stated in the comment: a reader whose EXIT STATUS is the answer. `| head -1` inside a diagnostic string stays, because losing that pipeline's status changes a message and no verdict. `analyze_stats.sh` has several and they are left alone on purpose rather than missed. The sweep was scripted and the script was wrong once, which is why the diff was read rather than trusted: matching `grep` instead of `grep -q` rewrote `echo "$err" | grep -oiE ... | head -1` into `grep -oiE ... | head -1 <<<"$err"`, redirecting HEAD's stdin and leaving grep reading the script's. That grep's output is used, not its status, so it was never in scope. The script now matches only `-q` and refuses any line where a further unquoted `|` means grep is not the last stage. Proved by removal: re-arming one instance fails the rule naming the exact file and line, and restoring it passes. All 19 affected suites re-run. VERIFICATION. All 19 affected suites re-run, 19 PASSED and 0 FAILED. That does not cover everything: `fuzz_arrow` and `fuzz_parquet` run 4 checks on a clean run, and the 14 rewrites in them are crash and sanitizer CLASSIFICATION branches that a clean fuzz run never enters. Those were checked separately rather than counted as covered -- both forms of all three conditions, over nine inputs including an ASAN report, a UBSAN misaligned-load report, a signal-11 log and a connection refusal: 27 comparisons, all agreeing, with a sanity line proving the classifications are not all "miss", which would make agreement vacuous. Co-Authored-By: Claude Opus 5 (1M context) --- test/alter_column_type.sh | 2 +- test/cancel_decode.sh | 2 +- test/fuzz_arrow.sh | 14 +++---- test/fuzz_parquet.sh | 14 +++---- test/harness_selftest.sh | 66 +++++++++++++++++++++++++++++++++ test/native_agg.sh | 4 +- test/native_groupagg.sh | 4 +- test/native_ownership.sh | 2 +- test/native_reclaim_cycles.sh | 2 +- test/native_truncate.sh | 2 +- test/parallel_copy.sh | 22 +++++------ test/parallel_export_parquet.sh | 2 +- test/parallel_vector_agg.sh | 20 +++++----- test/phase4.sh | 6 +-- test/phase5.sh | 2 +- test/pushdown_report.sh | 4 +- test/run_san.sh | 2 +- test/server_file_privilege.sh | 4 +- test/ungrouped_vector_agg.sh | 10 ++--- test/unique_conc.sh | 2 +- 20 files changed, 126 insertions(+), 60 deletions(-) diff --git a/test/alter_column_type.sh b/test/alter_column_type.sh index ac95fe07..28b6e49c 100755 --- a/test/alter_column_type.sh +++ b/test/alter_column_type.sh @@ -55,7 +55,7 @@ conv() { # label, column type, value expression, alter clause err="$(psql_run "ALTER TABLE $c ALTER COLUMN v TYPE $4;" 2>&1 || true)" psql_run "ALTER TABLE $h ALTER COLUMN v TYPE $4;" >/dev/null 2>&1 - if echo "$err" | grep -qiE "corrupt encoded chunk|server closed|terminated"; then + if grep -qiE "corrupt encoded chunk|server closed|terminated" <<<"$err"; then check "$1" "failed: $(echo "$err" | grep -oiE 'corrupt encoded chunk[^\"]*|server closed' | head -1)" "ok" return fi diff --git a/test/cancel_decode.sh b/test/cancel_decode.sh index 877def9d..bbb9dec5 100755 --- a/test/cancel_decode.sh +++ b/test/cancel_decode.sh @@ -79,7 +79,7 @@ cancel_ms() { c=$(date +%s%N) out="$(raw "SET statement_timeout = 100; $(printf "$(QUERY "$t")")")" d=$(date +%s%N) - echo "$out" | grep -qi "canceling statement" || { echo FAILED; return; } + grep -qi "canceling statement" <<<"$out" || { echo FAILED; return; } ms=$(( (d - c) / 1000000 )) [ -z "$best" ] || [ "$ms" -lt "$best" ] && best="$ms" done diff --git a/test/fuzz_arrow.sh b/test/fuzz_arrow.sh index 946eb6d1..86bc7a9f 100644 --- a/test/fuzz_arrow.sh +++ b/test/fuzz_arrow.sh @@ -173,8 +173,8 @@ run_stmt() { -c "SET statement_timeout = '20s'; $stmt" 2>&1)" rc=$? - if echo "$out" | grep -qE 'could not connect|No such file or directory|Connection refused' && - ! echo "$out" | grep -q '^ERROR:'; then + if grep -qE 'could not connect|No such file or directory|Connection refused' <<<"$out" && + ! grep -q '^ERROR:' <<<"$out"; then if ! wait_for_cluster; then echo " FATAL: cluster unreachable and did not return" return 1 @@ -183,14 +183,14 @@ run_stmt() { log_since newlog="$(cat "$NEWLOG" 2>/dev/null)" - if echo "$newlog" | grep -qE 'AddressSanitizer|runtime error:|UndefinedBehaviorSanitizer|LeakSanitizer'; then + if grep -qE 'AddressSanitizer|runtime error:|UndefinedBehaviorSanitizer|LeakSanitizer' <<<"$newlog"; then sanitizer=$((sanitizer + 1)) save_finding san "$seedfile" "$mutseed" "$stmt" "$newlog" wait_for_cluster || return 1 return 1 fi - if echo "$newlog" | grep -qE 'was terminated by signal|server process .* exited with|crashed'; then + if grep -qE 'was terminated by signal|server process .* exited with|crashed' <<<"$newlog"; then crashes=$((crashes + 1)) save_finding crash "$seedfile" "$mutseed" "$stmt" "$newlog" wait_for_cluster || echo " (cluster did not come back)" @@ -204,20 +204,20 @@ run_stmt() { return 1 fi - if echo "$out" | grep -q 'canceling statement due to statement timeout'; then + if grep -q 'canceling statement due to statement timeout' <<<"$out"; then hangs=$((hangs + 1)) save_finding hang "$seedfile" "$mutseed" "$stmt" "$out" return 1 fi - if echo "$out" | grep -qE 'server closed the connection unexpectedly|connection to server was lost|terminating connection'; then + if grep -qE 'server closed the connection unexpectedly|connection to server was lost|terminating connection' <<<"$out"; then crashes=$((crashes + 1)) save_finding crash "$seedfile" "$mutseed" "$stmt" "$out" wait_for_cluster || echo " (cluster did not come back)" return 1 fi - if echo "$out" | grep -q '^ERROR:'; then + if grep -q '^ERROR:' <<<"$out"; then errors=$((errors + 1)) else clean=$((clean + 1)) diff --git a/test/fuzz_parquet.sh b/test/fuzz_parquet.sh index dd5762ee..2979ab0a 100755 --- a/test/fuzz_parquet.sh +++ b/test/fuzz_parquet.sh @@ -191,8 +191,8 @@ run_stmt() { rc=$? # Never let a connection failure be read as a clean result again. - if echo "$out" | grep -qE 'could not connect|No such file or directory|Connection refused' && - ! echo "$out" | grep -q '^ERROR:'; then + if grep -qE 'could not connect|No such file or directory|Connection refused' <<<"$out" && + ! grep -q '^ERROR:' <<<"$out"; then if ! wait_for_cluster; then echo " FATAL: cluster unreachable and did not return" return 1 @@ -202,14 +202,14 @@ run_stmt() { newlog="$(cat "$NEWLOG" 2>/dev/null)" # A sanitizer report is a finding even when the statement then succeeds. - if echo "$newlog" | grep -qE 'AddressSanitizer|runtime error:|UndefinedBehaviorSanitizer|LeakSanitizer'; then + if grep -qE 'AddressSanitizer|runtime error:|UndefinedBehaviorSanitizer|LeakSanitizer' <<<"$newlog"; then sanitizer=$((sanitizer + 1)) save_finding san "$seedfile" "$mutseed" "$stmt" "$newlog" wait_for_cluster || return 1 return 1 fi - if echo "$newlog" | grep -qE 'was terminated by signal|server process .* exited with|crashed'; then + if grep -qE 'was terminated by signal|server process .* exited with|crashed' <<<"$newlog"; then crashes=$((crashes + 1)) save_finding crash "$seedfile" "$mutseed" "$stmt" "$newlog" wait_for_cluster || echo " (cluster did not come back)" @@ -225,20 +225,20 @@ run_stmt() { # statement_timeout firing is a hang too: the decode did not finish in 20s # on a file that is at most a few hundred kilobytes. - if echo "$out" | grep -q 'canceling statement due to statement timeout'; then + if grep -q 'canceling statement due to statement timeout' <<<"$out"; then hangs=$((hangs + 1)) save_finding hang "$seedfile" "$mutseed" "$stmt" "$out" return 1 fi - if echo "$out" | grep -qE 'server closed the connection unexpectedly|connection to server was lost|terminating connection'; then + if grep -qE 'server closed the connection unexpectedly|connection to server was lost|terminating connection' <<<"$out"; then crashes=$((crashes + 1)) save_finding crash "$seedfile" "$mutseed" "$stmt" "$out" wait_for_cluster || echo " (cluster did not come back)" return 1 fi - if echo "$out" | grep -q '^ERROR:'; then + if grep -q '^ERROR:' <<<"$out"; then errors=$((errors + 1)) else clean=$((clean + 1)) diff --git a/test/harness_selftest.sh b/test/harness_selftest.sh index 303041e4..37e54623 100755 --- a/test/harness_selftest.sh +++ b/test/harness_selftest.sh @@ -363,6 +363,72 @@ done < <(listed_suites) check "every registered suite has a file" \ "$([ -z "$missing_file" ] && echo none || echo "missing:$missing_file")" "none" +# ---- no suite pipes a captured string into an early-exit reader (#486) ------- +# +# `echo "$s" | grep -q PATTERN` under `set -o pipefail` answers "not found" when +# the WRITER fails, whatever the string contained. The reader exits as soon as it +# has its answer, the writer takes EPIPE, and pipefail calls the pipeline failed. +# The `&&` arm never runs and the helper reports absence. +# +# This is not theoretical and it is not new here. #473 found it in this file's own +# membership test, and it came back in native_agg.sh, where it reported "the +# metadata aggregate node did not run" on PG18 CI for a plan that contained the +# node -- a red that reads exactly like a planner regression in the area #133 and +# #140 live in. The tell was a `Broken pipe` line beside a result the same run's +# summary contradicted. +# +# The failure direction is what makes it worth a rule: it always reports the +# thing you were looking for as ABSENT, which is the answer that sends someone +# looking for a defect that is not there. +# +# The control below runs first, because a rule with no demonstrated failure is a +# style preference, and this one is not. +_epipe_demo="$PGC_WORKDIR/epipe_demo.sh" +cat > "$_epipe_demo" <<'DEMO' +set -uo pipefail +big="MATCHME +$(head -c 300000 /dev/zero | tr '\0' 'y')" +piped() { echo "$1" | grep -q 'MATCHME' && echo yes || echo no; } +cased() { case "$1" in *MATCHME*) echo yes ;; *) echo no ;; esac; } +echo "piped=$(piped "$big" 2>/dev/null) cased=$(cased "$big")" +DEMO +_epipe_result="$(bash "$_epipe_demo" 2>/dev/null)" + +# The string CONTAINS the pattern, on its first line, in both arms. Only the +# answers differ. Written large on purpose: at a few kilobytes the write fits in +# the pipe buffer and completes before the reader can exit, which is why this +# shape passes almost every time and then does not. +check "control: piping a large string into grep -q reports a match as absent" \ + "$_epipe_result" "piped=no cased=yes" + +# The rule itself. A pipeline whose left side is a shell builtin writing a +# captured string, and whose right side is a reader that exits early AND whose +# EXIT STATUS is the answer being read. That last part is the whole rule: the +# damage is a wrong verdict, not a wrong message. +# +# So `grep -q` is in scope and `| head -1` inside a diagnostic string is not. +# Those exist here (analyze_stats.sh prints a plan's first line that way) and +# they can lose their pipeline's status without changing any check, because the +# substitution is used as text. They are left alone on purpose rather than +# missed; the worst they do is print to stderr. +# +# Scoped to echo and printf deliberately for the same reason. A pipeline out of +# psql or a file is a different question with a different answer, and a rule that +# flagged those too would be argued with rather than kept. +_epipe_hits="$(grep -rnE '(echo|printf)[^|]*\|[[:space:]]*grep -[a-zA-Z]*q' \ + "$TESTDIR"/*.sh 2>/dev/null | grep -v '/harness_selftest.sh:' || true)" +_epipe_count="$(printf '%s' "$_epipe_hits" | grep -c . || true)" +[ -n "$_epipe_hits" ] || _epipe_count=0 +check "no suite pipes a captured string into an early-exit reader" \ + "$_epipe_count" "0" +[ "$_epipe_count" = "0" ] || printf '%s\n' "$_epipe_hits" | sed 's/^/ /' | head -20 + +# And the scan has to be looking at something. A glob that matched nothing, or a +# TESTDIR that moved, would report zero hits and read as compliance. +_epipe_scanned="$(grep -rlE 'grep' "$TESTDIR"/*.sh 2>/dev/null | grep -c . || true)" +check "and the scan examined the suites rather than finding nothing to read" \ + "$([ "${_epipe_scanned:-0}" -ge 20 ] && echo yes || echo "no (scanned $_epipe_scanned)")" "yes" + # ---- no suite hands every run the same default port ------------------------ # #184 derived the port per run in lib.sh and in the matrix runner, which is diff --git a/test/native_agg.sh b/test/native_agg.sh index 81136df0..c68ad227 100644 --- a/test/native_agg.sh +++ b/test/native_agg.sh @@ -38,8 +38,8 @@ plan() { # The metadata aggregate path reports "Columnar Vectorized Aggregates: N" and has # no separate Aggregate node above a scan; a fallback plan has an Aggregate node # and no such line. -has_aggnode() { echo "$1" | grep -q 'Columnar Vectorized Aggregates' && echo yes || echo no; } -has_gather() { echo "$1" | grep -q 'Gather' && echo yes || echo no; } +has_aggnode() { grep -q 'Columnar Vectorized Aggregates' <<<"$1" && echo yes || echo no; } +has_gather() { grep -q 'Gather' <<<"$1" && echo yes || echo no; } # Same, with a parallel plan available: workers allowed, no setup charge, and no # minimum table size, since the suite runs with max_parallel_workers_per_gather=0 diff --git a/test/native_groupagg.sh b/test/native_groupagg.sh index 018c4721..c46c86b0 100644 --- a/test/native_groupagg.sh +++ b/test/native_groupagg.sh @@ -232,7 +232,7 @@ oc_out="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postg SET enable_hashagg=off; SET enable_sort=off; SET pgcolumnar.groupagg_max_groups=100; SELECT g, count(*) FROM t_col GROUP BY g" 2>&1 || true)" -n_err="$(printf '%s' "$oc_out" | grep -q 'groupagg_max_groups' && echo yes || echo no)" +n_err="$(grep -q 'groupagg_max_groups' <<<"$oc_out" && echo yes || echo no)" check "over-cap stops with a groupagg_max_groups error" "$n_err" yes diff_query "oracle exact: default cap runs the high-cardinality key" \ "SELECT g, count(*), sum(i4) FROM %T GROUP BY g" @@ -323,7 +323,7 @@ ov_out="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postg -d "$PGC_DB" -Atc "SET pgcolumnar.enable_group_vectorization=on; SELECT k, avg(d) FROM ov_col GROUP BY k" 2>&1 || true)" check "avg(float8) overflow errors like core" \ - "$(printf '%s' "$ov_out" | grep -qi 'out of range' && echo yes || echo no)" yes + "$(grep -qi 'out of range' <<<"$ov_out" && echo yes || echo no)" yes # ---- 8. degenerate inputs -------------------------------------------------- diff --git a/test/native_ownership.sh b/test/native_ownership.sh index dbedb554..3fbcc648 100644 --- a/test/native_ownership.sh +++ b/test/native_ownership.sh @@ -31,7 +31,7 @@ refused() { local out out="$(as_alice "SELECT pgcolumnar.$1;")" check "non-owner refused: ${1%%(*}" \ - "$(printf '%s' "$out" | grep -qi 'must be owner' && echo yes || echo "no")" "yes" + "$(grep -qi 'must be owner' <<<"$out" && echo yes || echo "no")" "yes" } refused "compact('n')" diff --git a/test/native_reclaim_cycles.sh b/test/native_reclaim_cycles.sh index 153e58c2..05e9c09e 100644 --- a/test/native_reclaim_cycles.sh +++ b/test/native_reclaim_cycles.sh @@ -51,7 +51,7 @@ for r in 1 2 3 4 5; do -d "$PGC_DB" -At -c "SELECT pgcolumnar.compact_rewrite('n', 0.02);" 2>&1)" # a healthy call returns an integer group count; the bug returned an ERROR line check "compact_rewrite cycle $r returns a count (no self-conflict)" \ - "$(printf '%s' "$rw" | grep -Eq '^[0-9]+$' && echo ok || echo "bad:$rw")" "ok" + "$(grep -Eq '^[0-9]+$' <<<"$rw" && echo ok || echo "bad:$rw")" "ok" check "parity after compact_rewrite cycle $r" "$(hash_n)" "$(hash_h)" sizes[$r]="$(fsize)" done diff --git a/test/native_truncate.sh b/test/native_truncate.sh index eae0912d..088947d8 100644 --- a/test/native_truncate.sh +++ b/test/native_truncate.sh @@ -84,7 +84,7 @@ psql_run "SELECT pgcolumnar.compact('n');" before_guard="$(size)" errout="$(raw "BEGIN; SELECT pgcolumnar.truncate('n'); ROLLBACK;")" check "truncate refused inside a transaction block" \ - "$(printf '%s' "$errout" | grep -qi 'cannot run inside a transaction block' && echo yes || echo no)" "yes" + "$(grep -qi 'cannot run inside a transaction block' <<<"$errout" && echo yes || echo no)" "yes" check "file unchanged after refused in-txn truncate" "$(size)" "$before_guard" check "parity after refused in-txn truncate" "$(hash_n)" "$(hash_h)" diff --git a/test/parallel_copy.sh b/test/parallel_copy.sh index 429da234..5c1a442e 100755 --- a/test/parallel_copy.sh +++ b/test/parallel_copy.sh @@ -190,7 +190,7 @@ check_reconstruct "$F_NONL" 4 "no trailing newline / 4 workers" err_out="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ -d "$PGC_DB" -Atc "SELECT pgcolumnar.file_split_offsets('$F', 0)" 2>&1 || true)" check "workers < 1 is rejected" \ - "$(printf '%s' "$err_out" | grep -qi "at least 1" && echo ok || echo no)" ok + "$(grep -qi "at least 1" <<<"$err_out" && echo ok || echo no)" ok # a directory is rejected, not reported as an 8-exabyte splittable file (regression # for the missing fstat/S_ISREG guard: lseek(SEEK_END) on a directory fd returns a @@ -198,7 +198,7 @@ check "workers < 1 is rejected" \ dir_err="$(env PATH="$PGC_BINDIR:$PATH" psql -h 127.0.0.1 -p "$PGC_PORT" -U postgres \ -d "$PGC_DB" -Atc "SELECT pgcolumnar.file_split_offsets('$DATADIR', 1)" 2>&1 || true)" check "file_split_offsets: a directory is rejected (not an 8-exabyte file)" \ - "$(printf '%s' "$dir_err" | grep -qi "not a regular file" && echo ok || echo no)" ok + "$(grep -qi "not a regular file" <<<"$dir_err" && echo ok || echo no)" ok # ---- coordinator: pgcolumnar.parallel_copy (partition-parallel, atomic 2PC) --- # Each worker loads a DISTINCT partition (distinct storage id -> parallel AND @@ -265,7 +265,7 @@ psql_run "DROP TABLE IF EXISTS t_txtkey CASCADE; CREATE TABLE t_txtkey_b PARTITION OF t_txtkey FOR VALUES FROM ('m') TO (MAXVALUE) USING pgcolumnar;" >/dev/null tk_err="$(err_of "SELECT pgcolumnar.parallel_copy('t_txtkey'::regclass, '$F', 2)")" check "text partition key is rejected (not a crash)" \ - "$(printf '%s' "$tk_err" | grep -qi "numeric or date/time" && echo ok || echo no)" ok + "$(grep -qi "numeric or date/time" <<<"$tk_err" && echo ok || echo no)" ok check "text partition key: server still up (no crash)" "$(q "SELECT 1")" 1 # ---- key NOT in column 1, with a generated column before it ------------------- @@ -299,7 +299,7 @@ F_BADKEY="$DATADIR/badkey.txt" mkpart t_pcp 4 bk_err="$(err_of "SELECT pgcolumnar.parallel_copy('t_pcp'::regclass, '$F_BADKEY', 4)")" check "atomic: a bad partition-key value is rejected" \ - "$(printf '%s' "$bk_err" | grep -qi 'invalid input syntax' && echo ok || echo no)" ok + "$(grep -qi 'invalid input syntax' <<<"$bk_err" && echo ok || echo no)" ok check "atomic: bad-key load leaves the target empty" "$(q "SELECT count(*) FROM t_pcp")" 0 check "atomic: no prepared-transaction leak after bad-key" "$(q "SELECT count(*) FROM pg_prepared_xacts")" 0 @@ -314,7 +314,7 @@ psql_run "DROP TABLE IF EXISTS t_gap CASCADE; CREATE TABLE t_gap_b PARTITION OF t_gap FOR VALUES FROM (3000) TO (MAXVALUE) USING pgcolumnar;" >/dev/null gap_err="$(err_of "SELECT pgcolumnar.parallel_copy('t_gap'::regclass, '$F', 4)")" check "atomic: a loader failure fails the whole load" \ - "$(printf '%s' "$gap_err" | grep -qiE 'no partition of relation|failed' && echo ok || echo no)" ok + "$(grep -qiE 'no partition of relation|failed' <<<"$gap_err" && echo ok || echo no)" ok check "atomic: loader-failure leaves the target empty (siblings rolled back)" \ "$(q "SELECT count(*) FROM t_gap")" 0 check "atomic: no prepared-transaction leak after loader failure" \ @@ -327,7 +327,7 @@ shuf "$F" > "$F_SHUF"; [ "$(id -u)" = "0" ] && chown postgres "$F_SHUF" mkpart t_pcp 4 shuf_err="$(err_of "SELECT pgcolumnar.parallel_copy('t_pcp'::regclass, '$F_SHUF', 4)")" check "unsorted input is rejected" \ - "$(printf '%s' "$shuf_err" | grep -qi "not sorted" && echo ok || echo no)" ok + "$(grep -qi "not sorted" <<<"$shuf_err" && echo ok || echo no)" ok check "unsorted rejection loads nothing" "$(q "SELECT count(*) FROM t_pcp")" 0 # ---- a DEFAULT partition is rejected (it could catch any worker's rows) ------- @@ -337,7 +337,7 @@ psql_run "DROP TABLE IF EXISTS t_def CASCADE; CREATE TABLE t_def_d PARTITION OF t_def DEFAULT USING pgcolumnar;" >/dev/null def_err="$(err_of "SELECT pgcolumnar.parallel_copy('t_def'::regclass, '$F', 4)")" check "DEFAULT partition target is rejected" \ - "$(printf '%s' "$def_err" | grep -qi "DEFAULT partition" && echo ok || echo no)" ok + "$(grep -qi "DEFAULT partition" <<<"$def_err" && echo ok || echo no)" ok # ---- the max_prepared_transactions guard fires up front ---------------------- # max_prepared_transactions is 8 (set above); a target with more partitions than @@ -345,14 +345,14 @@ check "DEFAULT partition target is rejected" \ mkpart t_pcp10 10 guard_err="$(err_of "SELECT pgcolumnar.parallel_copy('t_pcp10'::regclass, '$F', 10)")" check "max_prepared_transactions guard fires" \ - "$(printf '%s' "$guard_err" | grep -qi "max_prepared_transactions" && echo ok || echo no)" ok + "$(grep -qi "max_prepared_transactions" <<<"$guard_err" && echo ok || echo no)" ok check "guard rejects before loading anything" "$(q "SELECT count(*) FROM t_pcp10")" 0 # ---- a missing file errors cleanly ------------------------------------------- mkpart t_pcp 4 mf_err="$(err_of "SELECT pgcolumnar.parallel_copy('t_pcp'::regclass, '$DATADIR/nope.txt', 2)")" check "missing file: errors, not crashes" \ - "$(printf '%s' "$mf_err" | grep -qiE "could not (open|stat)|no such file|not a regular file" && echo ok || echo no)" ok + "$(grep -qiE "could not (open|stat)|no such file|not a regular file" <<<"$mf_err" && echo ok || echo no)" ok check "missing file: server still up (no worker crash)" "$(q "SELECT 1")" 1 check "missing file: nothing loaded" "$(q "SELECT count(*) FROM t_pcp")" 0 @@ -360,7 +360,7 @@ check "missing file: nothing loaded" "$(q "SELECT count(*) FROM t_pcp")" 0 psql_run "DROP TABLE IF EXISTS t_heaptgt; CREATE TABLE t_heaptgt (id int, txt text);" >/dev/null nc_err="$(err_of "SELECT pgcolumnar.parallel_copy('t_heaptgt'::regclass, '$F', 2)")" check "non-columnar target is rejected" \ - "$(printf '%s' "$nc_err" | grep -qi "not a pgcolumnar table" && echo ok || echo no)" ok + "$(grep -qi "not a pgcolumnar table" <<<"$nc_err" && echo ok || echo no)" ok # ---- single columnar table: workers write ONE storage concurrently ----------- # The storage-row creation lock is skipped by the loaders (the coordinator @@ -429,7 +429,7 @@ check "witness: no prepared-transaction leak" \ psql_run "DROP TABLE IF EXISTS t_single; CREATE TABLE t_single (id int, txt text) USING pgcolumnar;" >/dev/null st_bad="$(err_of "SELECT pgcolumnar.parallel_copy('t_single'::regclass, '$F_BADKEY', 4)")" check "single table: a bad row fails the whole load" \ - "$(printf '%s' "$st_bad" | grep -qiE 'invalid input syntax|failed' && echo ok || echo no)" ok + "$(grep -qiE 'invalid input syntax|failed' <<<"$st_bad" && echo ok || echo no)" ok check "single table: failed load leaves the target empty" "$(q "SELECT count(*) FROM t_single")" 0 check "single table: no prepared-transaction leak after failure" "$(q "SELECT count(*) FROM pg_prepared_xacts")" 0 diff --git a/test/parallel_export_parquet.sh b/test/parallel_export_parquet.sh index 9bbb19e6..99d21ad7 100644 --- a/test/parallel_export_parquet.sh +++ b/test/parallel_export_parquet.sh @@ -39,7 +39,7 @@ err_of() { expect_error() { local label="$1" sql="$2" out out="$(err_of "$sql")" - check "$label" "$(printf '%s' "$out" | grep -qi "ERROR" && echo error || echo ok)" error + check "$label" "$(grep -qi "ERROR" <<<"$out" && echo error || echo ok)" error } # ---- single columnar table: split by row-group ranges ----------------------- diff --git a/test/parallel_vector_agg.sh b/test/parallel_vector_agg.sh index 8ac3d6ae..c3ab30ce 100644 --- a/test/parallel_vector_agg.sh +++ b/test/parallel_vector_agg.sh @@ -48,13 +48,13 @@ q -c "DROP TABLE IF EXISTS t; # ---- premise: the parallel plan is actually chosen ------------------------- PLAN="$(q -c "$PAR $UG $PP" -c "EXPLAIN (COSTS OFF) SELECT count(*), avg(v), sum(v) FROM t WHERE k < 700")" check "premise: Finalize Aggregate present" \ - "$(printf '%s' "$PLAN" | grep -qi 'Finalize Aggregate' && echo y || echo n)" y + "$(grep -qi 'Finalize Aggregate' <<<"$PLAN" && echo y || echo n)" y check "premise: Gather present" \ - "$(printf '%s' "$PLAN" | grep -qiE 'Gather' && echo y || echo n)" y + "$(grep -qiE 'Gather' <<<"$PLAN" && echo y || echo n)" y check "premise: the partial columnar agg node present" \ - "$(printf '%s' "$PLAN" | grep -qi 'Columnar Vectorized Aggregates' && echo y || echo n)" y + "$(grep -qi 'Columnar Vectorized Aggregates' <<<"$PLAN" && echo y || echo n)" y check "premise: the partial runs the batch fold" \ - "$(printf '%s' "$PLAN" | grep -qi 'Batch Fold: yes' && echo y || echo n)" y + "$(grep -qi 'Batch Fold: yes' <<<"$PLAN" && echo y || echo n)" y # ---- values: count exact, avg/sum vs core PARALLEL agg within tolerance ----- CNT_VEC="$(q -c "$PAR $UG $PP" -c "SELECT count(*) FROM t WHERE k < 700")" @@ -79,7 +79,7 @@ check "avg(w::float8) f4 ~= core parallel agg" "$(reldiff 'avg(w)::float8')" # equal the serial oracle EXACTLY, and the plan must be the parallel fold. PLAN_I="$(q -c "$PAR $UG $PP" -c "EXPLAIN (COSTS OFF) SELECT sum(k), avg(k) FROM t WHERE k < 700")" check "premise: int sum/avg takes the parallel fold" \ - "$(printf '%s' "$PLAN_I" | grep -qiE 'Gather' && printf '%s' "$PLAN_I" | grep -qi 'Batch Fold: yes' && echo y || echo n)" y + "$(grep -qiE 'Gather' <<<"$PLAN_I" && grep -qi 'Batch Fold: yes' <<<"$PLAN_I" && echo y || echo n)" y IK_VEC="$(q -c "$PAR $UG $PP" -c "SELECT sum(k), avg(k) FROM t WHERE k < 700")" IK_SER="$(q -c "SET max_parallel_workers_per_gather=0;" -c "SELECT sum(k), avg(k) FROM t WHERE k < 700")" check "sum(k)+avg(k) int: parallel fold == serial (exact)" "$IK_VEC" "$IK_SER" @@ -169,17 +169,17 @@ GEA="$(q -c "$PAR $GVP" -c "EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF # A HashAggregate finalize, not core's GroupAggregate-over-Sort. check "premise: grouped Finalize HashAggregate present (#349)" \ - "$(printf '%s' "$GEA" | grep -qi 'Finalize HashAggregate' && echo y || echo n)" y + "$(grep -qi 'Finalize HashAggregate' <<<"$GEA" && echo y || echo n)" y # OUR grouped node, and it is the parallel-aware one. check "premise: the partial grouped node is under the Gather (#349)" \ - "$(printf '%s' "$GEA" | grep -qi 'Parallel Custom Scan (PgColumnarScan)' && - printf '%s' "$GEA" | grep -qi 'Columnar Vectorized Group Keys' && echo y || echo n)" y + "$(grep -qi 'Parallel Custom Scan (PgColumnarScan)' <<<"$GEA" && + grep -qi 'Columnar Vectorized Group Keys' <<<"$GEA" && echo y || echo n)" y # Planned is not launched: a leader-only run would satisfy every value check # below while never exercising a worker. Assert launch AND our node together, so # core's parallel plan cannot satisfy this on its own. check "premise: workers actually launched for OUR grouped node (#349)" \ - "$(printf '%s' "$GEA" | grep -qiE 'Workers Launched: [1-9]' && - printf '%s' "$GEA" | grep -qi 'Columnar Vectorized Group Keys' && echo y || echo n)" y + "$(grep -qiE 'Workers Launched: [1-9]' <<<"$GEA" && + grep -qi 'Columnar Vectorized Group Keys' <<<"$GEA" && echo y || echo n)" y # Each participant emits its own partial per group, so the Gather carries a # MULTIPLE of the group count and the Finalize collapses it back. 50 groups with diff --git a/test/phase4.sh b/test/phase4.sh index 6fd60026..83222d74 100755 --- a/test/phase4.sh +++ b/test/phase4.sh @@ -94,7 +94,7 @@ assert_plan() { # be costed below the index scan; turning it off makes the planner pick the # index scan so the plan shape can be asserted. plan="$(run_pg "$PSQL -c \"SET enable_seqscan=off; SET pgcolumnar.enable_custom_scan=off; EXPLAIN (COSTS OFF) $sql\"")" - if echo "$plan" | grep -q "$want" && ! echo "$plan" | grep -q "$notwant"; then + if grep -q "$want" <<<"$plan" && ! grep -q "$notwant" <<<"$plan"; then echo "PASS $name: $(echo "$plan" | grep -E 'Scan' | head -1 | sed 's/^ *//')" else echo "FAIL $name: plan was:" @@ -222,7 +222,7 @@ q "CREATE INDEX ios_a_idx ON ios (a);" >/dev/null # With index-only scans disabled, a covering query falls back to an Index Scan. # The custom scan is turned off so the planner picks the index scan (see assert_plan). iosoff_plan="$(run_pg "$PSQL -c \"SET pgcolumnar.enable_index_only_scan=off; SET enable_seqscan=off; SET pgcolumnar.enable_custom_scan=off; EXPLAIN (COSTS OFF) SELECT a FROM ios WHERE a = 100;\"")" -if echo "$iosoff_plan" | grep -q "Index Scan" && ! echo "$iosoff_plan" | grep -q "Index Only Scan"; then +if grep -q "Index Scan" <<<"$iosoff_plan" && ! grep -q "Index Only Scan" <<<"$iosoff_plan"; then echo "PASS IOS off: plain index scan" else echo "FAIL IOS off: plain index scan: plan was:"; echo "$iosoff_plan" | sed 's/^/ /'; fail=1 @@ -234,7 +234,7 @@ check "covering value" "$(q 'SET enable_seqscan=off; SELECT a FROM ios WHERE assert_plan_seq() { local plan plan="$(run_pg "$PSQL -c \"SET enable_indexscan=off; SET enable_bitmapscan=off; EXPLAIN (COSTS OFF) SELECT * FROM ios WHERE a = 100;\"")" - if echo "$plan" | grep -qE "Seq Scan|Custom Scan \(PgColumnarScan\)"; then + if grep -qE "Seq Scan|Custom Scan \(PgColumnarScan\)" <<<"$plan"; then echo "PASS full-table scan available" else echo "FAIL full-table scan available: $plan"; fail=1 diff --git a/test/phase5.sh b/test/phase5.sh index 4e839c85..99b11b4b 100755 --- a/test/phase5.sh +++ b/test/phase5.sh @@ -92,7 +92,7 @@ assert_plan() { local name="$1" sql="$2" want="$3" notwant="${4:-}" local plan plan="$(run_pg "$PSQL -c \"$sql\"")" - if echo "$plan" | grep -q "$want" && { [ -z "$notwant" ] || ! echo "$plan" | grep -q "$notwant"; }; then + if grep -q "$want" <<<"$plan" && { [ -z "$notwant" ] || ! grep -q "$notwant" <<<"$plan"; }; then echo "PASS $name" else echo "FAIL $name: plan was:" diff --git a/test/pushdown_report.sh b/test/pushdown_report.sh index 0954d636..e9892b22 100755 --- a/test/pushdown_report.sh +++ b/test/pushdown_report.sh @@ -50,7 +50,7 @@ field() { echo "$1" | grep -oE "$2: [0-9]+" | head -1 | grep -oE '[0-9]+$'; } # positive grep for the node's marker is a stronger test than an absence test, # because a plan that fell back to a seq scan has no Columnar lines to be absent. is_scalar_scan() { - echo "$1" | grep -q 'Columnar Projected Columns' && echo yes || echo no + grep -q 'Columnar Projected Columns' <<<"$1" && echo yes || echo no } on="$(plan on)" @@ -236,7 +236,7 @@ aggplan() { # aggplan SET $1 = on; EXPLAIN (ANALYZE, COSTS OFF, TIMING OFF, SUMMARY OFF) $2;" } -has() { echo "$1" | grep -q "$2" && echo yes || echo no; } +has() { grep -q "$2" <<<"$1" && echo yes || echo no; } UAGG=pgcolumnar.enable_ungrouped_vector_agg u_usable="$(aggplan $UAGG "SELECT count(*), sum(plain) FROM pdr_u WHERE plain > $((ROWS - 10000))")" diff --git a/test/run_san.sh b/test/run_san.sh index a669b2d5..ccd3e5ba 100644 --- a/test/run_san.sh +++ b/test/run_san.sh @@ -98,7 +98,7 @@ for s in $SUITES; do # suite's own crash checks or a nonzero rc), or the text appears in the output. san="$(printf '%s' "$out" | grep -icE 'runtime error:|AddressSanitizer|UndefinedBehaviorSanitizer|SUMMARY: .*Sanitizer|terminated by signal 6')" if [ "$rc" = 66 ] && [ "$san" = 0 ] && \ - printf '%s' "$out" | grep -q 'SKIPPED (ran no checks)'; then + grep -q 'SKIPPED (ran no checks)' <<<"$out"; then # pgc_summary's skipped state: the suite ran no checks. # # This does NOT cover the pyarrow-gated suites, and an earlier version of diff --git a/test/server_file_privilege.sh b/test/server_file_privilege.sh index 7f72d6a4..f08a170d 100644 --- a/test/server_file_privilege.sh +++ b/test/server_file_privilege.sh @@ -95,13 +95,13 @@ for e in "${entry_points[@]}"; do # 1. no role -> refused, and the error names the role (reached the gate) reply="$(run_as t_none "$sql")" check "no role is refused by the $role gate: $label" \ - "$(printf '%s' "$reply" | grep -qi "$role" && echo yes || echo no)" "yes" + "$(grep -qi "$role" <<<"$reply" && echo yes || echo no)" "yes" # 2. with the role -> past the gate (no role name in the output) rm -rf "$out".parquet "$out".arrow "$outdir" 2>/dev/null || true preply="$(run_as t_priv "$sql")" check "the $role role gets past the gate: $label" \ - "$(printf '%s' "$preply" | grep -qi "$role" && echo yes || echo no)" "no" + "$(grep -qi "$role" <<<"$preply" && echo yes || echo no)" "no" done # coverage: every SQL function that declares a file-path argument must be listed diff --git a/test/ungrouped_vector_agg.sh b/test/ungrouped_vector_agg.sh index bdec60bb..48dbe1d0 100644 --- a/test/ungrouped_vector_agg.sh +++ b/test/ungrouped_vector_agg.sh @@ -65,18 +65,18 @@ psql_run "DROP TABLE IF EXISTS t; P_ON="$(plan on "SELECT count(*), sum(v), avg(v) FROM t WHERE v > 0")" P_OFF="$(plan off "SELECT count(*), sum(v), avg(v) FROM t WHERE v > 0")" check "premise: vectorized agg node used when GUC on" \ - "$(printf '%s' "$P_ON" | grep -qi 'Columnar Vectorized Aggregates' && echo yes || echo no)" yes + "$(grep -qi 'Columnar Vectorized Aggregates' <<<"$P_ON" && echo yes || echo no)" yes check "premise: core Agg (no vectorized node) when GUC off" \ - "$(printf '%s' "$P_OFF" | grep -qi 'Columnar Vectorized Aggregates' && echo yes || echo no)" no + "$(grep -qi 'Columnar Vectorized Aggregates' <<<"$P_OFF" && echo yes || echo no)" no check "premise: the filter is pushed into the scan (EXPLAIN shows it)" \ - "$(printf '%s' "$P_ON" | grep -qi 'Columnar Pushed-Down Filters' && echo yes || echo no)" yes + "$(grep -qi 'Columnar Pushed-Down Filters' <<<"$P_ON" && echo yes || echo no)" yes # batch fold: an all-eligible shape (float sum/avg/count + numeric filter) folds # column-at-a-time; an ineligible aggregate (min/max) falls back to the row path. check "premise: eligible shape uses the batch fold" \ - "$(printf '%s' "$P_ON" | grep -qi 'Batch Fold: yes' && echo yes || echo no)" yes + "$(grep -qi 'Batch Fold: yes' <<<"$P_ON" && echo yes || echo no)" yes P_MM="$(plan on "SELECT min(v), max(v) FROM t WHERE k > 5")" check "premise: min/max falls back off the batch fold" \ - "$(printf '%s' "$P_MM" | grep -qi 'Batch Fold: no' && echo yes || echo no)" yes + "$(grep -qi 'Batch Fold: no' <<<"$P_MM" && echo yes || echo no)" yes # ---- filtered aggregates over each type (the q6 shape) ----------------------- ab "filtered float sum" "SELECT sum(v)::text FROM t WHERE k > 500" diff --git a/test/unique_conc.sh b/test/unique_conc.sh index 40e04899..ead78fb4 100755 --- a/test/unique_conc.sh +++ b/test/unique_conc.sh @@ -528,7 +528,7 @@ ctl_q "CREATE TABLE s_ss (k int) USING pgcolumnar;" >/dev/null ctl_q "CREATE UNIQUE INDEX s_ss_uidx ON s_ss (k);" >/dev/null ss_err="$(run_pg "$SPSQL -c \"INSERT INTO s_ss SELECT 7 FROM generate_series(1,2);\" 2>&1" )" check "7 same-statement duplicate raises unique_violation" \ - "$(echo "$ss_err" | grep -qF 's_ss_uidx' && echo yes || echo no)" "yes" + "$(grep -qF 's_ss_uidx' <<<"$ss_err" && echo yes || echo no)" "yes" check "7 same-statement duplicate inserted no rows" \ "$(ctl_q "SELECT count(*) FROM s_ss;")" "0"