From f8dc78e74bf928d8835dc2679e93ec2aa116b834 Mon Sep 17 00:00:00 2001 From: Conduction Release Bot Date: Sun, 16 Aug 2026 17:56:51 +0200 Subject: [PATCH 1/2] fix(invariants): complete the #477 demotion in the two suites it missed, and bound it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `.github@main`'s `Package invariants` job has been red since 13:15 today. Both failures come from #478 (`e1fe8a4`, gate-19 demoted to advisory), and NEITHER of them is the empty-scope property itself — that property still holds and is still asserted. WHICH PR, SETTLED BY RUNNING THE TESTS RATHER THAN READING CI #476's CI run was CANCELLED (concurrency `cancel-in-progress`, #478 pushed 7s later), so it was neither passed nor failed and could not be exonerated from CI. The two suites were run directly against each merge commit's tree instead: ff66c3b pre-#476 both suites GREEN 0b2c457 #476 merged (gate-35/25) both suites GREEN <- exonerated 530a849 #478 merged (gate-19) both suites RED, 1 assertion each 6f756a8 main (#479 merged) both suites RED, same two assertions WHAT EACH ASSERTION ASSERTED, AND WHAT IT GOT 1. test_gate_empty_scope_never_passes.sh, ARM 1 asserted: `[gate-19] ...: FAIL` over a fixture carrying one planted unannotated scenario. got: `[gate-19] e2e-coverage: WARNING — 1 scenario(s) missing @e2e`. ARM 1 is the POSITIVE CONTROL for the fixture — everything after it is meaningless unless the gate demonstrably sees the planted defect. #477 deliberately moved gate-19's verdict word, not its detection. The arm matched the word rather than the property. 2. test_gate_discarded_counts_and_empty_deltas.sh, ARM P3 asserted: the literal `(PASS|FAIL|NOT APPLICABLE|SKIPPED)` appears in BOTH gate_fixture_support.sh and test_gate_acceptance_matrix.sh. got: both files now read `(PASS|FAIL|WARNING|NOT APPLICABLE|SKIPPED)` — changed IN STEP by #478, exactly as their comments require. The stated property is "the two parsers must stay in step". The implemented mechanism froze the vocabulary instead, so a correct in-step change failed it, and its own repair path was to retype a literal nobody can check against the files it polices. WHICH SIDE WAS WRONG The tests, on both counts — but only in mechanism, and the repair strengthens rather than relaxes: - ARM 1 now accepts FAIL *or* WARNING for gate-19 AND requires the line to name a non-zero count, because a demotion's real failure mode is the finding ceasing to be read. gate-25 is not demoted and keeps the strict FAIL. PASS and NOT APPLICABLE are still refused, and ARM 2 still requires NOT APPLICABLE over the empty scope, so the two remain distinguishable. - A new assertion requires the run SUMMARY to announce the advisory findings and to say that green means "nothing BLOCKING failed". - ARM P3 now extracts the alternation from each parser and requires them to be byte-identical, with the four core verdict forms as a floor so "agreement" cannot be reached by gutting both. - NEW ARM P4: the set of gates allowed to call `_warn` is an explicit allowlist (`19`, .github#477). `_warn` is otherwise one line away from being a soft-fail switch for any gate in the suite, and nothing in this package would have noticed a second one. This does not stop a demotion; it stops a silent one. The empty-scope property is untouched. ARM 2 (`NOT APPLICABLE`, never PASS) and ARM 6 (`no gate reports PASS over a diff that excludes every planted defect`, positive control: 18 gates FAIL the planted tree at full scope) pass unchanged before and after. POSITIVE CONTROLS — each mutation run against an unmodified copy of the package gate-19 `_warn` -> `_pass 19` ARM 1 FAILS: "got: PASS" gate-19 warning with the count removed ARM 1 FAILS: "named no non-zero count" WARNING added to ONE parser only ARM P3 FAILS, naming both alternations gate-18's advisory turned into `_warn` ARM P4 FAILS: "found '18 19', allowed '19'" Both suites also carry the drift/allowlist controls inline, so a future extractor that returns empty for everything cannot call that agreement. The assertion floor in the discarded-counts suite moves 34 -> 37 to match. NOT DONE, DELIBERATELY #478's demotion of gate-19 is a fleet policy decision with a stated reason and a tracking issue; reverting it is the coordinator's call, not this PR's. gate-18's advisory line is printed AFTER its verdict, so `head -1` parsers are safe today — but it is now shape-matchable as a verdict, and that safety is an ordering rather than a mechanism. --- ..._gate_discarded_counts_and_empty_deltas.sh | 107 ++++++++++++++++-- .../lib/test_gate_empty_scope_never_passes.sh | 67 ++++++++++- 2 files changed, 162 insertions(+), 12 deletions(-) diff --git a/hydra-gates/scripts/lib/test_gate_discarded_counts_and_empty_deltas.sh b/hydra-gates/scripts/lib/test_gate_discarded_counts_and_empty_deltas.sh index e67173eb..d8f04de7 100644 --- a/hydra-gates/scripts/lib/test_gate_discarded_counts_and_empty_deltas.sh +++ b/hydra-gates/scripts/lib/test_gate_discarded_counts_and_empty_deltas.sh @@ -706,21 +706,114 @@ esac # repo-shaped suites; the acceptance matrix has its own copy, and that copy is # the one that filtered NOTHING. A comment saying "keep these in step" is not a # mechanism; this is. +# +# ⚠️ THE FIRST MECHANISM WAS THE WRONG SHAPE, AND IT SAID SO OUT LOUD. +# It asserted that the literal string `(PASS|FAIL|NOT APPLICABLE|SKIPPED)` +# appeared in BOTH files. That is not "the parsers agree" — it is "the verdict +# vocabulary is frozen at these four words". `.github#477` added WARNING to +# both files, IN STEP, exactly as their comments require, and this assertion +# went red anyway; it was one of the two that reddened `.github@main` from +# 13:15 on 2026-08-16. Worse, its own repair path was to retype the literal in +# the test, which nobody can check against the files it is supposed to police. +# +# So the property is asserted directly instead: EXTRACT the alternation each +# parser actually uses and require the two to be byte-identical. Vocabulary may +# grow — it just cannot grow in one file only. The shape half of the property +# is kept as a floor: whatever the alternation contains, it must still contain +# the four core verdict forms, so nobody can satisfy "identical" by gutting +# both parsers back to a denylist. _matrix="${GF_PKG_ROOT}/scripts/lib/test_gate_acceptance_matrix.sh" -_shape='(PASS|FAIL|NOT APPLICABLE|SKIPPED)' -if grep -qF -- "${_shape}" "${GF_PKG_ROOT}/scripts/lib/gate_fixture_support.sh" \ - && grep -qF -- "${_shape}" "${_matrix}"; then - _ok "both verdict parsers match by shape — gate_fixture_support.sh and the acceptance matrix agree" +_support="${GF_PKG_ROOT}/scripts/lib/gate_fixture_support.sh" + +# `\((PASS|FAIL)…\)` — the parenthesised alternation a verdict matcher uses. +# Anchored on PASS|FAIL so it cannot latch onto some unrelated group. +_verdict_alternation() { # -> the alternation, or empty + grep -oE '\(PASS\|FAIL[A-Z| ]*\)' "$1" | head -1 +} +_alt_support="$(_verdict_alternation "${_support}")" +_alt_matrix="$(_verdict_alternation "${_matrix}")" + +if [ -z "${_alt_support}" ] || [ -z "${_alt_matrix}" ]; then + _bad "a verdict parser no longer carries a recognisable \`(PASS|FAIL|…)\` alternation — support='${_alt_support:-none}' matrix='${_alt_matrix:-none}'. A parser this suite cannot read is a parser it cannot police." +elif [ "${_alt_support}" != "${_alt_matrix}" ]; then + _bad "the two verdict parsers have drifted apart: gate_fixture_support.sh matches ${_alt_support} and the acceptance matrix matches ${_alt_matrix}, so the same output would be graded differently by two suites" +else + _missing_form="" + for _form in 'PASS' 'FAIL' 'NOT APPLICABLE' 'SKIPPED'; do + printf '%s' "${_alt_support}" | grep -qF -- "${_form}" \ + || _missing_form="${_missing_form}${_form}, " + done + if [ -n "${_missing_form}" ]; then + _bad "both verdict parsers agree, but the shared alternation ${_alt_support} no longer covers ${_missing_form%, } — agreement satisfied by narrowing both is the .github#401 defect in a new coat" + else + _ok "both verdict parsers match by the SAME shape — gate_fixture_support.sh and the acceptance matrix agree on ${_alt_support}" + fi +fi + +# POSITIVE CONTROL for the assertion above. Without it, "extract and compare" +# is satisfiable by an extractor that returns empty for everything and calls +# that agreement. Drift a COPY of one parser by one word and require the same +# check to say NO. +_pdrift="${_tmp}/parser-drift"; mkdir -p "${_pdrift}" +sed 's/(PASS|FAIL/(PASS|FAIL|BOGUS/' "${_matrix}" > "${_pdrift}/matrix.sh" +_alt_drifted="$(_verdict_alternation "${_pdrift}/matrix.sh")" +if [ -n "${_alt_drifted}" ] && [ "${_alt_drifted}" != "${_alt_support}" ]; then + _ok "control: the drift check FAILS a parser changed on one side only (${_alt_drifted} vs ${_alt_support})" +else + _bad "control BROKEN: a one-sided parser change was not detected (drifted='${_alt_drifted:-none}', support='${_alt_support}') — the arm above proves nothing" +fi + +# --------------------------------------------------------------------------- +# ARM P4 — WHICH GATES MAY BE ADVISORY IS BOUNDED BY NAME. +# +# `_warn` (added by .github#477) is one line away from being a soft-fail switch +# for any gate in the suite: a gate that reports a real finding and does not +# reach `_FAILED`. That is the invisible pass, in the package whose entire job +# is preventing them — and nothing else in this package would notice a second +# one landing. +# +# So the allowlist lives here, and demoting another gate means editing this +# test and stating why. This does not stop a demotion; it stops a SILENT one. +# +# ⚠️ Note this counts `_warn` CALL SITES, not the word WARNING. gate-18 prints +# a long-standing advisory line that is not a demotion of its verdict. +# --------------------------------------------------------------------------- +# gate -> the reason it is permitted to be non-blocking. +_ADVISORY_ALLOWED_GATES="19" # e2e-coverage — .github#477, temporary, owned. + +_runner_src="${GF_PKG_ROOT}/scripts/run-hydra-gates.sh" +_warn_gates="$(grep -oE '^[[:space:]]*_warn[[:space:]]+[0-9]+' "${_runner_src}" \ + | grep -oE '[0-9]+$' | sort -un | tr '\n' ' ')" +_warn_gates="${_warn_gates% }" +if [ "${_warn_gates}" = "${_ADVISORY_ALLOWED_GATES}" ]; then + _ok "the advisory (_warn) gates are exactly the allowlisted ones: ${_ADVISORY_ALLOWED_GATES:-none}" +else + _bad "the set of gates demoted to advisory has changed without this test being told: found '${_warn_gates:-none}', allowed '${_ADVISORY_ALLOWED_GATES:-none}'. A gate that reports a real finding and does not block is a soft-fail; if the demotion is intended, add it here WITH its tracking issue." +fi + +# POSITIVE CONTROL — the allowlist check must be able to say NO. Plant a second +# `_warn` call site in a COPY of the runner and require the same extraction to +# come back with a set the allowlist rejects. +_wdrift="${_tmp}/warn-drift"; mkdir -p "${_wdrift}" +{ cat "${_runner_src}"; printf ' _warn 7 "no-admin-idor" "pretend"\n'; } > "${_wdrift}/runner.sh" +_warn_drifted="$(grep -oE '^[[:space:]]*_warn[[:space:]]+[0-9]+' "${_wdrift}/runner.sh" \ + | grep -oE '[0-9]+$' | sort -un | tr '\n' ' ')" +_warn_drifted="${_warn_drifted% }" +if [ "${_warn_drifted}" != "${_ADVISORY_ALLOWED_GATES}" ] && [ -n "${_warn_drifted}" ]; then + _ok "control: a second _warn call site is detected (${_warn_drifted}) — the allowlist can refuse" else - _bad "the two verdict parsers have drifted apart: one matches a verdict by shape and the other does not, so the same output would be graded differently by two suites" + _bad "control BROKEN: planting a _warn on gate-7 produced '${_warn_drifted:-none}', which the allowlist would accept — ARM P4 proves nothing" fi echo echo "== summary ==" echo " assertions: ${_asserts}" echo " failures: ${_failures}" -if [ "${_asserts}" -lt 34 ]; then - echo "FAIL — only ${_asserts} assertions ran; this suite declares 34+. A short run is not a green run." +# 34 -> 37: ARM P3 gained its positive control, and ARM P4 (the advisory-gate +# allowlist) plus its control are new. The floor tracks the suite, otherwise a +# whole arm can be deleted and the run still reads green. +if [ "${_asserts}" -lt 37 ]; then + echo "FAIL — only ${_asserts} assertions ran; this suite declares 37+. A short run is not a green run." exit 1 fi [ "${_failures}" -eq 0 ] || exit 1 diff --git a/hydra-gates/scripts/lib/test_gate_empty_scope_never_passes.sh b/hydra-gates/scripts/lib/test_gate_empty_scope_never_passes.sh index 96d624b8..0f033081 100755 --- a/hydra-gates/scripts/lib/test_gate_empty_scope_never_passes.sh +++ b/hydra-gates/scripts/lib/test_gate_empty_scope_never_passes.sh @@ -133,16 +133,73 @@ _verdict() { grep -oE "^\[gate-$2\] [^:]+: [A-Z]+( [A-Z]+)*( \([a-z]+\))?" "$1" # --------------------------------------------------------------------------- # ARM 1 — the planted true positives are still caught, full-tree. +# +# ⚠️ WHAT THIS ARM ASSERTS, AND WHAT IT DELIBERATELY DOES NOT (.github#477) +# ------------------------------------------------------------------------ +# This arm is the POSITIVE CONTROL for the fixture, not a statement about +# blocking policy. Everything below it — "an empty scope is NOT APPLICABLE" — +# proves nothing unless the gate demonstrably SEES the planted defect when the +# scope is open. So what has to hold is: the gate opened the scope, computed a +# real finding, and named a count. +# +# `.github#477` demoted gate-19 (e2e-coverage) from `_fail` to `_warn`: it +# still runs, still counts and still names every unannotated scenario, but the +# finding no longer stops a merge. That moved gate-19's verdict word from FAIL +# to WARNING and this arm — which matched the word rather than the property — +# went red on `.github@main` from 13:15 on 2026-08-16, alongside ARM P3 of +# test_gate_discarded_counts_and_empty_deltas.sh. Three sibling suites were +# taught the new word in the same commit (exit-code-semantics, +# base-ref-delivery-channel, the acceptance matrix + its expect.conf); these +# two were not, because they were not run. That is the hand-maintained-list +# decay this package's own runner exists to prevent — see +# tests/run-helper-suites.sh. +# +# So gate-19 is accepted at FAIL *or* WARNING, and NOT on the word alone: the +# verdict line must also carry a non-zero finding count, because a demotion's +# real failure mode is the finding quietly ceasing to be read. gate-25 is NOT +# demoted and keeps the strict FAIL — accepting WARNING for a gate nobody +# demoted would be exactly the invisible pass this file guards. +# +# Note what this arm still catches for gate-19 without the word FAIL: a gate +# that went blind prints PASS, and a gate that self-scoped to nothing prints +# NOT APPLICABLE. Neither is accepted here, and ARM 2 separately requires +# NOT APPLICABLE over the empty scope, so the two arms remain distinguishable. +# +# Which gates may be advisory at all is bounded by name — see ARM P4 of +# test_gate_discarded_counts_and_empty_deltas.sh, so a second demotion cannot +# land without reddening this package. # --------------------------------------------------------------------------- _full="${_tmp}/full.txt" _run "${_full}" -for _g in 19 25; do - if grep -qE "^\[gate-${_g}\][^:]*: FAIL" "${_full}"; then - _ok "gate-${_g} still catches its planted true positive over the full tree" + +# gate-25 — NOT demoted. Strict FAIL. +if grep -qE "^\[gate-25\][^:]*: FAIL" "${_full}"; then + _ok "gate-25 still catches its planted true positive over the full tree" +else + _bad "gate-25 did NOT catch its planted true positive — got: $(_verdict "${_full}" 25)" +fi + +# gate-19 — advisory since .github#477. FAIL or WARNING, and it must NAME a count. +_g19_line="$(grep -E "^\[gate-19\][^:]*: (FAIL|WARNING)\b" "${_full}" | head -1)" +if [ -n "${_g19_line}" ]; then + if printf '%s' "${_g19_line}" | grep -qE '[1-9][0-9]* scenario'; then + _ok "gate-19 still catches its planted true positive over the full tree, and names the count — ${_g19_line#*: }" else - _bad "gate-${_g} did NOT catch its planted true positive — got: $(_verdict "${_full}" "${_g}")" + _bad "gate-19 reached a FAIL/WARNING verdict but named no non-zero scenario count — a demotion whose finding stops being legible is the invisible pass. Got: ${_g19_line}" fi -done +else + _bad "gate-19 did NOT catch its planted true positive — got: $(_verdict "${_full}" 19)" +fi + +# …and the SUMMARY must say so. A demotion is only safe while the reader is +# told that a green verdict means "nothing BLOCKING failed", not "nothing was +# found". Without this, `_warn` degrades into a silent soft-fail. +if grep -qF 'reported ADVISORY findings' "${_full}" \ + && grep -qF "means 'nothing BLOCKING failed'" "${_full}"; then + _ok "the run's summary announces the advisory findings and says green != 'nothing was found'" +else + _bad "a gate reported an advisory finding and the summary did not say so — a demoted gate whose finding is not announced is a soft-fail nobody reads" +fi # Full-tree must actually OPEN the manifests rather than diff-scope itself to # nothing: 62/63 are clean in this fixture, so they must PASS, not SKIP. From 6c11baa06733153152d5ea58a2db4c56c9e88dae Mon Sep 17 00:00:00 2001 From: Conduction Release Bot Date: Sun, 16 Aug 2026 18:04:19 +0200 Subject: [PATCH 2/2] fix(invariants): the THIRD verdict parser was never taught WARNING, and it fails silently MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit There are three verdict parsers in this package, not two. #477 taught `gf_verdict` and the acceptance matrix the new WARNING form. It left `_verdict_set` in test_gate_base_ref_delivery_channel.sh — the one that decides "every gate returned the same verdict through both channels" — matching only (PASS|FAIL|NOT APPLICABLE|SKIPPED). Nothing went red, and that is the defect. A verdict word this pattern does not know is DROPPED, not flagged. So gate-19 fell out of both channels' verdict sets, the comparison went on matching over a set that no longer contained the gate that suite exists to watch, and it reported 12 passed / 0 failed. MEASURED, on the real fixture: main @ 6f756a8b "every gate returned the same verdict ... (69 gate(s) compared)" this branch "every gate returned the same verdict ... (70 gate(s) compared)" POSITIVE CONTROL — an A/B against a REAL leak, not an argument. A copy of the runner was made to inflate gate-19's count by one when the base arrived through the environment rather than through --base, i.e. exactly the .github#416 defect this suite was built to catch, planted in gate-19: suite as on main today PASS — 12 passed, 0 failed, "(69 gate(s) compared)" suite with this commit FAIL — "THE DELIVERY CHANNEL CHANGED THE VERDICT" The main-today arm prints both arms' differing counts (`WARNING — 1` vs `WARNING — 2`) two lines above the assertion that then declares them identical. Both this and the ordinary case are now mechanised: ARM P3 of test_gate_discarded_counts_and_empty_deltas.sh compares ALL THREE parsers by name and requires one shared alternation, so a fourth parser or a fourth verdict word cannot be added to some of them. --- .../test_gate_base_ref_delivery_channel.sh | 14 +++++- ..._gate_discarded_counts_and_empty_deltas.sh | 48 ++++++++++++++----- 2 files changed, 48 insertions(+), 14 deletions(-) diff --git a/hydra-gates/scripts/lib/test_gate_base_ref_delivery_channel.sh b/hydra-gates/scripts/lib/test_gate_base_ref_delivery_channel.sh index 9d7ab839..2b7f86ba 100755 --- a/hydra-gates/scripts/lib/test_gate_base_ref_delivery_channel.sh +++ b/hydra-gates/scripts/lib/test_gate_base_ref_delivery_channel.sh @@ -103,10 +103,20 @@ trap 'rm -rf "${WORK}"' EXIT # EXTRACT ON THE VERDICT SHAPE, NEVER BY EXCLUDING WHAT LOOKS LIKE A PASS: a # FAIL line whose remedial prose contains the word "pass" was silently dropped # by a `grep -v PASS` elsewhere in this programme on 2026-08-12. +# +# ⚠️ WARNING BELONGS IN THIS SET (.github#477, added late). +# When gate-19 was demoted to advisory, WARNING was added to the package's two +# OTHER verdict parsers and not to this one. Nothing went red: a verdict word +# this pattern does not know is simply DROPPED, so gate-19 vanished from both +# channels' sets and the comparison went on matching — over a set that no +# longer contained the gate this whole suite was built to watch. A parser that +# narrows silently is the same defect as two parsers that disagree loudly, and +# only the loud one has ever been caught. This set is asserted identical to the +# other two by ARM P3 of test_gate_discarded_counts_and_empty_deltas.sh. _verdict_set() { printf '%s\n' "$1" \ - | grep -E '^\[gate-[0-9]+\] [a-z0-9-]+: (PASS|FAIL|NOT APPLICABLE|SKIPPED)' \ - | sed -E 's/^\[gate-([0-9]+)\] [a-z0-9-]+: (PASS|FAIL — [0-9]+|FAIL|NOT APPLICABLE|SKIPPED).*/\1|\2/' \ + | grep -E '^\[gate-[0-9]+\] [a-z0-9-]+: (PASS|FAIL|WARNING|NOT APPLICABLE|SKIPPED)' \ + | sed -E 's/^\[gate-([0-9]+)\] [a-z0-9-]+: (PASS|FAIL — [0-9]+|FAIL|WARNING — [0-9]+|WARNING|NOT APPLICABLE|SKIPPED).*/\1|\2/' \ | sort -t'|' -k1,1n -u } diff --git a/hydra-gates/scripts/lib/test_gate_discarded_counts_and_empty_deltas.sh b/hydra-gates/scripts/lib/test_gate_discarded_counts_and_empty_deltas.sh index d8f04de7..42ae54ec 100644 --- a/hydra-gates/scripts/lib/test_gate_discarded_counts_and_empty_deltas.sh +++ b/hydra-gates/scripts/lib/test_gate_discarded_counts_and_empty_deltas.sh @@ -702,7 +702,7 @@ case "${_pv2}" in _bad "gf_verdict returned gate-53's PRE-EXISTING advisory instead of the verdict. Got: ${_pv2}" ;; esac -# ARM P3 — THE TWO PARSERS MUST STAY IN STEP. `gf_verdict` is used by the +# ARM P3 — EVERY VERDICT PARSER MUST STAY IN STEP. `gf_verdict` is used by the # repo-shaped suites; the acceptance matrix has its own copy, and that copy is # the one that filtered NOTHING. A comment saying "keep these in step" is not a # mechanism; this is. @@ -722,33 +722,57 @@ esac # is kept as a floor: whatever the alternation contains, it must still contain # the four core verdict forms, so nobody can satisfy "identical" by gutting # both parsers back to a denylist. -_matrix="${GF_PKG_ROOT}/scripts/lib/test_gate_acceptance_matrix.sh" -_support="${GF_PKG_ROOT}/scripts/lib/gate_fixture_support.sh" +# ⚠️ THERE ARE THREE PARSERS, NOT TWO. `.github#477` taught two of them the new +# WARNING form and left `_verdict_set` in test_gate_base_ref_delivery_channel.sh +# behind — and THAT one failed silently, because a verdict word the pattern does +# not know is simply dropped: gate-19 disappeared from both channels' verdict +# sets and the comparison went on matching over a set that no longer contained +# the gate that suite exists to watch. A parser that narrows quietly is worse +# than two that disagree loudly. All three are compared here, by name. +_PARSER_FILES=" +gate_fixture_support.sh +test_gate_acceptance_matrix.sh +test_gate_base_ref_delivery_channel.sh +" # `\((PASS|FAIL)…\)` — the parenthesised alternation a verdict matcher uses. # Anchored on PASS|FAIL so it cannot latch onto some unrelated group. _verdict_alternation() { # -> the alternation, or empty grep -oE '\(PASS\|FAIL[A-Z| ]*\)' "$1" | head -1 } -_alt_support="$(_verdict_alternation "${_support}")" -_alt_matrix="$(_verdict_alternation "${_matrix}")" -if [ -z "${_alt_support}" ] || [ -z "${_alt_matrix}" ]; then - _bad "a verdict parser no longer carries a recognisable \`(PASS|FAIL|…)\` alternation — support='${_alt_support:-none}' matrix='${_alt_matrix:-none}'. A parser this suite cannot read is a parser it cannot police." -elif [ "${_alt_support}" != "${_alt_matrix}" ]; then - _bad "the two verdict parsers have drifted apart: gate_fixture_support.sh matches ${_alt_support} and the acceptance matrix matches ${_alt_matrix}, so the same output would be graded differently by two suites" +_alt_ref=""; _alt_ref_file=""; _alt_unreadable=""; _alt_drift="" +for _pf in ${_PARSER_FILES}; do + _alt="$(_verdict_alternation "${GF_PKG_ROOT}/scripts/lib/${_pf}")" + if [ -z "${_alt}" ]; then + _alt_unreadable="${_alt_unreadable}${_pf} " + continue + fi + if [ -z "${_alt_ref}" ]; then + _alt_ref="${_alt}"; _alt_ref_file="${_pf}" + elif [ "${_alt}" != "${_alt_ref}" ]; then + _alt_drift="${_alt_drift}${_pf} matches ${_alt}; " + fi +done + +if [ -n "${_alt_unreadable}" ]; then + _bad "verdict parser(s) ${_alt_unreadable}no longer carry a recognisable \`(PASS|FAIL|…)\` alternation. A parser this suite cannot read is a parser it cannot police — and an unreadable parser is not an agreeing one." +elif [ -n "${_alt_drift}" ]; then + _bad "the verdict parsers have drifted apart: ${_alt_ref_file} matches ${_alt_ref}, but ${_alt_drift%; } — the same output would be graded differently by different suites, and a parser missing a word DROPS that gate rather than complaining" else _missing_form="" for _form in 'PASS' 'FAIL' 'NOT APPLICABLE' 'SKIPPED'; do - printf '%s' "${_alt_support}" | grep -qF -- "${_form}" \ + printf '%s' "${_alt_ref}" | grep -qF -- "${_form}" \ || _missing_form="${_missing_form}${_form}, " done if [ -n "${_missing_form}" ]; then - _bad "both verdict parsers agree, but the shared alternation ${_alt_support} no longer covers ${_missing_form%, } — agreement satisfied by narrowing both is the .github#401 defect in a new coat" + _bad "all verdict parsers agree, but the shared alternation ${_alt_ref} no longer covers ${_missing_form%, } — agreement satisfied by narrowing every parser is the .github#401 defect in a new coat" else - _ok "both verdict parsers match by the SAME shape — gate_fixture_support.sh and the acceptance matrix agree on ${_alt_support}" + _ok "all 3 verdict parsers match by the SAME shape — gate_fixture_support.sh, the acceptance matrix and the base-ref channel set agree on ${_alt_ref}" fi fi +_alt_support="${_alt_ref}" +_matrix="${GF_PKG_ROOT}/scripts/lib/test_gate_acceptance_matrix.sh" # POSITIVE CONTROL for the assertion above. Without it, "extract and compare" # is satisfiable by an extractor that returns empty for everything and calls