From 6be4b7e06083701334a5bca487430a0624c6cb2e Mon Sep 17 00:00:00 2001 From: Lukas Wuttke Date: Thu, 13 Aug 2026 14:39:22 +0200 Subject: [PATCH 1/3] ci(env): one CLIENT_ENV vocabulary, four languages, now checked (backend#1729 sweep 5) Sweep 5 of the inert-verification epic. The epic states the class as: "A verification written in the same vocabulary as the thing it verifies cannot detect a vocabulary error." and its evidence was 366 chart tests covering dev/stg/prod/unset/unknown while NOT ONE set `staging` -- the alias the chart's own docs recommend. That measured gap is now closed (staging has 9 cases, development 2, production 4). What is NOT closed is the structure underneath it: the same three alias->canonical mappings are declared FOUR times, in four languages, and nothing compares them. 1 client/templates/_helpers.tpl $aliases := dict ... Go template 2 client/values.schema.json the CLIENT_ENV enum JSON Schema 3 scripts/lib/common.sh tb_client_env() bash case 4 scripts/install-k8s.ps1 Get-TraceblocClientEnv PowerShell switch Two of the four have already drifted, separately, and been repaired separately: backend#1723 fixed the chart, backend#1745 fixed the bash installer -- whose own comment records the cost, "a raw `staging` fell through to the prod branch, so verify_credentials() checked staging credentials against the production backend and reported them invalid". So adding a seventh spelling to the template leaves both installers silently not reducing it, which is #1745 reintroduced in a repo that has already paid for it once. THE GUARD DERIVES, IT DOES NOT RESTATE. It parses all four declarations and compares them to each other; it holds no copy of the vocabulary, because a fifth hand-written list is the defect rather than the fix (the lesson of backend#1780 and backend#1828, where hand-copied declarations each claimed the others kept them honest and nothing crossed the boundary). It also asserts every accepted spelling is exercised by at least one helm-unittest case -- the specific thing #1729 measured. ARMED WHILE GREEN, deliberately: all four agree today and all six spellings are tested, so this imports no backlog. Arming a red check trains people to skip the tier -- the same reasoning as .github#235 and the opposite of what happened when a fleet-wide copies: bump reddened the org audit for 2h20m on 2026-08-12. Mutations, all five behaving correctly: template gains a 4th alias, installers unchanged 3 findings bash reducer drops staging (the #1745 shape) 1 finding PowerShell maps staging -> dev 1 finding schema accepts a spelling no reducer maps 2 findings a parser goes stale (dict renamed) EXIT 2, fail-closed That last one matters most: zero parsed pairs compares equal to zero parsed pairs, so a stale parser would report agreement between four declarations it never read. It exits 2 with a diagnostic instead. Wired into SHELLCHECK_FILES and `make check` beside its sibling, and into helm-ci with path filters covering all four declarations -- including scripts/lib/common.sh and scripts/install-k8s.ps1, so a change to either installer runs it. make check green; chart-env-vocabulary 28/28; gen-manifest --check clean. Co-Authored-By: Claude Opus 5 --- .github/workflows/helm-ci.yaml | 11 ++ Makefile | 4 +- scripts/tests/env-vocabulary-agreement.sh | 190 ++++++++++++++++++++++ 3 files changed, 204 insertions(+), 1 deletion(-) create mode 100755 scripts/tests/env-vocabulary-agreement.sh diff --git a/.github/workflows/helm-ci.yaml b/.github/workflows/helm-ci.yaml index c8a126e9..602ea2b0 100644 --- a/.github/workflows/helm-ci.yaml +++ b/.github/workflows/helm-ci.yaml @@ -11,6 +11,9 @@ on: - 'scripts/tests/e2e-full-seal.sh' - 'scripts/tests/lib/e2e-common.sh' - 'scripts/tests/chart-env-vocabulary.sh' + - 'scripts/tests/env-vocabulary-agreement.sh' + - 'scripts/lib/common.sh' + - 'scripts/install-k8s.ps1' - 'scripts/lib/**' - '.github/workflows/helm-ci.yaml' pull_request: @@ -23,6 +26,9 @@ on: - 'scripts/tests/e2e-full-seal.sh' - 'scripts/tests/lib/e2e-common.sh' - 'scripts/tests/chart-env-vocabulary.sh' + - 'scripts/tests/env-vocabulary-agreement.sh' + - 'scripts/lib/common.sh' + - 'scripts/install-k8s.ps1' - 'scripts/lib/**' - '.github/workflows/helm-ci.yaml' # Manual runs — mainly to fire full-seal-e2e on demand (e.g. right after the @@ -82,6 +88,11 @@ jobs: # turns them on with no change to this file. run: bash scripts/tests/chart-env-vocabulary.sh + # One vocabulary, four languages (backend#1729 sweep 5). Runs on a change + # to ANY of the four declarations, including the two installers. + - name: CLIENT_ENV vocabulary agreement (backend#1729) + run: bash scripts/tests/env-vocabulary-agreement.sh + template: timeout-minutes: 10 name: Template render diff --git a/Makefile b/Makefile index d6231a68..09ca6fdc 100644 --- a/Makefile +++ b/Makefile @@ -36,7 +36,8 @@ SHELLCHECK_FILES := \ scripts/tests/e2e-proxy.sh \ scripts/tests/lib/e2e-common.sh \ scripts/tests/path-persist.sh \ - scripts/tests/chart-env-vocabulary.sh + scripts/tests/chart-env-vocabulary.sh \ + scripts/tests/env-vocabulary-agreement.sh # The bats total, DERIVED — never written down. It moves on most PRs that add a # test, nothing enforces it, and the help text had drifted from its hardcoded @@ -196,6 +197,7 @@ helm-lint: .PHONY: helm-vocab helm-vocab: bash scripts/tests/chart-env-vocabulary.sh + bash scripts/tests/env-vocabulary-agreement.sh # helm-template: helm-ci.yaml `template`. kubeconform is pinned by # version AND digest in CI; rather than re-implement that download here, diff --git a/scripts/tests/env-vocabulary-agreement.sh b/scripts/tests/env-vocabulary-agreement.sh new file mode 100755 index 00000000..0801a8e7 --- /dev/null +++ b/scripts/tests/env-vocabulary-agreement.sh @@ -0,0 +1,190 @@ +#!/usr/bin/env bash +# +# env-vocabulary-agreement.sh — the CLIENT_ENV alias vocabulary is declared in +# FOUR languages, and until this script nothing checked they agreed. +# +# WHY THIS EXISTS (backend#1729, sweep 5) +# -------------------------------------- +# backend#1729 names the class: "a verification written in the same vocabulary +# as the thing it verifies cannot detect a vocabulary error." Its evidence was +# 366 chart tests covering dev/stg/prod/unset/unknown while NOT ONE set +# `staging` — the alias the chart's own docs tell users to write. +# +# That instance is fixed. This is the structural half: the same three +# alias->canonical mappings are written out four separate times, in four +# different languages, and each was fixed independently when it broke. +# +# 1 client/templates/_helpers.tpl `$aliases := dict ...` (Go template) +# 2 client/values.schema.json the CLIENT_ENV `enum` (JSON Schema) +# 3 scripts/lib/common.sh `tb_client_env()` (bash case) +# 4 scripts/install-k8s.ps1 `Get-TraceblocClientEnv` (PowerShell switch) +# +# backend#1723 fixed the chart. backend#1745 fixed the bash installer, whose +# own comment records the cost: "a raw `staging` fell through to the prod +# branch, so verify_credentials() checked staging credentials against the +# production backend and reported them invalid." Two of the four have already +# drifted, separately, and been repaired separately. +# +# So: add a seventh spelling to the template and the installers silently will +# not reduce it -- which is backend#1745 reintroduced, in a repo that has +# already paid for it once. +# +# THIS SCRIPT DERIVES, IT DOES NOT RESTATE. It parses all four declarations and +# compares them to each other. It deliberately holds no copy of the vocabulary: +# a fifth hand-written list is the defect, not the fix (the lesson of +# backend#1780 and backend#1828, where hand-copied declarations each claimed +# the others kept them honest and nothing crossed the boundary). +# +# It also asserts every accepted spelling is exercised by at least one +# helm-unittest case, which is the specific gap #1729 measured. +# +# READ-ONLY. Exit 0 clean, 1 disagreement, 2 cannot tell (fail closed). +set -uo pipefail + +here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +root="$(cd "$here/../.." && pwd)" + +TPL="$root/client/templates/_helpers.tpl" +SCHEMA="$root/client/values.schema.json" +BASH_LIB="$root/scripts/lib/common.sh" +PS1_FILE="$root/scripts/install-k8s.ps1" +TESTS_DIR="$root/client/tests" + +fail_closed() { printf 'ERROR: %s\n' "$1" >&2; exit 2; } +for f in "$TPL" "$SCHEMA" "$BASH_LIB" "$PS1_FILE"; do + [ -r "$f" ] || fail_closed "cannot read ${f#"$root"/} -- refusing to report agreement between declarations one of which was not read" +done +[ -d "$TESTS_DIR" ] || fail_closed "cannot read ${TESTS_DIR#"$root"/}" + +# --- 1. the Go template's alias dict ------------------------------------- +# `$aliases := dict "development" "dev" "staging" "stg" "production" "prod"` +tpl_pairs() { + sed -n 's/.*\$aliases := dict \(.*\)-}}.*/\1/p' "$TPL" \ + | tr -d '"' \ + | awk '{ for (i = 1; i < NF; i += 2) print $i "=" $(i+1) }' \ + | sort +} + +# --- 2. the JSON Schema enum (accepted spellings, not mappings) ---------- +schema_accepted() { + python3 -c ' +import json, sys +d = json.load(open(sys.argv[1])) +node = d.get("properties", {}).get("env", {}).get("properties", {}).get("CLIENT_ENV", {}) +vals = node.get("enum") +if vals is None: + sys.exit(3) +for v in vals: + if v != "": + print(v) +' "$SCHEMA" | sort +} + +# --- 3. bash tb_client_env() -------------------------------------------- +bash_pairs() { + awk '/^tb_client_env\(\)/ { inf = 1 } + inf && /^\}/ { exit } + inf && /printf/ { + line = $0 + if (match(line, /^[[:space:]]*[a-z]+\)/)) { + key = substr(line, RSTART, RLENGTH - 1); gsub(/[[:space:]]/, "", key) + if (match(line, /printf '"'"'[a-z]+'"'"'/)) { + val = substr(line, RSTART + 8, RLENGTH - 9) + print key "=" val + } + } + }' "$BASH_LIB" | sort +} + +# --- 4. PowerShell Get-TraceblocClientEnv ------------------------------- +ps1_pairs() { + awk '/function Get-TraceblocClientEnv/ { inf = 1 } + inf && /^\}/ { exit } + inf && /return/ { + if (match($0, /"[a-z]+"[[:space:]]*\{[[:space:]]*return[[:space:]]*"[a-z]+"/)) { + s = substr($0, RSTART, RLENGTH) + n = split(s, parts, /"/) + print parts[2] "=" parts[4] + } + }' "$PS1_FILE" | sort +} + +findings=0 +note() { findings=$((findings + 1)); printf '\nFINDING %d: %s\n' "$findings" "$1"; shift; for l in "$@"; do printf ' %s\n' "$l"; done; } + +tpl="$(tpl_pairs)" +bsh="$(bash_pairs)" +ps1="$(ps1_pairs)" +sch="$(schema_accepted)" || fail_closed "values.schema.json has no CLIENT_ENV enum -- the vocabulary is no longer closed there, which is a bigger finding than a disagreement" + +# An empty parse must never read as "they agree". Every declaration that stopped +# matching is reported, because zero pairs compares equal to zero pairs. +[ -n "$tpl" ] || fail_closed "parsed NO alias pairs from _helpers.tpl; the parser is stale and every comparison below would be vacuous" +[ -n "$bsh" ] || fail_closed "parsed NO alias pairs from common.sh tb_client_env()" +[ -n "$ps1" ] || fail_closed "parsed NO alias pairs from install-k8s.ps1 Get-TraceblocClientEnv" +[ -n "$sch" ] || fail_closed "parsed NO accepted values from the schema enum" + +printf 'CLIENT_ENV alias vocabulary, as declared by each of the four sources:\n' +printf ' _helpers.tpl %s\n' "$(echo "$tpl" | tr '\n' ' ')" +printf ' common.sh %s\n' "$(echo "$bsh" | tr '\n' ' ')" +printf ' install-k8s.ps1 %s\n' "$(echo "$ps1" | tr '\n' ' ')" +printf ' values.schema.json accepts: %s\n' "$(echo "$sch" | tr '\n' ' ')" + +# --- the three reducers must agree exactly ------------------------------ +if [ "$tpl" != "$bsh" ]; then + note "the chart and the bash installer disagree about the alias mappings" \ + "_helpers.tpl: $(echo "$tpl" | tr '\n' ' ')" \ + "common.sh : $(echo "$bsh" | tr '\n' ' ')" \ + "backend#1745 was exactly this: a spelling the chart reduced and the" \ + "installer did not, so verify_credentials() checked staging credentials" \ + "against the production backend and called them invalid." +fi +if [ "$tpl" != "$ps1" ]; then + note "the chart and the PowerShell installer disagree about the alias mappings" \ + "_helpers.tpl : $(echo "$tpl" | tr '\n' ' ')" \ + "install-k8s.ps1: $(echo "$ps1" | tr '\n' ' ')" \ + "A Windows install would resolve a different environment than the chart it" \ + "then deploys -- the same defect as #1745 on the other installer." +fi + +# --- the schema must accept exactly canonical + aliases ----------------- +canon="$(echo "$tpl" | cut -d= -f2 | sort -u)" +alias_keys="$(echo "$tpl" | cut -d= -f1 | sort -u)" +expected_accepted="$(printf '%s\n%s\n' "$canon" "$alias_keys" | sort -u)" +if [ "$sch" != "$expected_accepted" ]; then + note "values.schema.json's enum is not exactly {canonical} + {aliases}" \ + "schema accepts: $(echo "$sch" | tr '\n' ' ')" \ + "derived from the template: $(echo "$expected_accepted" | tr '\n' ' ')" \ + "A spelling the schema accepts but no reducer maps reaches the template raw." \ + "A spelling a reducer maps but the schema rejects is unreachable, so the" \ + "reducer's branch is dead code that looks like support." +fi + +# --- every accepted spelling must be exercised by a test --------------- +# This is the gap backend#1729 MEASURED: 366 cases, none setting `staging`. +while IFS= read -r spelling; do + [ -n "$spelling" ] || continue + n=$(grep -rhoE "CLIENT_ENV: *\"?${spelling}\"?[[:space:]]*$" "$TESTS_DIR" 2>/dev/null | wc -l | tr -d ' ') + if [ "${n:-0}" -eq 0 ]; then + note "no helm-unittest case sets CLIENT_ENV=$spelling" \ + "It is an accepted spelling, so a user can write it, and nothing renders" \ + "the chart with it. backend#1729 measured this exact gap: 366 cases" \ + "covering dev/stg/prod/unset/unknown and not one setting 'staging'," \ + "the alias the chart's own docs recommend." \ + "Fix: add a case to client/tests/ that sets env.CLIENT_ENV: $spelling." + else + printf 'ok CLIENT_ENV=%-12s exercised by %s test case(s)\n' "$spelling" "$n" + fi +done < Date: Thu, 13 Aug 2026 16:35:36 +0200 Subject: [PATCH 2/3] fix(env-check): don't misreport a missing python3 as a closed-vocabulary gap env-vocabulary-agreement.sh reads the CLIENT_ENV enum out of values.schema.json with python3, but `make setup` never installed (or even checked for) python3, so on the pre-push `make check` path a missing interpreter failed closed with the false diagnosis that the schema has no CLIENT_ENV enum and the vocabulary is no longer closed. The Go/bash/PowerShell reducers are jq-free by rule, so python3 is the JSON parser here and was the one unguarded dependency. Preflight `command -v python3` before the enum is read, and at the call site branch on the helper's exit status: 3 (its sys.exit for a genuinely-absent enum) still reports the real closed-vocabulary finding, while any other non-zero (a python3 that fails to run, malformed JSON) reports a distinct tooling/parse error. Add python3 to `make setup`'s prereq check so the gap is caught up front. Bugbot, client#706. Co-Authored-By: Claude Opus 4.8 --- Makefile | 8 ++++---- scripts/tests/env-vocabulary-agreement.sh | 21 ++++++++++++++++++++- 2 files changed, 24 insertions(+), 5 deletions(-) diff --git a/Makefile b/Makefile index 09ca6fdc..1d26cf8b 100644 --- a/Makefile +++ b/Makefile @@ -91,16 +91,16 @@ check-all: lint drift helm-lint helm-vocab bats helm-template helm-unittest .PHONY: setup setup: @missing=""; \ - for t in bash shellcheck bats helm; do \ + for t in bash shellcheck bats helm python3; do \ command -v $$t >/dev/null 2>&1 || missing="$$missing $$t"; \ done; \ if [ -n "$$missing" ]; then \ echo "missing:$$missing"; \ - echo " macOS: brew install shellcheck bats-core helm"; \ - echo " Debian/Ubuntu: apt-get install shellcheck bats (helm: https://helm.sh/docs/intro/install/)"; \ + echo " macOS: brew install shellcheck bats-core helm python"; \ + echo " Debian/Ubuntu: apt-get install shellcheck bats python3 (helm: https://helm.sh/docs/intro/install/)"; \ exit 1; \ fi; \ - echo "==> setup: shellcheck, bats and helm are all present; run 'make check'" + echo "==> setup: shellcheck, bats, helm and python3 are all present; run 'make check'" @$(MAKE) --no-print-directory install-hooks # install-hooks: put a pre-push hook in place that runs `make check`, so the diff --git a/scripts/tests/env-vocabulary-agreement.sh b/scripts/tests/env-vocabulary-agreement.sh index 0801a8e7..e9bfc869 100755 --- a/scripts/tests/env-vocabulary-agreement.sh +++ b/scripts/tests/env-vocabulary-agreement.sh @@ -56,6 +56,14 @@ for f in "$TPL" "$SCHEMA" "$BASH_LIB" "$PS1_FILE"; do done [ -d "$TESTS_DIR" ] || fail_closed "cannot read ${TESTS_DIR#"$root"/}" +# Reading the JSON Schema enum needs a JSON parser; per the repo's "no jq in +# installer scripts" rule that parser is python3 (see schema_accepted below). +# `make setup` does not install python3, so on the pre-push `make check` path a +# missing interpreter would otherwise surface below as "the schema has no enum" -- +# a false schema-gap diagnosis. A missing tool is not a vocabulary change: fail +# here, loud and distinct, before any empty result can be read as a gap. +command -v python3 >/dev/null 2>&1 || fail_closed "python3 is required for this check -- it reads the CLIENT_ENV enum out of values.schema.json (which is JSON) and was not found on PATH. Install python3 or add it to 'make setup'; this is a missing tool, not a schema change." + # --- 1. the Go template's alias dict ------------------------------------- # `$aliases := dict "development" "dev" "staging" "stg" "production" "prod"` tpl_pairs() { @@ -115,7 +123,18 @@ note() { findings=$((findings + 1)); printf '\nFINDING %d: %s\n' "$findings" "$1 tpl="$(tpl_pairs)" bsh="$(bash_pairs)" ps1="$(ps1_pairs)" -sch="$(schema_accepted)" || fail_closed "values.schema.json has no CLIENT_ENV enum -- the vocabulary is no longer closed there, which is a bigger finding than a disagreement" +# schema_accepted's python helper exits 3 only for a genuinely-absent enum, and 0 +# with output when it is present. Any OTHER non-zero (a python3 that passed the +# preflight but then failed to run, malformed JSON, a mid-run read error) is a +# tooling/parse failure, NOT evidence the schema changed -- so keep the two +# diagnoses apart: a real missing enum is still reported as the closed-vocabulary +# finding, and a tooling gap is never misreported as one. +sch="$(schema_accepted)"; schema_rc=$? +if [ "$schema_rc" -eq 3 ]; then + fail_closed "values.schema.json has no CLIENT_ENV enum -- the vocabulary is no longer closed there, which is a bigger finding than a disagreement" +elif [ "$schema_rc" -ne 0 ]; then + fail_closed "could not read the CLIENT_ENV enum from values.schema.json (python3 exited $schema_rc) -- a tooling or parse failure, not evidence the schema changed; refusing to guess" +fi # An empty parse must never read as "they agree". Every declaration that stopped # matching is reported, because zero pairs compares equal to zero pairs. From bcbfc0e7783682defd487f276da73a5a9ae58b61 Mon Sep 17 00:00:00 2001 From: Lukas Wuttke Date: Thu, 13 Aug 2026 17:14:55 +0200 Subject: [PATCH 3/3] ci(env): shellcheck env-vocabulary-agreement in the static job The Makefile's SHELLCHECK_FILES already lists env-vocabulary-agreement.sh, but installer-tests.yaml's static job shellchecked only through chart-env-vocabulary.sh, so CI never linted the new guard. Append it to both the error-gate and warning-advisory invocations so the workflow list matches the Makefile again (same files, same order). Bugbot, client#706. Co-Authored-By: Claude Opus 4.8 --- .github/workflows/installer-tests.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/installer-tests.yaml b/.github/workflows/installer-tests.yaml index 05ee822a..a6c476d1 100644 --- a/.github/workflows/installer-tests.yaml +++ b/.github/workflows/installer-tests.yaml @@ -77,11 +77,11 @@ jobs: # below for visibility but don't fail the gate. shellcheck --severity=error --shell=bash \ scripts/install.sh scripts/install-k8s.sh scripts/gen-manifest.sh scripts/check-facts.sh scripts/check-style.sh scripts/resolve-ingestor-digest.sh scripts/lib/*.sh \ - scripts/tests/check-drift.sh scripts/tests/distro-prereqs.sh scripts/tests/e2e-auto-upgrade.sh scripts/tests/e2e-seal-check.sh scripts/tests/e2e-full-seal.sh scripts/tests/e2e-cluster.sh scripts/tests/e2e-journey.sh scripts/tests/e2e-proxy.sh scripts/tests/lib/e2e-common.sh scripts/tests/path-persist.sh scripts/tests/chart-env-vocabulary.sh + scripts/tests/check-drift.sh scripts/tests/distro-prereqs.sh scripts/tests/e2e-auto-upgrade.sh scripts/tests/e2e-seal-check.sh scripts/tests/e2e-full-seal.sh scripts/tests/e2e-cluster.sh scripts/tests/e2e-journey.sh scripts/tests/e2e-proxy.sh scripts/tests/lib/e2e-common.sh scripts/tests/path-persist.sh scripts/tests/chart-env-vocabulary.sh scripts/tests/env-vocabulary-agreement.sh echo "── shellcheck warnings (advisory, non-blocking) ──" shellcheck --severity=warning --shell=bash \ scripts/install.sh scripts/install-k8s.sh scripts/gen-manifest.sh scripts/check-facts.sh scripts/check-style.sh scripts/resolve-ingestor-digest.sh scripts/lib/*.sh \ - scripts/tests/check-drift.sh scripts/tests/distro-prereqs.sh scripts/tests/e2e-auto-upgrade.sh scripts/tests/e2e-seal-check.sh scripts/tests/e2e-full-seal.sh scripts/tests/e2e-cluster.sh scripts/tests/e2e-journey.sh scripts/tests/e2e-proxy.sh scripts/tests/lib/e2e-common.sh scripts/tests/path-persist.sh scripts/tests/chart-env-vocabulary.sh || true + scripts/tests/check-drift.sh scripts/tests/distro-prereqs.sh scripts/tests/e2e-auto-upgrade.sh scripts/tests/e2e-seal-check.sh scripts/tests/e2e-full-seal.sh scripts/tests/e2e-cluster.sh scripts/tests/e2e-journey.sh scripts/tests/e2e-proxy.sh scripts/tests/lib/e2e-common.sh scripts/tests/path-persist.sh scripts/tests/chart-env-vocabulary.sh scripts/tests/env-vocabulary-agreement.sh || true - name: Installer manifest is current (supply-chain, R8) # The bootstrap verifies each sub-script against scripts/manifest.sha256