From 5b965dd5a49773208bf3452d176ef91300f63d2c Mon Sep 17 00:00:00 2001 From: Mark Date: Sun, 2 Aug 2026 12:11:08 +0800 Subject: [PATCH 1/3] Stop the plugin wildcard breaking the Zabbix agent install MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Installing the agent on a fresh Debian 13 host failed mid-apt, leaving zabbix-agent2 half-configured and aborting the script. Root cause, proved by A/B in a container: the install line used the zabbix-agent2-plugin-* wildcard, which pulls in zabbix-agent2-plugin-nvidia-gpu. On a host with no NVIDIA driver the agent's own config test aborts with "NVML Shared Library couldn't be found or loaded". Agent alone validates successfully; agent plus the wildcard exits 1. Because ExecStartPre runs that config test, the service fails to start, the postinst fails with it, and dpkg is left with the package unpacked but never configured. The script already has a "disable unneeded loadable plugins" step written for exactly this failure, but it runs after the install, so on a fresh install it can never fire — it only ever protected a re-run. Changes: - Install zabbix-agent2 only. Loadable plugins are installed per detected service; postgresql now joins the existing mssql and redis handling, so the wildcard's useful coverage is preserved without installing plugins for hardware and services the host does not have. - Defer service starts for the duration of the package work with a temporary policy-rc.d, so no plugin can fail the package configure again. It is restored before the agent is started, with an EXIT trap so it is never left behind, and it only ever removes the file it created — a pre-existing policy-rc.d belongs to the host and now survives. - Clear a pre-existing half-configured dpkg state before installing. With starts deferred this succeeds where it previously failed, so re-running the script recovers a host already stuck in this state. - Validate the config before starting the service, so a failure reports the agent's own message instead of a bare systemd exit code. Tested 10/10 in Debian 13 containers: policy-rc.d save/restore semantics, fresh install (clean dpkg state, config test passes, nvidia plugin not installed), reproduction of the reported failure, and recovery of a host broken by it. --- Progress.md | 13 ++++ install-zabbix-agent-linux-tactical-rmm.sh | 90 +++++++++++++++++++++- 2 files changed, 102 insertions(+), 1 deletion(-) diff --git a/Progress.md b/Progress.md index 9f1a002..aa830af 100644 --- a/Progress.md +++ b/Progress.md @@ -144,6 +144,19 @@ documenting hardcoded choices (the config writes `ProxyMode=0` literally) — co ## Completed +- 2026-08-02: Zabbix agent install died mid-`apt` on Debian 13 with the package left half-configured. **Root cause + proved by A/B in a container**: the install line used the `zabbix-agent2-plugin-*` wildcard, which pulls in + `zabbix-agent2-plugin-nvidia-gpu`; on a host with no NVIDIA driver the agent's own `-T` config test dies with + "NVML Shared Library couldn't be found or loaded". Agent alone → `Validation successful`; agent + wildcard → + exit 1. Since `ExecStartPre` runs that test, the postinst fails and dpkg is left half-configured. The script's + own "disable unneeded plugins" guard is written for exactly this but runs *after* the install, so on a fresh + install it can never fire. Fix on branch `zabbix-plugin-install-fix`: install the agent only and add loadable + plugins per detected service (postgresql joined mssql/redis); defer service starts during package operations + with a temporary `policy-rc.d` so no plugin can fail the package configure again; clear a pre-existing + half-configured state (this makes a re-run recover a stuck host); validate config before starting so failures + show the agent's message rather than a bare systemd exit code. Tested 10/10 in Debian 13 containers covering + fresh install, the nvidia reproduction, and recovery. Caught in my own change: the EXIT trap would have deleted + a *pre-existing* `policy-rc.d` — it now only removes the file it created. - 2026-06-29: VERIFIED WORKING end-to-end on a live agent: 2.10.0 -> 2.11.0, service running. Committed the TRMM bootstrap as `trmm-self-update-bootstrap.sh`, updated README (manual vs TRMM-bootstrap usage, with the self-restart/cgroup explanation) and CLAUDE.md structure. - 2026-06-29: Found the actual root cause of the exit-1 compile failure (after disproving download/rate-limit and Go-version theories via live diagnostics): the systemd-run transient unit used by the TRMM bootstrap runs with a stripped env and no HOME, so `go build` aborts instantly with "GOCACHE is not defined". Direct build with HOME set succeeds. Fix (commit f3a3612): pin HOME/GOCACHE/GOPATH before compile; drop `--simple` and capture output so real build errors are no longer hidden. go.mod requires go 1.20 (agent has 1.25.6 — version was never the issue). - 2026-06-29: Debugged fleet-wide TRMM Linux agent update failures. Root cause: community script downloads rmmagent source via single no-retry `wget -q` under `set -e`; transient HTTP error (429 when many agents hit codeload.github.com at once) → exit 8, instant abort, nothing compiled. Confirmed agents on 2.10.0, master=2.11.0 (real update pending), CGO_ENABLED=0 (no gcc needed — red herring). systemd-run detachment + bootstrap worked fine. Fix: pre-fetch source with retry+backoff + neutralise community wget, startup jitter (non-interactive), retry compile once. Commit 1a846c6. diff --git a/install-zabbix-agent-linux-tactical-rmm.sh b/install-zabbix-agent-linux-tactical-rmm.sh index ace20de..357b8bb 100644 --- a/install-zabbix-agent-linux-tactical-rmm.sh +++ b/install-zabbix-agent-linux-tactical-rmm.sh @@ -102,6 +102,52 @@ send_discord() { service_active() { systemctl is-active --quiet "$1" 2>/dev/null; } package_installed() { dpkg -s "$1" &>/dev/null; } +# --- Defer service start during package installs ----------------------------- +# The zabbix-agent2 postinst starts the service, whose ExecStartPre runs +# `zabbix_agent2 -T` (config test). Any loadable plugin that cannot initialise +# makes that test exit 1, which fails the postinst and leaves dpkg +# half-configured — the package is unpacked but never configured, and every +# later apt call complains until it is cleared. +# +# A temporary policy-rc.d telling dpkg not to start services during the install +# removes that failure mode entirely: we configure the agent and disable the +# plugins we do not need first, then start it ourselves at the end. It also lets +# a re-run recover a host already stuck in that half-configured state. +POLICY_RC_D="/usr/sbin/policy-rc.d" +POLICY_RC_D_SAVED="" +POLICY_RC_D_OURS=0 + +defer_service_start() { + if [[ -e "$POLICY_RC_D" ]]; then + POLICY_RC_D_SAVED="${POLICY_RC_D}.zabbix-install-$$" + mv "$POLICY_RC_D" "$POLICY_RC_D_SAVED" || { warn "Could not set aside existing policy-rc.d"; return 1; } + fi + if printf '#!/bin/sh\nexit 101\n' > "$POLICY_RC_D" && chmod 0755 "$POLICY_RC_D"; then + POLICY_RC_D_OURS=1 + else + warn "Could not install policy-rc.d — the package may try to start the agent during install" + # Put back whatever was there before rather than leaving the host bare. + [[ -n "$POLICY_RC_D_SAVED" && -e "$POLICY_RC_D_SAVED" ]] && mv "$POLICY_RC_D_SAVED" "$POLICY_RC_D" + POLICY_RC_D_SAVED="" + fi +} + +# Only ever removes the policy-rc.d we created — a pre-existing one belongs to +# the host and must survive, including when this runs from the EXIT trap after +# a path that never deferred anything. +restore_service_policy() { + [[ "$POLICY_RC_D_OURS" -eq 1 ]] || return 0 + rm -f "$POLICY_RC_D" + POLICY_RC_D_OURS=0 + if [[ -n "$POLICY_RC_D_SAVED" && -e "$POLICY_RC_D_SAVED" ]]; then + mv "$POLICY_RC_D_SAVED" "$POLICY_RC_D" || warn "Could not restore the original policy-rc.d" + POLICY_RC_D_SAVED="" + fi +} + +# Never leave the host with service starts globally disabled, whatever happens. +trap restore_service_policy EXIT + # --- Gather system info ------------------------------------------------------ SYS_HOSTNAME=$(hostname -f) IP_ADDRESS=$(hostname -I | awk '{print $1}') @@ -190,8 +236,25 @@ fi apt-get update -qq +# Hold off service starts for the rest of the run; we start the agent ourselves +# once its config is written and the unneeded plugins are disabled. +defer_service_start + +# Clear any half-configured state left by a previous failed attempt. With starts +# deferred this can now succeed where it previously failed on the config test. +if [[ -n "$(dpkg --audit 2>/dev/null)" ]]; then + warn "dpkg has packages in a broken state — reconfiguring before install..." + dpkg --configure -a >/dev/null 2>&1 || warn "dpkg --configure -a did not fully succeed" +fi + log "Installing zabbix-agent2..." -DEBIAN_FRONTEND=noninteractive apt-get install -y -qq -o Dpkg::Options::="--force-confold" zabbix-agent2 zabbix-agent2-plugin-* +# Install the agent ONLY. Loadable plugin packages are installed further down, +# per detected service. Installing them all (the old `zabbix-agent2-plugin-*` +# wildcard) breaks the install on any host lacking the matching hardware or +# libraries: the NVIDIA plugin aborts the agent's own config test with +# "NVML Shared Library couldn't be found or loaded" on every machine without an +# NVIDIA driver, which is nearly all of them. +DEBIAN_FRONTEND=noninteractive apt-get install -y -qq -o Dpkg::Options::="--force-confold" zabbix-agent2 fi # end skip-on-reconfigure @@ -338,6 +401,15 @@ fi if service_active postgresql; then log " [FOUND] PostgreSQL"; DETECTED_SERVICES+=("PostgreSQL") + # Loadable plugin — previously pulled in by the plugin wildcard, now installed + # only where PostgreSQL is actually running. + if ! package_installed zabbix-agent2-plugin-postgresql; then + log " Installing zabbix-agent2-plugin-postgresql..." + DEBIAN_FRONTEND=noninteractive apt-get install -y -qq -o Dpkg::Options::="--force-confold" zabbix-agent2-plugin-postgresql 2>/dev/null \ + || warn " Could not install zabbix-agent2-plugin-postgresql" + else + log " zabbix-agent2-plugin-postgresql already installed" + fi cat > "${AGENT_CONF_D}/postgresql.conf" <<'EOF' # PostgreSQL - Zabbix Agent 2 Plugin # CREATE USER zabbix WITH PASSWORD 'StrongPassword!'; @@ -538,12 +610,28 @@ else fi # --- Enable & restart agent -------------------------------------------------- +# Config is written and unneeded plugins are disabled, so it is safe to let +# services start again before we bring the agent up. +restore_service_policy + +# Validate the config before starting, so a failure reports the agent's own +# message rather than a bare systemd exit code. +log "Validating configuration..." +if ! CONFIG_TEST=$(zabbix_agent2 -T -c "$AGENT_CONF" 2>&1); then + echo "$CONFIG_TEST" | sed 's/^/ /' + send_discord "❌ Zabbix Agent Config Invalid" \ + "**Host:** \`$SYS_HOSTNAME\`\n**IP:** \`$IP_ADDRESS\`\n**Reason:** zabbix_agent2 -T failed — see script output" 15158332 + echo "ERROR: zabbix_agent2 configuration test failed — not starting the agent"; exit 1 +fi +log "Configuration valid" + log "Restarting zabbix-agent2..." systemctl enable zabbix-agent2 --quiet systemctl restart zabbix-agent2 sleep 2 if ! systemctl is-active --quiet zabbix-agent2; then + systemctl status zabbix-agent2 --no-pager 2>&1 | tail -15 | sed 's/^/ /' send_discord "❌ Zabbix Agent Failed to Start" \ "**Host:** \`$SYS_HOSTNAME\`\n**IP:** \`$IP_ADDRESS\`\n**Version:** \`$NEW_VERSION\`\n**Proxy:** \`$ZABBIX_PROXY\`" 15158332 echo "ERROR: zabbix-agent2 failed to start"; exit 1 From d7836fb659cb993e50070c925d19a1db29d7cfe1 Mon Sep 17 00:00:00 2001 From: Mark Date: Sun, 2 Aug 2026 13:14:23 +0800 Subject: [PATCH 2/3] Fix version short-circuit never matching (package epoch) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Zabbix's Debian/Ubuntu packages carry an epoch, e.g. 1:7.4.13-1+debian13. Comparing that raw against the target major.minor ('7.4') never matched, so the 'already on target version, nothing to do' short-circuit could never fire and every run — including every scheduled TacticalRMM run — re-entered the full repository-add and install path. Strip the epoch before comparing, and anchor on the dot so 7.4 does not match 7.44. Verified against real package version strings. --- install-zabbix-agent-linux-tactical-rmm.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/install-zabbix-agent-linux-tactical-rmm.sh b/install-zabbix-agent-linux-tactical-rmm.sh index 357b8bb..0977aa7 100644 --- a/install-zabbix-agent-linux-tactical-rmm.sh +++ b/install-zabbix-agent-linux-tactical-rmm.sh @@ -178,9 +178,15 @@ ACTION="" if package_installed zabbix-agent2; then PREV_VERSION=$(dpkg -s zabbix-agent2 | grep '^Version:' | awk '{print $2}') + # Zabbix's Debian/Ubuntu packages carry an epoch, e.g. "1:7.4.13-1+debian13". + # Comparing that raw against "7.4" never matches, so the "already on target + # version" short-circuit below could never fire and every run — including + # every scheduled TacticalRMM run — re-entered the full install path. + # Strip the epoch before comparing. + PREV_VERSION_CMP="${PREV_VERSION#*:}" log "Installed version: $PREV_VERSION" - if [[ "$PREV_VERSION" == ${ZABBIX_VERSION}* ]]; then + if [[ "$PREV_VERSION_CMP" == ${ZABBIX_VERSION}.* || "$PREV_VERSION_CMP" == "${ZABBIX_VERSION}" ]]; then if [[ "$FORCE_RUN" == "force" ]]; then log "Already on version $ZABBIX_VERSION — force flag set, reconfiguring." ACTION="Reconfigured" From dfd5d3dba6c948832da10beb38aa462c8b33e7d2 Mon Sep 17 00:00:00 2001 From: Mark Date: Sun, 2 Aug 2026 13:40:27 +0800 Subject: [PATCH 3/3] Drop Progress.md from this branch to avoid a three-way merge conflict Progress.md is updated once, on the repair-script branch. Keeping the same edits on three branches would conflict on every merge after the first. --- Progress.md | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/Progress.md b/Progress.md index aa830af..9f1a002 100644 --- a/Progress.md +++ b/Progress.md @@ -144,19 +144,6 @@ documenting hardcoded choices (the config writes `ProxyMode=0` literally) — co ## Completed -- 2026-08-02: Zabbix agent install died mid-`apt` on Debian 13 with the package left half-configured. **Root cause - proved by A/B in a container**: the install line used the `zabbix-agent2-plugin-*` wildcard, which pulls in - `zabbix-agent2-plugin-nvidia-gpu`; on a host with no NVIDIA driver the agent's own `-T` config test dies with - "NVML Shared Library couldn't be found or loaded". Agent alone → `Validation successful`; agent + wildcard → - exit 1. Since `ExecStartPre` runs that test, the postinst fails and dpkg is left half-configured. The script's - own "disable unneeded plugins" guard is written for exactly this but runs *after* the install, so on a fresh - install it can never fire. Fix on branch `zabbix-plugin-install-fix`: install the agent only and add loadable - plugins per detected service (postgresql joined mssql/redis); defer service starts during package operations - with a temporary `policy-rc.d` so no plugin can fail the package configure again; clear a pre-existing - half-configured state (this makes a re-run recover a stuck host); validate config before starting so failures - show the agent's message rather than a bare systemd exit code. Tested 10/10 in Debian 13 containers covering - fresh install, the nvidia reproduction, and recovery. Caught in my own change: the EXIT trap would have deleted - a *pre-existing* `policy-rc.d` — it now only removes the file it created. - 2026-06-29: VERIFIED WORKING end-to-end on a live agent: 2.10.0 -> 2.11.0, service running. Committed the TRMM bootstrap as `trmm-self-update-bootstrap.sh`, updated README (manual vs TRMM-bootstrap usage, with the self-restart/cgroup explanation) and CLAUDE.md structure. - 2026-06-29: Found the actual root cause of the exit-1 compile failure (after disproving download/rate-limit and Go-version theories via live diagnostics): the systemd-run transient unit used by the TRMM bootstrap runs with a stripped env and no HOME, so `go build` aborts instantly with "GOCACHE is not defined". Direct build with HOME set succeeds. Fix (commit f3a3612): pin HOME/GOCACHE/GOPATH before compile; drop `--simple` and capture output so real build errors are no longer hidden. go.mod requires go 1.20 (agent has 1.25.6 — version was never the issue). - 2026-06-29: Debugged fleet-wide TRMM Linux agent update failures. Root cause: community script downloads rmmagent source via single no-retry `wget -q` under `set -e`; transient HTTP error (429 when many agents hit codeload.github.com at once) → exit 8, instant abort, nothing compiled. Confirmed agents on 2.10.0, master=2.11.0 (real update pending), CGO_ENABLED=0 (no gcc needed — red herring). systemd-run detachment + bootstrap worked fine. Fix: pre-fetch source with retry+backoff + neutralise community wget, startup jitter (non-interactive), retry compile once. Commit 1a846c6.