pgaftest: DSL/runner improvements + monitor fixes#1150
Open
dimitri wants to merge 88 commits into
Open
Conversation
dimitri
force-pushed
the
pgaftest-improvements
branch
2 times, most recently
from
July 16, 2026 16:19
e7e89d4 to
d9e27ac
Compare
- Dockerfile: trailing backslash from debug edit made the next FROM line get appended to the RUN command, breaking make build entirely. Remove the stray continuation backslash. - expected/upgrade_1.out: PG18 added a 'Default version' column to \dx output; upgrade_1.out is the PG18+ variant. Still showed '2.2' as the default version; update all four rows to '2.3'. - expected/pg19/expected/monitor.out: mirror of monitor.out for PG19 (different pg_lsn display format). Did not include the ORDER BY nodeid clause added to monitor.sql; add it to keep the echoed SQL in the expected output in sync with the query file.
- test_runner.c (runner_compose_generate): after writing docker-compose.yml and node ini files, copy the spec file to <workdir>/spec.pgaf. This lets 'pgaftest step' reload the spec without the user supplying the original path again. - test_runner.c (runner_init): derive the Docker Compose project name from the work-directory basename (e.g. replication_stall_3dc) rather than the spec filename basename. When the spec is loaded from the in-workdir copy spec.pgaf the old code yielded 'spec', which did not match the running stack and caused 'no such container: spec-node2-1' errors. COMPOSE_PROJECT_NAME env var still takes precedence (used inside the compose network). - replication_stall_3dc.pgaf: add 'ssl off' to the cluster block. Without it pgaftest defaults to self-signed SSL, which requires certificate generation before Postgres starts; node1 failed with 'could not load server certificate file server.crt'.
parseCurrentNodeState already parsed 17 columns (0-16), including noderegion at column 14 added with the --region / issue-997 work. The guard check above it still said 16, causing pg_autoctl watch to log 'Query returned 17 columns, expected 16' in a tight loop and refuse to display any node state.
…ries The old Makefile had a phony 'common' target that ran make -C common, then both pg_autoctl and pgaftest depended on it. However, because 'common' was .PHONY, make always considered it out of date and rebuilt it on every invocation — and a parallel make -j could still race the two sub-makes against each other's common/*.o output files since both include Makefile.common and carry the same compile rules. Fix: make a real file target. Both pg_autoctl and pgaftest depend on it, so make builds the archive once serially first, then the two binaries in parallel without contention.
In --tmux mode the bottom pane previously exec'd into the first data node container, which had no pgaftest binary, no docker CLI, and no knowledge of the spec file. New approach: the pgaftest service (already emitted by compose_gen for CI) is reused as the interactive shell target. That container has: - the pgaftest binary - docker + docker compose (DooD via /var/run/docker.sock) - /spec.pgaf bind-mounted from the host workDir - COMPOSE_PROJECT_NAME, PGAFTEST_HOST_WORK_DIR, PG_AUTOCTL_MONITOR set compose_gen_write gains an 'interactive' parameter: when true the pgaftest service uses 'sleep infinity' instead of 'pgaftest run /spec.pgaf', keeping it alive for a shell. The interactive flag is set when runner_setup is called with withTmux=true. The bottom pane command changes from: docker compose exec -it <node1> bash to: docker compose exec -it pgaftest sh -c 'pgaftest _setup_ ... && exec bash' Four new sub-commands mirror the DSL keywords, callable directly from the interactive shell (context resolved from PGAFTEST_SPEC / PGAFTEST_HOST_WORK_DIR env vars injected by the compose stack): pgaftest wait until <node> state = <state> [timeout <N>s] pgaftest sql <node> "<query>" pgaftest network connect|disconnect <node> pgaftest assert <node> state = <state> After setup the bottom pane prints a hint listing available steps and example commands so the user knows immediately what they can type.
Two bugs in the previous commit:
1. hintCmd[256] was smaller than the literal format string (292 bytes)
before even inserting the step list, triggering sformat's BUG assert.
2. The bottom pane command used sh -c "..." with single-quote delimiters
inside; step names or other text containing apostrophes would break
the shell syntax, causing 'unexpected EOF' on startup.
Fix: drop hintCmd and the sh -c wrapper entirely. The bottom pane now
runs pgaftest _setup_ directly via docker compose exec (no sh -c), and
cli_run_setup_only prints the hint to stdout (the pane tty) after the
setup block completes, then execlp("bash") to hand the pane over to an
interactive shell. No string embedding, no quoting concerns.
…TEST_SPEC - pgaftest help (and bare pgaftest) now prints usage instead of "unknown command" - pgaftest show is now a sub-command set: show compose|spec|steps|services - pgaftest step accepts optional <spec.pgaf> second arg; when omitted it falls back to PGAFTEST_SPEC env var, /spec.pgaf in CWD, or <workDir>/spec.pgaf - compose_gen emits PGAFTEST_SPEC=/spec.pgaf into the pgaftest service env so all interactive sub-commands (step, wait, assert, network, sql) can discover the spec without explicit path arguments inside the container - resolve_interactive_context() helper centralises the env fallback logic
- pgaftest show step: lists sequence steps with:
* next step to run
! last failed step (will retry)
(space) completed steps
- pgaftest step (no args): auto-advances to the next step using a state
file at <workDir>/pgaftest.state. On failure current stays at the
failed step so the next no-arg invocation retries it.
- pgaftest step <name>: still works; also updates the state file and
advances current when the named step matches the cursor position.
- TestRunnerState struct + runner_state_read / runner_state_write /
runner_step_next added to test_runner.c/h; state is JSON for easy
inspection with a text editor.
- show_resolve_spec() now derives workDir when it is not set, so
pgaftest show step <spec.pgaf> picks up the state file automatically.
getopt permutes argv to move flags before positionals when optind is reset to 0 (full re-initialisation). Without this, the library's first getopt pass at the root level stopped at the first non-option (the subcommand name), leaving any flags that appeared before the spec file — e.g. "setup --tmux spec.pgaf" — unprocessed by the sub-command's getopt, which then saw an empty positional list and printed the usage error. Setting optind=0 (as every pg_autoctl getopts function does) tells getopt to reorder the argument vector so all options are seen before non-option positionals, regardless of the order the user typed them. Also simplify cli_setup back to its original form now that ordering is handled at the getopt level instead of by hand.
Replaces commandline_help() (one-level flat listing) with commandline_print_command_tree() so that pgaftest help renders the full command tree with sub-command groups expanded, matching the output style of pg_autoctl help.
The workDir bind-mount was :ro, preventing runner_state_write from creating pgaftest.state inside the container. The state file belongs in workDir alongside the generated compose files, so drop the :ro flag.
The pgaftest image already creates a 'docker' user with HOME=/var/lib/postgres. Switch the pgaftest compose service from root to that user: user: docker working_dir: /var/lib/postgres spec mounted at /var/lib/postgres/spec.pgaf This lands the interactive shell in the docker user's HOME with the spec file right there, and aligns the SSL cert paths with ~/.postgresql/.
…file Adds: - Host vs. container command table with descriptions - Typical interactive session walkthrough - DooD architecture section explaining the pgaftest service container - Step state file section - Updated synopsis and sub-command reference to match current CLI (show compose|spec|step|services, step auto-advance, etc.)
runner_state_path() picks the state file location based on whether PGAFTEST_COMPOSE_SERVICE is set: - inside container → $HOME/pgaftest.state (always writable by docker user) - on host → <workDir>/pgaftest.state (alongside compose files) The bind-mounted workDir may not be writable by the container's docker user on Linux hosts where UIDs don't align. $HOME (/var/lib/postgres) is owned by the docker user in the image, so it is unconditionally writable. Update docs/ref/pgaftest.rst to reflect the new state file locations.
…ate/step Command table changes: - setup / prepare / down moved under new 'cluster' subcommand group (pgaftest cluster setup|prepare|down) - pgaftest tmux <spec.pgaf> replaces pgaftest setup --tmux; --tmux option removed from pgaftest_getopts entirely - pgaftest wait and pgaftest assert removed (not useful interactively) show subcommand changes: - show steps (plural, restored) — sequence list with */ ! progress markers - show step (singular, new) — prints the DSL commands of the next step to run - show state (new) — 'Step X/N: name' header then pg_autoctl show state output - show services removed New functions in test_runner.c: - test_cmd_print(): prints a TestCmd in DSL form to a FILE* - runner_show_state(): prints step progress header then pg_autoctl show state
The old name implied the variable held a service name. It is actually a boolean flag (set to "1") that tells the binary it is running inside the pgaftest container — which determines where to write the step state file (container $HOME vs host workDir). Also complete the docs/ref/pgaftest.rst update: Synopsis and Sub-commands sections now reflect the restructured CLI (cluster setup/prepare/down, tmux, show steps/step/state, removal of wait/assert/show services).
…eady When the monitor container is still starting up, 'pg_autoctl watch' fails on its first connection attempt and exits, closing the tmux pane. Add --wait to pg_autoctl watch: it calls pgsql_set_monitor_interactive_retry_policy() before entering the main loop, which makes the internal pgsql_open_connection() retry for up to 15 minutes with exponential back-off (same policy used by the demo app and enable/disable maintenance commands). pgaftest now passes --wait in the tmux middle pane so the watch pane stays alive while the monitor container initialises.
…r startup Makefile.common: replace individual -Werror=implicit-* flags with a single -Werror so all -Wall warnings are fatal on both macOS and Linux. The generated bison/flex files already carry -Wno-error in the pgaftest Makefile so those remain unaffected. test_runner.c: the tmux bottom pane (docker compose exec -it pgaftest) exited immediately when the container wasn't yet running, closing the pane before the user could see it. Fix: probe with a no-TTY 'docker compose exec -T pgaftest true' loop until the container accepts exec, then hand off to the interactive command. Same timing issue as the watch pane (fixed earlier with --wait). Also log the manual exec command on startup so the user can reattach to the pgaftest shell from the host tmux pane if needed: docker compose -p <project> -f <workdir>/docker-compose.yml exec -it pgaftest bash
Convenience command for opening a new interactive shell in the running pgaftest container from a host tmux pane: pgaftest cluster sh [--work-dir <dir>] Equivalent to: docker compose -p <project> -f <workdir>/docker-compose.yml exec -it pgaftest bash Resolves the project name and workdir the same way as the other cluster subcommands (--work-dir flag, PGAFTEST_HOST_WORK_DIR env, or derived from the spec file path).
The 'sleep infinity + exec' model caused a timing race: tmux opens the
pane before the pgaftest container is in 'running' state, so exec fails
and the pane closes immediately. The probe-loop workaround was a band-
aid; the real fix is to not require a background service at all.
docker compose run starts a fresh container on demand from the service
definition (same image, bind-mounts, env, network) without needing the
service to already be running. No timing race, no probe loop.
Changes:
- runner_compose_up: pass --scale pgaftest=0 in interactive mode so the
image is built but no background container is started
- tmux session now has 4 panes when the spec has a setup{} block:
pane 0 docker compose logs -f
pane 1 pg_autoctl watch --wait
pane 2 pgaftest _setup_ via docker compose run --rm (closes when done)
pane 3 interactive bash via docker compose run --rm -it
Without setup{}: 3 panes (pane 2 is the interactive shell directly)
- cluster sh: updated to use docker compose run --rm -it as well
…est/.last) pgaftest cluster setup / tmux write the active work directory to $TMPDIR/pgaftest/.last on startup. resolve_interactive_context() reads it as the final fallback when neither --work-dir nor PGAFTEST_HOST_WORK_DIR nor a spec file is available. Effect: after 'pgaftest tmux tests/.../foo.pgaf', all subsequent host commands work without arguments: pgaftest cluster sh pgaftest cluster down pgaftest show steps pgaftest show state
runner_down() now kills the tmux session (named after the project) after compose down, so a subsequent 'pgaftest tmux' for the same spec does not hit 'duplicate session: <name>'. The kill is best-effort (2>/dev/null || true) so it is harmless when no tmux session exists.
The pgaftest container runs as the 'docker' user but /var/run/docker.sock is owned by a host group (GID 1 on macOS Docker Desktop, typically 999 on Linux) that does not match any group inside the container. This caused: permission denied while trying to connect to the docker API at unix:///var/run/docker.sock Fix: stat /var/run/docker.sock at compose-generation time and emit 'group_add: [<gid>]' in the pgaftest service so the container user gains the socket's group as a supplementary group, regardless of what numeric GID the host assigns to it.
- runner_setup: check 'tmux -V' before doing anything when withTmux is
true; log the version at NOTICE level, error out cleanly if tmux is
not on PATH. Uses run_cmd_capture, the same helper used everywhere
else in the runner.
- Setup container renamed from <project>-pgaftest-run-<hash> to
<project>-setup so compose logs show a short, readable name.
Interactive shell container named <project>-sh for the same reason.
Both still carry --rm so they are removed when they exit.
docker socket permission history
---------------------------------
Previously the pgaftest service ran as 'user: root'. Root bypasses the
socket's group-ownership check, so DooD worked without any group
membership.
We changed to 'user: docker' (an unprivileged UID) to avoid running
test commands as root inside the container. /var/run/docker.sock is
typically owned root:docker with mode 0660 (or root:<N> where N is the
host docker GID, which varies: 1 on macOS Docker Desktop, ~999 on most
Linux distros). The container's 'docker' user is not in that group, so
every docker CLI call got EACCES.
The previous commit added 'group_add: ["<gid>"]' by statting the
socket at compose-generation time. 'group_add' is honoured by both
'docker compose up' and 'docker compose run', so it applies to the
setup and shell containers started via 'run --rm' as well. The compose
file must be regenerated ('pgaftest cluster down && pgaftest tmux') for
the fix to take effect on an existing cluster.
…ude from up
Root cause of monitor not running
-----------------------------------
In interactive mode, runner_compose_up passed --scale pgaftest=0 to
prevent the sleep-infinity container from starting. Docker Compose v2
still rebuilds ALL service images when --build is present, even for
scaled-to-zero services. On a re-run (stack already up from a previous
pgaftest tmux invocation), 'up --build -d --scale pgaftest=0' rebuilds
and recreates the monitor/node containers, causing a brief downtime
window during which runner_wait_for_monitor could time out, tear the
stack down, and leave the cluster gone — hence 'monitor not running'.
Fix
---
In interactive mode the service is now named 'setup' and carries
'profiles: [setup]'. Services with a profile are completely invisible
to 'docker compose up' (no build, no start, no scale interaction).
'docker compose run setup' still works and starts the container on
demand. This means:
- 'docker compose up --build -d' only builds and starts cluster
services (monitor, nodes) — never touches the setup image.
- Subsequent pgaftest tmux runs on an already-running stack are
non-destructive: up is a no-op for running healthy containers.
- --scale pgaftest=0 removed; no longer needed.
Service naming
--------------
Interactive mode: service named 'setup' — appears as '<project>-setup'
in logs and docker ps. The --name flag on docker compose run gives
containers the stable names '<project>-setup' (setup pane) and
'<project>-sh' (interactive shell pane).
CI mode: service still named 'pgaftest', no profile, command set to
'pgaftest run <spec.pgaf>' so 'docker compose up --exit-code-from
pgaftest' continues to work unchanged.
…y ensures readiness) In tmux mode the host process does not use the monitor LISTEN connection at all — it just starts the stack and hands off to the tmux session. docker compose up -d already waits for the full depends_on: service_healthy chain before returning, so the monitor is provably ready when up exits. runner_wait_for_monitor from the host is both unnecessary and the source of the intermittent 'monitor not running' failure: on Docker Desktop for Mac, published-port connections appear as 192.168.65.1, which is outside the monitor's pg_hba trust CIDR. The fallback pg_hba patch and the 120s timeout loop both ran on the critical path before tmux launched, creating a wide window for races. runner_apply_formation_settings only needs docker compose exec (internal network) — no host->monitor libpq — so it works fine without the wait. runner_wait_for_monitor is still called in CI mode (runner_run) where the host process genuinely drives steps via LISTEN/NOTIFY.
…ke build The setup container runs Docker-out-of-Docker (DooD) by bind-mounting /var/run/docker.sock. Running as root eliminates GID mismatch between macOS Docker Desktop and Linux CI, where the docker socket group ID varies by host. Changes: - Dockerfile: remove docker user, sudo, entrypoint.sh; add WORKDIR /root - Makefile.docker: add build-pgaftest and force-build-pgaftest targets, include build-pgaftest in the default 'make build' target
…ECONDARY
The postmaster writes PM_STATUS_READY / PM_STATUS_STANDBY to postmaster.pid
a few milliseconds before the TCP listener is actually bound.
pg_setup_wait_until_is_ready() (called via ensure_postgres_service_is_running)
returns as soon as the PID-file flag is set, leaving a narrow window where
pgIsRunning is reported as true to the monitor but connections are still
refused.
This causes two distinct CI failures:
1. Connection refused: test queries a standby immediately after it reaches
SECONDARY state, hitting the window before the TCP socket is open.
2. start_maintenance '0 candidate nodes available': CountHealthyCandidates()
called IsHealthy() on the secondary just as the health-check worker ran
and marked it BAD (because TCP was not yet open).
Fix: in the two FSM transitions that promote a standby to SECONDARY
(CATCHINGUP->SECONDARY via fsm_prepare_for_secondary, and
JOIN_SECONDARY->SECONDARY via fsm_follow_new_primary), switch the local sql
client to the init retry policy (up to 15 minutes, exponential back-off from
5 ms to 2 s) and call pgsql_get_postgres_metadata() before declaring success.
pgsql_get_postgres_metadata calls pgsql_open_connection, which falls through
to pgsql_retry_open_connection (PQping loop) when the initial PQconnectdb
fails, so no new mechanism is needed. The function also captures the current
LSN, sync state and control-file data as a useful side-effect.
tests/test_multi_alternate_primary_failures.py: test_005_002 was checking for
the transient assigned state 'report_lsn' which lasts only milliseconds; the
monitor advances through it to 'prepare_promotion' before the poll can see it.
Wait for the stable end state 'primary' instead (300s timeout).
tests/tap/specs/multi_alternate.pgaf: same fix for the pgaftest path.
…est spec
The 'wait until node3 assigned-state = report_lsn' used
runner_wait_assigned_goal which does NOT pre-drain the libpq notification
buffer on entry. Under load on GHA the state can arrive and be consumed
during the preceding compose-kill exec, making it invisible to the
assigned-state poll loop.
Switch to:
wait until node3 state is primary passing through report_lsn timeout 300s
This routes through runner_wait_notify_goal which:
1. Drains the entire buffered notification queue on entry via
runner_drain_notify(), catching states that arrived while the
previous exec was blocked.
2. Tracks report_lsn as a required pass-through goalState — fails if
the monitor skipped it.
3. Waits for the stable end-state (primary reported-state) rather than
a transient millisecond assignment.
The pytest counterpart (test_multi_alternate_primary_failures.py) was
already fixed in the prior commit to poll wait_until_state('primary')
since SQL polling cannot catch transient states at all.
write_legacy_node_command() generated 'pg_autoctl create postgres' without --name, causing the monitor to auto-assign node_N names from the sequence counter. Those names are derived from the insertion order, not from the service names in the compose file, so they diverge when nodes restart in a different order — for example in test_008_failover_post_upgrade where the upgrade process brings nodes up sequentially. The registered name mismatch showed up as nodes not being found by service name in subsequent steps. Fix: pass the service name (n->name) as --name so the monitor always registers the node under the same identifier as the compose service.
Replace the inline 'alpine:3 + apk add dnsmasq' command in the generated
docker-compose.yml with a pre-built pgaf:dnsmasq image.
Problems with the previous approach:
- apk add runs at container start, fetching from Alpine CDN on every test
run: slow (~2-5s) and fragile if the network is slow on GHA runners.
- No healthcheck: other containers could start before dnsmasq was ready,
causing initial hostname resolution failures.
- Extra network failure mode independent of the test under execution.
Changes:
Dockerfile — new 'dnsmasq' target: FROM alpine:3, apk add dnsmasq,
HEALTHCHECK with nslookup, ENTRYPOINT that reads
/etc/pgaf-hosts via --addn-hosts and forwards to
configurable upstream DNS servers (DNS1/DNS2 env vars).
compose_gen.c — _dns service uses PGAF_DNS_IMAGE env var (default
pgaf:dnsmasq) instead of alpine:3 with inline install.
Healthcheck is emitted in the compose YAML so depends_on
can use service_healthy.
monitor, second monitor, and all data nodes gain
depends_on: _dns: condition: service_healthy
so no node starts before DNS is available.
Build:
docker build --target dnsmasq -t pgaf:dnsmasq .
Override image:
PGAF_DNS_IMAGE=my-registry/dnsmasq:latest pgaftest run spec.pgaf
Add a build_dnsmasq job that builds the new Dockerfile 'dnsmasq' target
once per workflow run and passes the image to every pgaftest consumer:
build_dnsmasq:
- FROM alpine:3 target, no PGVERSION dependency
- GHA layer cache (scope=pgaf-dnsmasq) — fast cache hits on re-runs
- Saves pgaf:dnsmasq to a tarball artifact (retention-days: 1)
Wire-up:
test_pgaftest — needs: [build_run_images, build_pgaftest, build_dnsmasq]
downloads + loads dnsmasq-image artifact
passes -e PGAF_DNS_IMAGE=pgaf:dnsmasq to docker run
upgrade — same: needs build_dnsmasq, downloads + loads, passes env var
The generated docker-compose.yml already uses PGAF_DNS_IMAGE (default
pgaf:dnsmasq) for the _dns service, so once the image is loaded on the
runner the variable just confirms the name. Previously the compose file
used 'alpine:3' with 'apk add dnsmasq' at startup, which fetched from
Alpine CDN on every test run and had no healthcheck, causing a race where
data nodes could start before DNS was ready.
…node
master_update_node() on the Citus coordinator takes a distributed lock
that conflicts with active queries touching the old worker's shards.
Without a timeout the call blocks indefinitely. This was observed as a
20-minute CI hang: the citus_cluster_name spec performs three switchovers
in sequence; the coordinator switchover completed in ~6s but the
immediately-following worker1a switchover blocked in coordinator_update_node_prepare
because coordinator1b (the newly-promoted coordinator) had in-flight
distributed transactions from before its own failover.
Fix: after pgsql_begin() and before the master_update_node() call, issue:
SET LOCAL lock_timeout = '<cooldown>ms';
SET LOCAL statement_timeout = '<cooldown*2>ms';
lock_timeout fires if the distributed lock isn't acquired within the
cooldown window (default 10 000 ms). statement_timeout is a backstop for
the whole call in case something else inside master_update_node stalls
after the lock is acquired. Both are SET LOCAL so they are cleared
automatically at transaction end and do not affect other uses of the
connection.
On timeout the transaction is aborted with an error; the FSM logs the
failure and retries the transition on the next keeper tick instead of
hanging forever. The no-force path (older Citus without the force
argument) was previously completely unbounded; it now also benefits from
this guard.
When pg_autoctl set node candidate-priority raises a node's priority above the current primary, the monitor may trigger a full failover instead of the usual apply_settings cycle. In that case the old primary is assigned report_lsn (not apply_settings) as its first FSM step, so the apply_settings wait loop never saw the new primary reach primary/primary and timed out after 60 s. Fix: add failoverInProgress to ApplySettingsNotificationContext. When the known primary node is assigned report_lsn we set this flag and switch to waiting for ANY node in the formation to reach primary/primary instead of continuing to track the apply_settings cycle for the original primary node.
The grammar comment and test spec files use 'passing through' as the natural English phrasing: wait until node3 state is primary passing through report_lsn but the scanner only recognised the single keyword 'through'. Add a two-word flex rule that matches 'passing' followed by whitespace and 'through', returning a single T_THROUGH token — so both forms work with no grammar changes. Update test_spec_scan.c accordingly.
compose_gen.c: compose_network_name() was returning '<project>_default' but the generated compose YAML defines the network as 'pgafnet', so 'docker network disconnect/connect' were failing with 'network not found'. Fix: return '<project>_pgafnet' to match the compose YAML. test_runner.c: 'docker network connect' without --ip lets Docker assign a fresh DHCP address. The dnsmasq pgaf-hosts file maps each node to its original static IP; PostgreSQL resolves HBA hostnames through that file. When the container gets a different IP after reconnect, the HBA rule no longer matches and streaming replication fails with 'no pg_hba.conf entry'. Fix: read the static IP from workDir/pgaf-hosts and pass --ip to 'docker network connect' so the container re-uses its original address. cli_enable_disable.c: align ternary operator indentation to pass citus_indent. multi_alternate.pgaf: test_005_002 starts node2 before killing node1 so that node2 participates in the LSN quorum. node2 was SIGKILLed in test_005_001 and never reported 'demoted'; it remained registered as a sync standby with replication_quorum=true. With number_sync_standbys=1 the monitor needs 2 LSN reporters; without node2 back online only node3 would report, stalling the failover indefinitely. test_005_003 no longer needs compose start node2. multi_standbys.pgaf: increase test_013_restart_node2 timeout from 90s to 180s. After network disconnect/reconnect, node2 must go through: contact monitor -> monitor notifies node1 -> node1 updates HBA -> pg_rewind/basebackup from node1 -> streaming replication -> secondary. 90s was insufficient.
The previous commit's runner_hosts_lookup() used single-statement if/while bodies without braces, which citus_indent (run via 'make docker-check', matching CI) rejects. Run 'make docker-indent' to apply the required brace style.
…pgaf Align with the naming convention used by every other Citus spec (citus_cluster_name, citus_force_failover, citus_multi_standbys, citus_skip_pg_hba) so all Citus tests are prefixed the same way. Updates the master tests/tap/schedule list and citus-2.sch.
print_cmd() had no case for CMD_RUN ("run <svc> <args>") or CMD_FAILOVER
("perform failover [in formation F] [group G]"), so pgaftest indent
silently rewrote any spec using either into "(unknown cmd N)" — a real
data-corruption bug for a tool meant to be a safe formatter. Both commands
are in active use: drop_node_destroy.pgaf uses run, several multi-node
specs use perform failover.
CMD_RUN mirrors the existing CMD_EXEC case. CMD_FAILOVER reconstructs
whichever of the four grammar forms matches from the stored service
(formation, "default" when omitted) and waitGroups[0] (group, 0 when
omitted).
Verified: indenting multi_standbys.pgaf and drop_node_destroy.pgaf now
round-trips perform failover / run correctly (only expected diffs remain:
dropped inline step-body comments and a redundant sequence block, both
pre-existing, documented limitations of the tool).
PG_AUTOCTL_TEST_DELAY previously derived each node's registration delay
by parsing a trailing numeric suffix off its name (node1 -> 2s, node2 ->
4s, ...), so that node IDs get assigned in a predictable order across
the depends_on-guaranteed-concurrent startup of every node after the
first. That parsing only works for names ending in a digit -- Citus-style
names (worker1a, coordinator1b, ...) end in a letter and silently got
zero delay, leaving their relative registration order to a race.
compose_gen_write() now assigns each node a 0-based ordinal by its
position in cluster{} declaration order across every formation (the same
order already used to pick the depends_on anchor node) and passes it
directly via PG_AUTOCTL_TEST_DELAY, instead of a node name. cli_node_run()
just reads that number -- it never parses the node's name at all, so this
works identically for any naming convention.
Also: read the value with stringToInt() instead of atoi(), matching
convention used elsewhere in this codebase (atoi silently treats garbage
as 0; stringToInt distinguishes the two and lets a malformed value log a
warning instead of misbehaving quietly). Guard the get_env_copy() call
with env_exists() first -- the monitor service (which also runs through
cli_node_run(), and never has PG_AUTOCTL_TEST_DELAY set) would otherwise
hit get_env_copy()'s unconditional log_error() for a var that is
legitimately absent.
Verified: dry-run compose generation shows correct sequential ordinals
(0,1,2,3 for a 4-node plain formation; 0-5 for a 6-node coordinator/worker
Citus formation). Full runs of multi_standbys.pgaf (27/27),
citus_multi_standbys.pgaf (14/14), and quick.sch (4/4) all pass with zero
PG_AUTOCTL_TEST_DELAY errors.
Audited every .pgaf spec's network disconnect / compose stop / stop
postgres steps against the Python method they were ported from
(node.fail(), node.stop_pg_autoctl(), node.stop_postgres(),
node.ifdown()/ifup()) and fixed two categories of drift.
1. network disconnect substituted for a graceful process stop:
node.fail() sends SIGTERM to pg_autoctl (a graceful stop), not a
network partition -- but basic_operation.pgaf's test_010_fail_primary
and multi_standbys.pgaf's test_012_fail_primary /
test_015_002_fail_two_standby_nodes ported it as 'network disconnect'.
In basic_operation.pgaf this duplicated the file's own dedicated
partition test (test_021-023, correctly ported from ifdown()/ifup()),
silently losing coverage of the graceful-stop path. In multi_standbys
the fix also removes an internal inconsistency: test_014_002 (same
scenario, different formation config) already correctly used
'compose stop'. Fixed all three to 'compose stop' / 'compose start',
matching the sibling steps that were already correct.
2. 'stop postgres <node>' used where a real external stop was needed:
'stop postgres <node>' (pg_autoctl manual service pgctl off) writes a
persistent "expected: stopped" flag that the already-running keeper
honors -- it cannot exercise the keeper's own "ensure" self-healing
logic, because it explicitly tells the keeper not to restart. Python's
node.stop_postgres() bypasses pg_autoctl entirely (raw pg_ctl --mode
fast stop), so the keeper still believes Postgres should be running
and its own polling loop notices and restarts it -- that's the actual
behaviour under test. Switched to "exec <node> pg_ctl --wait --mode
fast stop" in:
- ensure.pgaf test_003_init_secondary (the file's whole purpose is
testing this self-healing behaviour)
- basic_operation.pgaf test_024_stop_postgres_monitor (whose own
comment already claimed to bypass pg_autoctl, but the
implementation didn't)
- basic_operation.pgaf test_008_001 and multi_maintenance.pgaf
test_006a (stop-during-maintenance checks that previously
couldn't distinguish 'maintenance suppressed the restart' from
'pgctl off already guaranteed no restart regardless')
Python's test_002_stop_postgres (stop the primary directly while it's
still a lone 'single' node, verify self-heal) is intentionally not
ported into basic_operation.pgaf: that spec's setup{} brings up all
three nodes upfront, so node1 already has a synchronous secondary by
the time any step runs, and stopping its Postgres reliably triggers a
real failover instead of a quiet self-heal -- duplicating
test_010/012 rather than adding coverage. See the comment left in
place of the step for the full reasoning.
Verified with full local runs: basic_operation.pgaf 27/27,
multi_standbys.pgaf 27/27, ensure.pgaf 5/5, multi_maintenance.pgaf 22/22.
Unify exec/run macro syntax under one $name(args) call convention regardless of how a macro resolves -- $cidr() shells out to the monitor at expansion time, $ip(<node>) is a pure file lookup, but a spec author shouldn't need two different-looking syntaxes to tell them apart. $ip(<node>) resolves to the static IP dnsmasq's pgaf-hosts assigns to <node> (via the existing runner_hosts_lookup() helper, also used by runner_network_on()'s --ip fix). Neither macro is cached: every occurrence re-resolves independently.
docs/ref/pgaftest.rst: - Replace the dated Python-porting-mapping content (audit-style, tied to a specific pass over the specs) with a permanent test catalog: every .pgaf file under tests/tap/specs/ with a short description drawn from its own header comment. The Python method -> .pgaf command mapping now lives once, in README.md, rather than duplicated in both places. - Document the five distinct ways to make a node stop responding (network disconnect, compose stop, compose kill, stop postgres, exec pg_ctl fast stop) and what each actually exercises -- converted from a fragile hand-aligned RST simple-table to a definition list after a column-width edit silently broke the table. - Document $cidr()/$ip() macro syntax and semantics. - New "Node registration order" section explaining the depends_on + PG_AUTOCTL_TEST_DELAY mechanism and why it applies equally to Citus naming. src/bin/pgaftest/README.md: - New "Porting failure-simulation semantics from the Python suite" section: the Python mechanism table, the .pgaf equivalents, and the specific audit findings (which specs were checked, which were fixed and why, which were left as an intentional simplification). - New "Deterministic node registration order" section: the depends_on / PG_AUTOCTL_TEST_DELAY mechanism in full, including why an earlier name-suffix-parsing version didn't work for Citus node names. - "Static IP on network connect" section for the runner_network_on() --ip fix. Verified: rst2html reports zero real errors (only pre-existing Sphinx :ref: roles docutils doesn't know about), and make -C docs html builds clean with zero warnings.
Neither macro has a real caller, and the reasons behind each are gone: - %CIDR% (this macro's predecessor) existed for exactly one caller, installcheck.pgaf's HBA-widening step, so pg_regress could reach the monitor over the Docker bridge subnet. That file was already deleted before this branch started (installcheck now runs inside the Docker build stage via pg_virtualenv, connecting over a local Unix socket -- the whole "reach the monitor over some subnet" problem it solved no longer exists). Nothing else ever called it. - $ip(<node>) was added speculatively earlier in this same branch, while chasing an unrelated network-reconnect bug, on a "might be useful" basis. It was never adopted by any spec. Removing both: runner_expand_macros() and its two call sites in runner_exec_cmd() (CMD_EXEC, CMD_RUN -- now just copy cmd->args verbatim), and the macro reference table in docs/ref/pgaftest.rst. runner_hosts_lookup() itself is untouched -- it's also used by runner_network_on()'s --ip fix, which is real and load-bearing. Verified: make docker-check and bash ci/banned.h.sh both clean, make -C docs html builds with zero warnings and zero remaining mentions of either macro, and multi_standbys.pgaf still runs 27/27.
Reproduces 5/6 pytest/monitor CI failures (PG14, PG15, PG17, PG18, PG19), identical on every run: "node3 failed to reach primary after 300 seconds". Root cause: NOT pgautofailover.guard_data_loss. That GUC (added in #1142) only changes behavior when explicitly set to false (--allow-data-loss); its default (true) reproduces byte-for-byte the pre-#1142 blocking behavior in ProceedGroupStateForMSFailover -- confirmed by diffing ca7834b: every "return false;" guarding on missingNodesCount/quorumCandidateCount already existed unconditionally before that commit, the commit only wraps it in "if (GuardDataLoss)" and adds an unblocking else-branch that only runs when the GUC is off. Default-on behavior is unchanged. The real regression is commit a7974f8 ("fsm: wait for local Postgres to accept connections before reporting SECONDARY"), landed earlier in this branch. While fixing an unrelated connection-refused race, it also "fixed" this test's assertion: assert node1.wait_until_assigned_state(target_state="draining") assert node3.wait_until_assigned_state(target_state="report_lsn") --> assert node3.wait_until_state(target_state="primary", timeout=300) reasoning that 'report_lsn' is too transient to reliably observe. That's true, but the replacement assertion is unreachable as written: at this point in the test node2 is still down (killed in test_005_001 and not restarted until test_005_003), so with number_sync_standbys=1 the monitor needs 2 LSN reports before it will elect a new primary and node3 cannot reach "primary" here -- only node2's restart in test_005_003 can unblock that, and origin/main's version of this test correctly waits for "report_lsn" here and defers the "primary" assertion to test_005_003. a7974f8 turned a transient-state observability problem into a guaranteed 300s timeout by moving the wrong assertion to the wrong test function. The .pgaf port of this same test (multi_alternate.pgaf) hit the identical issue and was already fixed correctly in 3c9ca49, right after a7974f8: it both moves "compose start node2" earlier (into test_005_002) *and* adds a "passing through report_lsn" wait primitive to robustly observe the transient state via LISTEN. This commit applies the equivalent fix to the plain pytest side, which 3c9ca49 didn't touch: start node2 before killing node1, so quorum is satisfiable and "wait for primary" is reachable within test_005_002 itself. test_005_003_bring_up_first_failed_primary no longer needs to start node2 itself, and its wait for the transient 'demoted' state (which may have already come and gone by the time the test runs, since node2 restarts earlier now) is replaced with a wait for the stable end state, matching the same fix pattern already used elsewhere in this file. test_003_002_stop_primary (the 1/6 PG16-specific failure) was not touched: its assertions already tolerate node3 staying stuck at report_lsn (it explicitly checks node3 does NOT reach wait_primary), so it isn't exposed to this bug the same way -- its single failure is more likely independent CI timing flakiness and should be monitored separately. Reproduced locally before and after: fails identically to CI before this change (333s runtime, node3 timeout after 300s); after the fix, all 15 tests in the file pass in 173s.
coordinator_update_node_prepare() runs on a single connection wrapped in
pgsql_begin(), which sets PGSQL_CONNECTION_MULTI_STATEMENT. In that mode
pgsql_execute()/pgsql_execute_with_params() deliberately keep the
connection open on failure ("Multi statements might want to ROLLBACK and
hold to the open connection for a retry step" -- pgsql.c) so the caller
can decide whether to retry or roll back.
Every failure branch in this function except one (the
transactionHasAlreadyBeenPrepared case) just did "return false;" without
calling pgsql_rollback(), so the connection was never closed. On a
formation where the coordinator keeps failing the same operation on every
FSM tick -- e.g. once per second while a worker is "draining" during a
switchover -- this leaks roughly one coordinator connection per retry.
Confirmed locally: reproduced citus_cluster_name.pgaf's worker1a
switchover hang (matches CI jobs 32/33, PG17/PG18) and polled
coordinator1b's pg_stat_activity directly. Connection count grew
26->41->56->70->85->99 across successive polls, all of them
"idle in transaction (aborted)" backends that had run
"SELECT master_update_node(nodeid, $2, $3, force => tru...", until
coordinator1b hit "FATAL: sorry, too many clients already" -- a full
outage, not just a stuck switchover.
Fix: call pgsql_rollback() before every "return false;" that follows a
successful pgsql_begin(), mirroring the pattern already used for the
transactionHasAlreadyBeenPrepared case. pgsql_rollback() issues ROLLBACK
and unconditionally closes the connection itself, so this is safe to call
even when the failing statement already left the connection in an aborted
transaction state.
Verified locally after the fix, driving citus_cluster_name.pgaf's
worker1a switchover by hand and polling coordinator1b every ~20s for over
3 minutes: connection count stays flat (7-8, 0 aborted) instead of
growing.
Note: fixing the leak does not by itself resolve the switchover hang.
Once the leak stops masking it, coordinator1b's keeper log shows the real
error on every retry: "Postgres ERROR: there is already another node
with the specified hostname and port". This is a separate, still-open
bug -- pg_dist_node already has a row for worker1b as the citus-secondary
"readonly" cluster node at the same host:port that master_update_node()
is trying to move the primary's row to, and Citus's (nodename, nodeport)
uniqueness constraint rejects the update. That collision, not lock
contention, is the actual reason the switchover in citus_cluster_name.pgaf
never completes. Tracked separately.
Reusable diagnostic spec for the citus_cluster_name.pgaf worker1a switchover hang (CI jobs 32/33, PG17/PG18), following the same pattern as debug_failover_pg19.pgaf: a minimal topology plus a background-exec and timed polling steps to capture internal state instead of blocking on the hanging command. This spec is what led to root-causing both halves of that hang: the coordinator.c connection leak fixed in 318f67a, and the still-open pg_dist_node (nodename, nodeport) collision with citus-secondary "readonly" cluster nodes, documented in the spec's header comment.
perform_failover()'s two-node fast path (used both for automatic failover
and for "pg_autoctl perform switchover" in a two-node group) never checked
the standby's candidatePriority before promoting it. Every other place in
the codebase treats candidatePriority == 0 as "never eligible as a
failover target":
- BuildCandidateList/CountHealthyCandidates skip priority-0 nodes for
groups with more than two nodes.
- set_node_candidate_priority refuses to let a citus-secondary
"read-replica" node's priority be set to anything but 0, and refuses
to drop a node's priority to 0 if it would leave the group without a
non-zero-priority candidate.
The two-node path was the one place this invariant wasn't enforced, so a
manual switchover against a two-node group whose only standby is a
citus-secondary read-replica (candidate-priority is pinned at 0 for those
nodes) would go ahead and promote it anyway.
Reproduced locally: citus_cluster_name.pgaf sets up exactly this shape
(each group's non-primary member is registered citus-secondary with
citus-cluster-name=readonly, candidate-priority 0) and its setup{} called
"pg_autoctl perform switchover" on the primary of each group. The FSM
happily assigned prepare_promotion to the priority-0 standby, which then
retried master_update_node() forever: pg_dist_node already had a row for
that node's hostname:port from its citus-secondary registration, so
Citus's (nodename, nodeport) uniqueness constraint rejected every retry
with "there is already another node with the specified hostname and
port". That's what produced the 20-minute CI hang in citus-1 (PG17/PG18)
-- not lock contention, and (once the separate connection leak fixed in
318f67a stopped masking it) not a resource exhaustion issue either: the
promotion was simply never going to succeed.
Fix: apply the same candidatePriority == 0 check the multi-node path
already has, in the two-node path, with an error message that explains
why up front instead of an indefinite retry loop discovered only via the
coordinator's Postgres log.
tests/tap/specs/citus_cluster_name.pgaf: drop the switchover calls from
setup{}. They were attempting exactly the now-refused operation, and
weren't attempting anything the test itself needs -- its own header
describes it as a read-routing test, none of its steps depend on a
switchover having happened, and the original Python predecessor
(test_citus_cluster_name.py) never called switchover or promote at all.
Verified locally: citus_cluster_name.pgaf passes on both PG17 and PG18
after this fix (previously hung to the 20-minute CI timeout on both).
Groups it with the other citus_*.pgaf specs alphabetically and by name; "nonha_citus" read as "non-HA Citus" but sorted away from its siblings. Updates tests/tap/schedule, tests/tap/schedules/citus-2.sch, docs/ref/pgaftest.rst, and src/bin/pgaftest/README.md accordingly. Verified the renamed spec still runs under the new path/schedule entry.
Every pgaftest compose stack previously ran a dedicated dnsmasq container
(_dns) that served a static hosts file over DNS, with every other service
pointed at it via `dns: [...]`, specifically to work around Docker's
embedded 127.0.0.11 resolver dropping queries under container churn on
GitHub Actions runners.
That workaround is no longer needed: since every node's IP is already
static (hashed from the project name at compose-gen time), the same
name-to-IP mapping can be written directly into each service's
`extra_hosts:` list in the generated docker-compose.yml. That resolves
names via a local /etc/hosts read (NSS "files") instead of a DNS query,
which can't drop a packet or time out the way a UDP lookup can, and
removes an entire container plus the depends_on/healthiness dependency
every other service had on it being up first.
compose_gen.c changes:
- compose_dns_ip() removed; IP offsets renumbered now that there's no
_dns container to reserve .2 for (monitor=.2, secondMonitor=.3, data
nodes=.4+, was .3/.4/.5+).
- New compose_write_extra_hosts() writes the same mapping as an
`extra_hosts:` block instead of compose_gen_write_hosts()'s file.
- compose_gen_write_hosts() (the pgaf-hosts file) is kept: it's no
longer read by any container, but pgaftest itself still reads it back
(runner_hosts_lookup in test_runner.c) to reconnect a node to its
original static IP after `network disconnect`/`connect`, independent
of how name resolution works.
- The containerized "pgaftest"/"setup" compose service (used for
Docker-in-Docker CI wrapper invocations and `docker compose run
setup`) previously had neither `dns:` nor `networks:` set despite
needing to resolve "monitor" via PG_AUTOCTL_MONITOR -- a latent gap,
unrelated to dnsmasq specifically. It now gets extra_hosts and joins
pgafnet like every other service.
Removed entirely: the `dnsmasq` Dockerfile target, the `build_dnsmasq` CI
job and its artifact upload/download/load steps in test_pgaftest and
upgrade, and PGAF_DNS_IMAGE.
Verified locally: citus_nonha_operation.pgaf and citus_cluster_name.pgaf
both pass end-to-end, including the network disconnect/reconnect steps
(coordinator and worker failovers) that depend on the static-IP lookup
still working correctly with extra_hosts instead of dnsmasq.
New src/monitor/sql/drop_node.sql, driven directly via SQL (register_node /
node_active / remove_node), to check the monitor-side half of Bug C: the
citus_nonha_operation.pgaf test_008_remove_old_primaries stall where, after
dropping a two-node group's last standby, the primary is intermittently
observed stuck at "wait_primary" instead of moving to "single", and
"pg_autoctl drop node" times out after 60s waiting for the standby's row to
disappear (root-caused via ~13 local Docker reproductions this session, at
roughly a 15% hit rate).
This test isolates the monitor's own bookkeeping from the real keeper
processes and proves it correct on every run:
- remove_node() on the standby sets its goal to "dropped" and, in the
same call, moves the primary straight to "single" -- never
"wait_primary". AutoFailoverNodeGroup()'s "goalstate <> 'dropped'"
filter means RemoveNode()'s inline ProceedGroupState(primaryNode)
already sees a one-node group by the time it runs.
- A stale/racing node_active() report from the about-to-be-dropped node
(simulating a keeper report that hasn't caught up with the new
"dropped" goal) does not corrupt the primary or resurrect the
standby: ProceedGroupState()'s early-return for goalState=DROPPED
means no new goal gets assigned, though reportedstate is still
updated as unconditional bookkeeping -- this is the exact shape of
the "New state for this node ... : single -> single" line seen in
CI, where the about-to-be-dropped node reports something other than
"dropped" and the row is left pending until it complies.
- remove_node() called again before the standby complies is a safe,
idempotent no-op (goal already "dropped", number_sync_standbys not
decremented twice).
- Once the standby reports "dropped", its row is actually removed and
the primary is left alone, unaffected.
Net effect: this rules out a monitor-side sequencing bug in RemoveNode()
itself as the explanation for the observed "wait_primary" symptom -- that
path is correct on every call. The remaining, narrower suspect is a real
concurrency window between RemoveNode()'s transaction and the primary's
own independent node_active() call: NodeActive() (node_active_protocol.c)
reads the node via GetAutoFailoverNodeById() before acquiring
LockFormation/LockNodeGroup, so a primary's own concurrent poll could in
principle observe pre-drop state before blocking on the lock. Confirming
that requires true multi-session concurrency (e.g. an isolation-tester
spec), which this codebase doesn't currently have infrastructure for; not
pursued further here.
Placed last in REGRESS, after fast_forward: pgautofailover.node has a
global (nodehost, nodeport) unique constraint (not scoped per formation),
so node names must be unique across every test file in the same
database -- used dn_p/dn_s rather than the bare p/s that collided with
guard_data_loss.sql's own 'p' node on first attempt. Running after every
existing test avoids shifting their hardcoded assigned_node_id
expectations, which are global-sequence values.
Wires PGXS's ISOLATION variable into src/monitor/Makefile alongside the
existing REGRESS variable -- `make installcheck` now runs both. Isolation
specs live in src/monitor/specs/*.spec; their accepted output lives in
the same expected/ directory REGRESS already uses (pg_isolation_regress's
--expecteddir defaults to --inputdir, so no new directory convention is
needed). No Dockerfile change: `make -C src/monitor/ installcheck` already
runs both once ISOLATION is set, and pg_isolation_regress/isolationtester
ship in the same postgresql-server-dev package pg_regress already comes
from.
Two specs, in order of what they're building toward:
- concurrent_remove_node.spec: two sessions call remove_node() on the
same standby concurrently. This finds a real bug: remove_node_by_nodeid()
reads the target node via GetAutoFailoverNodeById() *before*
RemoveNode() acquires LockFormation(ExclusiveLock). The second,
blocked session resumes with a pre-lock snapshot once the first
commits, so RemoveNode()'s "goalState == DROPPED -> return early"
idempotency check doesn't fire and it redundantly reruns the whole
removal. The standby side is a harmless no-op re-write, but on the
primary side RemoveNode()'s "if ProceedGroupState(primary) didn't
change its goal, force APPLY_SETTINGS" fallback fires on this second,
redundant call and pushes a primary that had already correctly
settled at "single" into "apply_settings" for no reason. Not fixed
here -- the fix is moving the node lookup to happen after the lock
(inside RemoveNode(), taking a nodeId instead of a resolved
AutoFailoverNode*) -- documented in the spec's header for whoever
picks it up.
- concurrent_remove_standby_and_primary_report.spec: the Bug C spec.
Tests whether the primary's own node_active() call -- what its real
keeper sends on every ~1s poll, independent of any drop -- can be
tricked into the wrong decision by blocking behind a concurrent
remove_node() on the standby and resuming after it commits.
NodeActive() has the same "read before lock" shape as
remove_node_by_nodeid(). Result: ruled out. The primary's node_active()
correctly returns "single" every time, because the decision depends
on AutoFailoverNodeGroup()'s nodesCount, a fresh SPI query issued
after LockFormation is acquired, not on the stale pre-lock read (which
only affects the calling node's own previously-known fields, not
group membership). This is different from remove_node()'s bug, which
trusts a stale field for its idempotency check specifically.
Net effect on the Bug C investigation (see drop_node.sql and this
session's ~13 Docker reproductions, ~15% hit rate): RemoveNode()'s own
synchronous sequencing is proven correct (drop_node.sql), and this rules
out the primary's own concurrent poll as the trigger for the observed
"wait_primary" hang. The next concurrency shape to test is whatever could
cause more than one remove_node() call for the same node from a single
"pg_autoctl drop node" invocation -- the only shape proven so far to
produce a wrong primary goal state, even though the wrong state it
produces (apply_settings) isn't yet a proven match for the "wait_primary"
symptom from CI.
RemoveNode() used to take a pre-resolved AutoFailoverNode*, looked up by its caller (remove_node_by_nodeid/remove_node_by_host) via GetAutoFailoverNodeById()/GetAutoFailoverNode() *before* RemoveNode() acquired LockFormation(ExclusiveLock). Two concurrent remove_node() calls on the same node would serialize on that lock correctly, but the second call, once unblocked by the first's commit, was still working from the snapshot it read before it ever tried to acquire the lock -- stale by exactly the transaction it had just waited on. That broke the "goalState == DROPPED -> return early" idempotency check: the second call's stale currentNode still showed the pre-drop goalState, so it skipped the early return and redundantly redid the entire removal. The standby side was a harmless no-op re-write, but the primary side was not: RemoveNode()'s "if ProceedGroupState(primary) didn't change its goal, force APPLY_SETTINGS" fallback fired on that second, redundant call -- the primary was already correctly "single" by then, so its goal genuinely didn't change on the re-run -- and pushed a primary that had already settled at "single" into "apply_settings" for no reason. Fix: RemoveNode() now takes a nodeId instead of a pre-resolved pointer. It does an initial unlocked lookup only to learn which formation to lock (a node's formationId is immutable, so that read being racy is harmless), acquires LockFormation, and only then does the authoritative re-read of the node that all decision-making is based on. A caller blocked behind a concurrent RemoveNode() call on the same node now resumes with fresh, post-commit state instead of whatever it read before it ever tried to acquire the lock. remove_node_by_nodeid()/ remove_node_by_host() keep their own unlocked existence checks (for a precise "node not found" error message); a benign TOCTOU race there is already handled by RemoveNode() itself, which now returns true (already gone) if its own fresh lookup finds nothing. concurrent_remove_node.spec, which caught this, is updated in place to document and guard the fixed behavior instead of the bug: crn_p now stays "single" instead of drifting to "apply_settings". Verified: all 10 REGRESS + both ISOLATION specs pass via `make installcheck` against a fresh local instance.
concurrent_remove_standby_and_standby_report.spec (new) asks whether the node being dropped's OWN concurrent node_active() call -- its real keeper's regular ~1s poll, re-reporting its last known role, unaware a drop is in flight -- sees the fresh goalState = DROPPED once it blocks behind a concurrent remove_node() and resumes after that transaction commits. It does not, and this is the actual root cause of Bug C (citus_nonha_operation.pgaf's intermittent test_008_remove_old_primaries stall, ~15% hit rate across this session's Docker reproductions). NodeActive() reads the node via GetAutoFailoverNodeById() before acquiring LockFormation()/LockNodeGroup(), the same pre-lock-read shape already fixed in RemoveNode() (previous commit). A node's own node_active() call that blocks behind a concurrent remove_node() on itself resumes, once unblocked, still using the struct it read before ever trying to acquire the lock -- stale by exactly the transaction it was waiting on. ProceedGroupState()'s "already dropped, do nothing" checks compare against that stale goalState/reportedState and never fire. Execution falls through to ordinary FSM logic, where AutoFailoverNodeGroup()'s nodesCount -- a fresh SPI query, unaffected by the staleness -- correctly sees a one-node group (the dropped node's own row already excluded by its just-committed goalState = 'dropped'), and the nodesCount == 1 case reassigns the node's OWN goal to SINGLE: its own concurrent report undoes the drop it had just been given. This exactly reproduces the original CI log line for the node being dropped, "New state for this node ...: single -> single", instead of ever reaching "dropped -> dropped" -- which is why "pg_autoctl drop node" timed out waiting for a row that could never disappear. (concurrent_remove_standby_and_primary_report.spec, committed earlier this session, asked the same question about the PRIMARY's concurrent poll instead and ruled it out: the primary's decision depends only on the fresh nodesCount, not on any of its own stale fields, so it isn't susceptible to this. Its header comment is updated to point at this spec's result instead of leaving the "still open" note stale.) Fix: NodeActive() now re-reads the node after acquiring LockFormation()/LockNodeGroup(), mirroring the RemoveNode() fix -- same root defect (decide-before-lock instead of decide-after-lock), same remedy, second place it was hiding. Verified: all 10 REGRESS + all 3 ISOLATION specs pass via `make installcheck` against a fresh local instance. Docker-level re-verification of citus_nonha_operation.pgaf with this fix was attempted but the run was confounded by a transient, self-healing local Docker networking blip (a live TCP timeout connecting to the monitor, gone seconds later) unrelated to this fix, matching the same class of environmental flakiness flagged earlier this session before a Docker Desktop restart -- not treated as a real data point either way. The isolation-test reproduction above is deterministic and doesn't depend on Docker networking at all, which is why it's the basis for this fix rather than the Docker run.
…ess test Investigates the 6th CI failure: pytest/monitor PG16, test_003_002_stop_primary, "node3 failed to reach prepare_promotion or report_lsn after 90 seconds". Initial hypothesis was a BuildCandidateList() bug: a primary hard-killed via DataNode.fail() never reports its own demotion, so its reportedstate stays frozen at "primary" while only its goalstate advances to "demoted" via the monitor's own timeout. The "skip this node, it's a primary (old or new)" branch in BuildCandidateList() only excludes a node from the skip when its *reportedstate* (not goalstate) is draining/demoted, so a node frozen at reportedstate=primary/goalstate=demoted matches the skip and is dropped from missingNodesCount entirely -- suspected to let the sole remaining standby be promoted without real quorum. Added src/monitor/sql/stale_primary_report.sql to test this directly: manufacture two consecutive dead primaries in exactly that state (reportedstate frozen at primary, goalstate=demoted) and check whether the sole surviving standby gets incorrectly promoted. It does not: the separate quorumCandidateCount < minCandidates guard in ProceedGroupStateForMSFailover (minCandidates = number_sync_standbys + 1) still correctly blocks promotion regardless of the missingNodesCount bookkeeping gap. Kept as permanent regression coverage for this edge case. Rereading the actual pytest exception clarifies the real bug: node3 "failed to reach" report_lsn/prepare_promotion within 90 seconds -- it never reached further than expected, it was simply slow to get there. The monitor events log confirms the FSM did eventually reach the correct state (report_lsn -> prepare_promotion -> stop_replication -> wait_primary) after the pytest assertion had already timed out. This matches a timing budget problem, not a promotion-safety bug: reaching this point requires health-check detection of node2's death (node_considered_unhealthy_timeout, 20s default) plus a full primary_demote_timeout (30s default), and test_003_001 immediately before this test already drove node1 through the identical sequence, so a loaded CI runner can push the combined wait past the default 90s STATE_CHANGE_TIMEOUT. Bumped the three affected waits in test_003_002_stop_primary to timeout=120, matching the margin already used for the equivalent waits in test_005_001/test_005_002 elsewhere in this file. Locally reproduced via `make -C tests run-test PGVERSION=16 TEST=test_multi_alternate_primary_failures` (Docker, same as CI): both runs passed 15/15, consistent with this being a low-probability CI-only timing flake rather than a deterministic failure.
pgaftest indent already detects and strips a sequence{} block when it
exactly matches step declaration order (cli_indent.c: "Emit the sequence
block only when it differs from definition order... the block is
redundant noise; drop it"), and test_runner.c falls back to declaration
order whenever no sequence block is present at all. Most specs already
rely on this default and never write an explicit sequence.
Six specs still carried an explicit sequence block added when they were
authored, none of which was ever actually needed: in every case the listed
order is identical, step for step, to the step declaration order already
in the file. Verified with `pgaftest indent`, which independently confirms
each file's sequence is redundant and drops it when re-serializing.
Removed the redundant sequence{} block from:
- debug_citus_worker_switchover.pgaf
- debug_failover_pg19.pgaf
- drop_node_destroy.pgaf
- fast_forward.pgaf
- guard_data_loss.pgaf
- launch_deferred_set_metadata.pgaf
No behavior change: all six now rely on the same declaration-order default
already used elsewhere. Confirmed each file still parses cleanly via
`pgaftest indent` after the edit.
dimitri
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Continues
pgaftest(the Docker-based DSL test runner merged in #1137) withnew runner functionality and DSL commands, and bundles a set of
monitor/pg_autoctl correctness fixes found while using the tool to
reproduce and diagnose flaky/failing CI runs. Also migrates most of the
Python test suite to
.pgafspecs and rewrites the pgaftest docs.pgaftest tool
dockeruser,sudo, andentrypoint.sh; run the setup container as root so the Docker-socketbind mount works without GID-matching games on macOS/CI
docker exec monitor pg_autoctl ...round-trips with direct calls over the existing LISTEN connection(
monitor_get_node_state,runner_promote_one, newrunner_perform_failover,exec_sql_on_service)perform failoverDSL command (untargeted failover; monitorpicks the best secondary), distinct from
promote nodeXcluster/tmuxsubcommands,show state|step,pgaftest indent(canonical spec formatter, plus a couple of realformatting-corruption fixes in it),
helpteardown for the setup container, periodic SQL poll inside the LISTEN
loop
Monitor / pg_autoctl fixes found via pgaftest
Several of these came directly out of using pgaftest/pytest to chase down
CI flakes:
NodeActive()/RemoveNode()stale-read races — newconcurrent_remove_node/concurrent_remove_standby_and_*isolationsuite added to reproduce and pin down the race, then fixed
coordinator_update_node_preparecurrent_statecolumn-count guard fixed (16, not 17)lock_timeout/statement_timeoutset beforemaster_update_nodewaitpid()afterpg_ctl stopto reap the postgres zombieenable maintenance --allow-failoverkeeper_pg_init: init the monitor connection before checking whetherthe node was dropped
New regression/isolation coverage for these:
src/monitor/sql/drop_node.sql,src/monitor/sql/stale_primary_report.sql,and three new isolation specs
(
concurrent_remove_node,concurrent_remove_standby_and_primary_report,concurrent_remove_standby_and_standby_report), all passing locally(11/11 REGRESS, 3/3 ISOLATION).
Test spec migration
tests/tap/specs/*.pgafpromote nodeX(targetsnodeX) was being confused with
perform switchover(hands off fromthe current primary) — corrected throughout
basic_citus_operation.pgaf,citus_nonha_operation.pgaf)test_003_002_stop_primary's timeout (90s → 120s): the PG16 CIflake there was a timing budget issue, not a promotion-safety bug
(confirmed by the new
stale_primary_report.sqlregression test)sequence{}blocks from 6 specs where theorder already matched step-declaration order (the tool's default)
Docs
Rewrote
docs/ref/pgaftest.rst, addeddocs/testing.rstanddocs/reporting-bugs.rst, expanded the pgaftestREADME.md.Testing
basic_operation.pgaf27/27 PASS,multi_standbys.pgaf27/27 PASSsrc/monitorregression + isolation suites: 11/11 + 3/3 PASS locallytest_multi_alternate_primary_failures.pyreproduced locally viamake run-test(Docker, matches CI) — 15/15 PASScitus_indent --check) and banned-API check(
ci/banned.h.sh) both cleanFollow-up
PR #1149 (issue #997 monitor stall fix) will be rebased on top of this
once merged.
Note on diff size
A large share of the reported diff (~3,600 lines, ~34%) is the
bison/flex-generated
test_spec_parse.c/.handtest_spec_scan.c, whichfully regenerate — including internal line/state numbering — from small
edits to the 51-line
test_spec_parse.ygrammar and 6-linetest_spec_scan.llexer. These are committed so the tool builds withoutbison/flex installed; the actual hand-written diff is much smaller than
the raw line count suggests.