Skip to content

monitor: run pg_regress tests serially via a real schedule file#1151

Merged
dimitri merged 1 commit into
mainfrom
fix/pg-regress-parallel-schedule
Jul 22, 2026
Merged

monitor: run pg_regress tests serially via a real schedule file#1151
dimitri merged 1 commit into
mainfrom
fix/pg-regress-parallel-schedule

Conversation

@dimitri

@dimitri dimitri commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Problem

src/monitor's pg_regress suite is flaky in CI: sporadic not ok - upgrade or not ok - lock_and_fetch_migration failures, most recently on PR #1149's "Build run image (PG18)" job.

Root cause

src/monitor/Makefile's REGRESS variable is a flat, space-separated list. Passed to pg_regress with no --schedule, that whole list runs as one parallel group — but none of these tests were written to tolerate that:

  • Every test registers nodes via pgautofailover.register_node() against the single pgautofailover.node table (and its one shared nodeid sequence) in the shared contrib_regression database. Their expected/*.out files hardcode small literal node ids (e.g. assigned_node_id | 2) on the assumption that they have that sequence, and the table, to themselves. Run two of them concurrently and one test's inserts bump the other's node ids and leak extra rows into the other's select * from pgautofailover.node output.
  • dummy_update.sql and upgrade.sql additionally each run their own CREATE/ALTER/DROP EXTENSION pgautofailover against the single shared extension object — a second, independent way for two tests to corrupt each other if run at once.

Confirmed locally: my first attempt at a fix only isolated the three extension-owning tests (dummy_update, drop_extension, upgrade) into their own serial steps and left the other 8 in one parallel batch, on the assumption that they only read the extension. Running that surfaced the node-id variant of the race directly — proof it isn't limited to the extension-lifecycle tests.

Fix

Add src/monitor/regress_schedule, a real pg_regress --schedule file that puts every test from the original REGRESS list on its own serial test: line, in dependency order (create_extension first — everything else needs the extension to exist — then the functional tests, then dummy_updatedrop_extensionupgrade last, matching the extension-lifecycle dependency between them).

src/monitor/Makefile:

  • REGRESS_OPTS += --schedule=$(SRC_DIR)regress_schedule
  • installcheck is overridden to invoke pg_regress without also passing the bare $(REGRESS) list. pg_regress treats trailing bare test-name arguments as extra tests, run one at a time after the schedule finishes (pg_regress.c's extra_tests/run_single_test()), so passing both the schedule and the bare list would silently re-run every test a second time against a since-mutated database — not merely redundant, but wrong.
  • REGRESS itself is left in place, still listing every test: PGXS's ifdef REGRESS gates the installcheck/clean recipes entirely (verified locally — an empty-valued REGRESS makes ifdef REGRESS false and silently drops the whole recipe), and it still documents the full test set for readers.

check is untouched: under this PGXS setup make check already prints "not supported, use install + installcheck instead" (no functioning temp-instance path exists here), so there was nothing working to preserve.

Verified

make build-pg18 — the exact Dockerfile/pg_virtualenv path CI uses for the "Build run image" jobs:

ok 1         - create_extension                           25 ms
ok 2         - monitor                                    24 ms
ok 3         - workers                                    10 ms
ok 4         - node_active_protocol                       27 ms
ok 5         - guard_data_loss                            18 ms
ok 6         - fast_forward                               19 ms
ok 7         - drop_node                                  16 ms
ok 8         - stale_primary_report                       17 ms
ok 9         - lock_and_fetch_migration                   22 ms
ok 10        - dummy_update                               10 ms
ok 11        - drop_extension                              9 ms
ok 12        - upgrade                                    21 ms
1..12
# All 12 tests passed.
...
# All 6 tests passed.

All 12 REGRESS tests and all 6 ISOLATION tests pass, correctly serialized in the intended order.

Cost

Losing intra-suite parallelism adds a small amount of wall-clock time (each test is ~10-30ms; going from one parallel batch to 12 serial steps costs roughly a couple hundred milliseconds total) in exchange for a suite that was never actually safe to parallelize.

pg_regress runs an unscheduled (bare) REGRESS list as a single parallel
group. None of src/monitor/'s regression tests were written to tolerate
that: they all register nodes via pgautofailover.register_node() against
the one pgautofailover.node table (and its one shared nodeid sequence) in
the whole contrib_regression database, and their expected/*.out files
hardcode small literal node ids on the assumption that they have that
sequence -- and the table -- to themselves. dummy_update.sql and
upgrade.sql additionally run their own CREATE/ALTER/DROP EXTENSION
pgautofailover against the single shared extension object, a second,
separate way for two tests to corrupt each other if run concurrently.

Observed in CI as sporadic "not ok - upgrade" and
"not ok - lock_and_fetch_migration" failures (most recently on PR #1149's
"Build run image (PG18)" job), depending on which pair of tests the
scheduler happened to interleave that run. Confirmed locally: running the
old flat REGRESS list surfaced a *second*, previously-undiagnosed instance
of the same class of bug -- monitor/workers/etc. leaking node-id and
formation rows into each other when run in a naive "everything but the
three extension-owning tests" parallel group.

Fix: add src/monitor/regress_schedule, a real pg_regress --schedule file
that puts every test in the original REGRESS list on its own serial
"test:" line, in dependency order (create_extension first, then the
functional tests, then dummy_update -> drop_extension -> upgrade last).
Wire it in via REGRESS_OPTS += --schedule=..., and override installcheck
to invoke pg_regress without also passing the bare $(REGRESS) list --
pg_regress treats trailing bare test-name arguments as extra tests run
one at a time *after* the schedule finishes (pg_regress.c's
extra_tests/run_single_test()), so passing both would silently re-run
every test a second time.

REGRESS itself is left in place, still listing every test: PGXS's
`ifdef REGRESS` gates the installcheck/clean recipes entirely (verified:
an empty-valued REGRESS makes `ifdef REGRESS` false), and it still
documents the full test set.

Verified: `make build-pg18` (the exact Dockerfile/pg_virtualenv path CI
uses) -- all 12 REGRESS tests and all 6 ISOLATION tests pass, in the
expected serialized order.
@dimitri dimitri self-assigned this Jul 22, 2026
@dimitri dimitri added Developer productivity Enhancements to ability to ship quality code Packaging and CI Enhancements to our CI integration labels Jul 22, 2026
@dimitri
dimitri merged commit a72644e into main Jul 22, 2026
211 of 216 checks passed
@dimitri
dimitri deleted the fix/pg-regress-parallel-schedule branch July 22, 2026 11:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Developer productivity Enhancements to ability to ship quality code Packaging and CI Enhancements to our CI integration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant