The aux port walk cannot wrap, so a draw at the top of the band fails with 1,999 free ports under it — not exhaustion, and resizing will not fix it

[ChronicallyJD]

replication.sh fails with no free port for the restore on runs that cannot possibly have exhausted anything — most recently on #545, whose entire diff is CONTEXT.md. The aux band is not full. The walk cannot wrap, so a draw near the top of the band has almost nowhere to go.

The mechanism, exactly

pick_sb_port (test/replication.sh:65) seeds from $$:

base=$(( PGC_AUX_PORT_LO + ($$ % (PGC_AUX_PORT_HI - PGC_AUX_PORT_LO)) ))

The band is 2000 wide, so base is uniform over [LO, HI-1].

Both draws return the same port, by construction. $$ inside $(...) is the invoking shell's PID, not the subshell's, so SB_PORT="$(pick_sb_port)" and RS_PORT="$(pick_sb_port)" compute an identical base and — in an empty band — return the identical port. That is not a bug; the while loop is written expecting it:

while [ "$RS_PORT" = "$SB_PORT" ] || ! pgc_port_free "$RS_PORT"; do
        RS_PORT=$((RS_PORT + 1))
        [ "$RS_PORT" -ge "$PGC_AUX_PORT_HI" ] && { echo "FAIL  no free port for the restore"; ... }
done

The bug is that the walk hard-fails at HI instead of wrapping to LO.

So when base == HI-1: SB_PORT = HI-1, RS_PORT = HI-1, the loop increments to HI, hits the bound, and the suite fails — with 1,999 free ports underneath it.

P(base == HI-1) = 1/2000 per replication run

Five majors per CI run makes it roughly 1 in 400 CI runs, which matches "rare, unattributable, and it moves between majors."

The same shape in the shared allocator

pgc_pick_free_port (test/portlib.sh:145) has it too:

for p in $(seq "$base" $(( base + 300 ))); do
        [ "$p" -ge "$hi" ] && break        # <- no wrap

A draw in the top 300 of the band gets a truncated scan — 15% of draws — and at base = hi-5 it has five candidates before giving up. This one is currently masked because the band is usually empty and the first probe succeeds, but it is the same defect and it gets worse exactly when the band is busy, which is when you need it to work.

What it is not

It is not band sizing, and resizing would not fix it. The failure needs base within one port of the ceiling; that stays a fixed 1-in-N-of-the-width whatever the width is. Making the band 20,000 wide changes the rate, not the defect, and the walk would still fail with the whole band free beneath it.

It is also not contention: the reported failure came from a docs-only PR, and I could not construct a path from a CONTEXT.md change to a port.

Fix

Wrap, in both places. For the replication.sh walk:

_tries=0
while [ "$RS_PORT" = "$SB_PORT" ] || ! pgc_port_free "$RS_PORT"; do
        RS_PORT=$((RS_PORT + 1))
        [ "$RS_PORT" -ge "$PGC_AUX_PORT_HI" ] && RS_PORT="$PGC_AUX_PORT_LO"
        _tries=$((_tries + 1))
        [ "$_tries" -gt "$((PGC_AUX_PORT_HI - PGC_AUX_PORT_LO))" ] && {
                echo "FAIL  no free port for the restore in [$PGC_AUX_PORT_LO,$PGC_AUX_PORT_HI) after a full sweep"
                PGC_FAIL=1; pgc_summary; }
done

The counter matters as much as the wrap: without it a genuinely full band spins forever, and the message should distinguish "I swept the whole band" from "I walked off the end of it", because those want different responses from whoever reads the log. The current message asserts the first while only the second has been established — the same defect #537 was filed about, in a different file.

Same treatment for pgc_pick_free_port: wrap to lo rather than break, and bound the loop by the band width.

What I have not done

Not reproduced the failure end to end — it needs a run whose PID lands on exactly one value, and I have not tried to force it by launching with a chosen PID. The arithmetic above is from reading, and the mechanism predicts a rate consistent with the observed rarity, but I would rather say that than imply I have seen it fire.