From e735bb7627c319a05753fc95905108629a662bbb Mon Sep 17 00:00:00 2001 From: wshallwshall Date: Sat, 8 Aug 2026 11:31:53 -0500 Subject: [PATCH] backlog: file #1103, the connscale API port range is unverified past its base Observed failing on CI, not hypothesised. PR #289's windows-2022 leg died with [Errno 13] bind on ('127.0.0.1', 62748): [winerror 10013] after 10,770 tests passed, with the step at 27:00 against a 55:00 cap. Not a timeout, and not the throughput SLO already filed as #1101. tests/test_connscale_smoke.py probes ONE free API port via _free_port(), which binds port 0, reads getsockname(), and closes the socket before returning. harness/load/connscale/runner.py:162 then binds api_port + step for every sweep arm. The failing run used 62746, 62747 and 62748; exactly one of the three was ever checked. The symptom hides the cause. A Windows bind onto a held port reports 10013, not the 10048 that reads as a collision, so it presents as an access-permissions problem and invites fixes that touch nothing relevant. The port number is the tell: 62748 is ephemeral, far outside the test's own inbound window [20000, 30000), so it is not the family #1014 reserved. This is #1014's defect one port-family over. #1014 gave the INBOUND block a contiguous reservation, a random anchor, a contiguity assertion at the acquisition site, and a loud failure instead of a silent fallback. None of that reached the API family. The test's comment reasons about the API ports only against the inbound block -- "The sink/API ports stay ephemeral (above the inbound window) and won't hit the block" -- which is true, and silent about the increment range colliding with anything else on the machine. The sink family carries the identical pattern at runner.py:278 and is dormant only because the smoke test passes sink_ports=1. Any profile raising it inherits this item. The fix should cover both families. #1014 removing the flaky(reruns=2) marker is why this is visible at all, and that was correct: its stated intent was that a genuine future collision surface as a RED rather than a masked retry. This is that collision, in the family it did not cover. The fix must not be a re-added retry. Records the mechanism, why the symptom misleads, the dormant sink instance, the shape a guard must take (assert the whole reserved range, not the base), and the separation from #1096 and #1101 with the measured step time. Number allocated via scripts/coord/alloc.ps1, never grepped. Verified: banner invariant holds, backlog hygiene 19 passed, the repo-wide link gate reports 5,359 links across 347 files all resolving. --- docs/BACKLOG.md | 80 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 80 insertions(+) diff --git a/docs/BACKLOG.md b/docs/BACKLOG.md index a2c397ab..7d528637 100644 --- a/docs/BACKLOG.md +++ b/docs/BACKLOG.md @@ -6849,3 +6849,83 @@ no engine content. Causal exclusion first (the diff reaches no engine code; the `pyproject.toml` `addopts`, so the PR's new test collects **after** `test_connscale_smoke.py` and had not run), then reproduced under contention rather than argued. The investigating session retracted two of its own mechanisms, above, before the conclusion was accepted. + +## 1103. the connscale API port range is derived by increment from a single probed port, so every port after the base is unverified + +> 🔢 **Filed 2026-08-08 - not started. Observed failing on `main`'s own CI, not hypothesised.** Value **4/10** · Difficulty **2/10**. `tests/test_connscale_smoke.py` probes **one** free API port and `harness/load/connscale/runner.py:162` then binds `api_port + step` for every sweep step. Only the base was ever checked. A taken port anywhere in that range kills the engine at startup, and on Windows it surfaces as `WinError 10013` -- *access forbidden*, not the `10048` that reads as a collision -- so the failure does not look like a port problem at all. + +**Cluster:** Testing / harness reliability. **Priority:** P2. **Verdict:** build. **Severity:** no product +effect and no PHI effect. The cost is a blocking, required check failing for a reason unconnected to the +change under test, on a leg that already carries two other unrelated failure modes. + +**Value 4:** the Developer Experience & CI ladder caps there, and the workaround (re-run the leg) is real +if expensive. It is not a product defect -- `harness/` is test scaffolding. + +**Observed.** PR #289, `test (windows-2022, py3.14)`, run `31261658519`, job `93113333978`: + +``` +FAILED tests/test_connscale_smoke.py::test_connscale_smoke_end_to_end + ConnScaleError: engine exited during startup: + ERROR uvicorn.error: [Errno 13] error while attempting to bind on address + ('127.0.0.1', 62748): [winerror 10013] an attempt was made to access a socket + in a way forbidden by its access permissions +1 failed, 10770 passed, 830 skipped in 1608.73s (0:26:48) +``` + +**It is not a timeout and not #1096.** The `Tests (pytest)` STEP ran **27:00** against a 55:00 cap. It is +also **not #1101** -- that is a throughput SLO assertion, this is a bind failure before the engine +finishes starting. Three distinct failure modes now share this leg; do not merge them. + +**The mechanism, traced rather than inferred.** + +* `_free_port()` (`tests/test_connscale_smoke.py`) binds `("127.0.0.1", 0)`, reads `getsockname()[1]`, + and **closes the socket in a `finally` before returning**. The returned port is free at the instant it + is read and reserved by nothing thereafter. +* The test calls it once for the API base: `api_port = _free_port()`. +* `harness/load/connscale/runner.py:162` passes **`api_port=api_port + step`**, with `step` incremented + per sweep arm (`:199`, `:206`). The failing run's log carries `62746`, `62747` and `62748`. +* So exactly one of the three was verified. `62748` was taken, and the engine died at startup. + +**Why the symptom hides the cause.** A Windows bind onto a port held by another socket reports **10013**, +not **10048**. Read as an access-permissions error it invites the wrong fixes -- run CI elevated, adjust +firewall rules, blame the runner image -- none of which touch a port-allocation defect. The number is the +tell: `62748` sits in the ephemeral range, far outside the test's own inbound window `[20000, 30000)`, so +it is not the family `#1014` reserved. + +**This is #1014's defect, one port-family over.** #1014 fixed exactly this for the **inbound** block: a +contiguous reservation, a random anchor to de-correlate concurrent worktrees, contiguity asserted at the +acquisition site, and a loud failure rather than a silent fixed fallback. None of it was applied to the +API family. The test's own comment reasons about the API ports *only* relative to the inbound block -- +*"The sink/API ports stay ephemeral (above the inbound window) and won't hit the block"* -- which is true, +and silent about the increment range colliding with anything else on the machine. **A correct statement +about one hazard reads as coverage of a hazard it never mentions.** + +**The same pattern is in the SINK family and is dormant only by configuration.** +`harness/load/connscale/runner.py:278` builds `ports=tuple(sink_port + i for i in range(sink_ports))` +from a single probed `sink_port`. `test_connscale_smoke.py` passes `sink_ports=1`, so today only the +verified base is used and it cannot fire. Any profile raising `sink_ports` above 1 inherits this item +without touching it. Fix both families in the same act. + +**#1014 removing `@pytest.mark.flaky(reruns=2)` is why this is visible, and that was correct.** The +retry would have absorbed it. #1014's stated intent was that *"a genuine future collision now surfaces +as a RED rather than a masked retry"* -- this is that collision, in the family #1014 did not cover. + +**The work.** Reserve the API range the way `_free_contiguous_ports` reserves the inbound block: probe a +contiguous run of the required width, assert contiguity at the acquisition site, fail loudly if no block +is available, and prefer a random anchor to de-correlate concurrent worktrees. The width is derivable -- +it is the number of sweep arms, which the profile already determines. Do the same for the sink family. +Do **not** fix this by re-adding a retry: that re-hides the class #1014 deliberately exposed. + +**Test it against the range, not the base.** A test that probes one port and asserts it binds cannot see +this. The guard has to assert that **every** port the sweep will use was reserved, which is the same +shape as the contiguity assertion `_free_contiguous_ports` already carries for the inbound family. + +**Related:** #1014 (the same defect in the inbound family; its fix is the model for this one), #1101 +(the other connscale failure on this leg -- a throughput SLO, not a bind), #1096 (the third failure mode +on this leg -- the step cap; distinct again), #1000 (a control whose evidence could not see the class it +covered -- the comment quoted above is that shape in prose). + +**Source:** found 2026-08-08 while triaging a red `test (windows-2022, py3.14)` leg on PR #289, the +`#1096` cap change. Diagnosed by tracing the port number through `_free_port` and the `api_port + step` +call site rather than by re-running: the failing port lies outside the inbound window, which is what +rules out the family `#1014` already fixed and points at the one it did not.