ci: re-derive the Windows and ubuntu step and job caps from measurement (BACKLOG #1096) - #289
Merged
Merged
Conversation
Six values, one act, because the two caps are coupled and moving one alone relocates the kill rather than removing it. ubuntu-latest step 19 -> 25 job 26 -> 37 windows-2022 step 36 -> 55 job 46 -> 66 windows-2025 step 36 -> 55 job 46 -> 66 Pool: every run of this workflow created 2026-08-01T00:00Z .. 2026-08-09T00:00Z, 688 runs, enumerated over a wider page range and filtered locally on created_at. Jobs via ?filter=all so an attempt killed at the cap is not hidden behind its passing re-run. Rows are each leg's Tests (pytest) STEP, kept on that STEP's conclusion, executions under 60s dropped. 1,618 leg-executions. UBUNTU WAS ALREADY BROKEN AND NOBODY HAD MEASURED IT. Its worst-case job wall against measured maxima stood at -0:43, meaning the JOB cap could fire before the STEP cap. A job-level kill reports no step conclusion, so that failure destroys the instrument this work depends on, exactly when it is needed. This was filed as a Windows-only problem. ONLY WINDOWS-2025 IS RIGHT-CENSORED, and that is measured rather than assumed: the largest ubuntu step FAILURE in 535 executions is 14:21 and the largest windows-2022 failure in 542 is 25:48, so neither leg has ever touched its cap and their maxima are true. windows-2025 was killed at the cap seven times in this window, three of them push runs on main, the earliest on 2026-08-04. So 35:47 is the largest step that FIT, not the largest the suite wants, and every ratio taken against it flatters itself. THE KILLS ARE SLOWNESS, NOT A WEDGE. Run 31149117314's pytest finished green at 35:48 and the step was killed 4.85 seconds later; no faulthandler native-stack dump appears in any kill log. That settles a question the file previously left open, and it is why the killed rows are treated as genuine population members. The step cap is sized on windows-2025 and the JOB cap on windows-2022, which has the worse setup (4:05 against 2:32). The file previously asserted both were sized on windows-2025; that sentence was false for the job cap and is corrected. The job cap is no longer a band over the whole sum. Banding step_timeout applies slack to a bound the runner enforces exactly -- the largest step overrun measured in 1,618 executions is eight seconds -- so the band now applies to the measured job overhead only. The 28:00 re-derive trigger is retired for 40:00 (step) and 50:00 (job). It had become universally exceeded, including by the median, which makes it a permanently-tripped alarm and indistinguishable from no alarm. Sized against the POST-#1027 population, not what runs today: PR #253 moves the web console suite inside Tests (pytest), so sizing on the current population would let it land and re-break the cap on contact. WHAT THIS DOES NOT FIX, stated because the number looks like a solution and is not: all three legs' medians are rising monotonically, Windows at +1:37 to +2:06 per day. At that rate the new trigger fires within days. This buys a working merge gate, not a stable one. The real fix is BACKLOG #320, not a larger integer. Collateral corrected in the same act so no contradicting sentence survives: two nesting figures and a web-console maximum in #344, a "2 x step_timeout" figure in #1096's own body, two now-stale ci.yml line citations, and an open ADR 0158 checkbox that asked precisely the question this measurement answers. Derived by an eight-agent workflow whose three adversarial verifiers refuted the first proposal; the numbers here are the corrected ones, raised from 48/60 after the sizing anchor was shown to be pre-#1027 and ubuntu shown to be negative. Verified: ci.yml parses; backlog hygiene 19 passed; the repo-wide link gate reports 5,359 links across 347 files all resolving.
wshallwshall
enabled auto-merge (squash)
August 8, 2026 14:19
wshallwshall
added a commit
that referenced
this pull request
Aug 8, 2026
…its base (#290) Observed failing on CI, not hypothesised. PR #289's windows-2022 leg died with [Errno 13] bind on ('127.0.0.1', 62748): [winerror 10013] after 10,770 tests passed, with the step at 27:00 against a 55:00 cap. Not a timeout, and not the throughput SLO already filed as #1101. tests/test_connscale_smoke.py probes ONE free API port via _free_port(), which binds port 0, reads getsockname(), and closes the socket before returning. harness/load/connscale/runner.py:162 then binds api_port + step for every sweep arm. The failing run used 62746, 62747 and 62748; exactly one of the three was ever checked. The symptom hides the cause. A Windows bind onto a held port reports 10013, not the 10048 that reads as a collision, so it presents as an access-permissions problem and invites fixes that touch nothing relevant. The port number is the tell: 62748 is ephemeral, far outside the test's own inbound window [20000, 30000), so it is not the family #1014 reserved. This is #1014's defect one port-family over. #1014 gave the INBOUND block a contiguous reservation, a random anchor, a contiguity assertion at the acquisition site, and a loud failure instead of a silent fallback. None of that reached the API family. The test's comment reasons about the API ports only against the inbound block -- "The sink/API ports stay ephemeral (above the inbound window) and won't hit the block" -- which is true, and silent about the increment range colliding with anything else on the machine. The sink family carries the identical pattern at runner.py:278 and is dormant only because the smoke test passes sink_ports=1. Any profile raising it inherits this item. The fix should cover both families. #1014 removing the flaky(reruns=2) marker is why this is visible at all, and that was correct: its stated intent was that a genuine future collision surface as a RED rather than a masked retry. This is that collision, in the family it did not cover. The fix must not be a re-added retry. Records the mechanism, why the symptom misleads, the dormant sink instance, the shape a guard must take (assert the whole reserved range, not the base), and the separation from #1096 and #1101 with the measured step time. Number allocated via scripts/coord/alloc.ps1, never grepped. Verified: banner invariant holds, backlog hygiene 19 passed, the repo-wide link gate reports 5,359 links across 347 files all resolving.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements BACKLOG #1096. Six values, one act.
step_timeoutjob_timeoutWhy six values and not the two the item asked for
ubuntu was already broken, and it had never been measured. Its worst-case job wall against measured maxima stood at −0:43 — the JOB cap could fire before the STEP cap. That failure reports no step conclusion, so it destroys the instrument this item was measured with, precisely when it is needed. #1096 was filed as a Windows problem. Publishing "ubuntu unchanged, still positive" beside a corrected Windows row would have been a compensating control resting on a false premise.
The two caps are coupled. Raising
step_timeoutalone relocates the kill to the job cap rather than removing it.job_timeout 46could not absorb any step raise: the windows-2025 non-Tests addends measure 5:16, so 46:00 already implied a 36:00 step cap exactly.The measurement
Pool: every run of this workflow created 2026-08-01T00:00Z .. 2026-08-09T00:00Z — 688 runs, enumerated over a wider page range (1,000 runs) and filtered locally on
created_at. Jobs via?filter=all, so an attempt killed at the cap is not hidden behind its passing re-run. Rows are each leg'sTests (pytest)STEP, kept on that step's conclusion, executions under 60s dropped. 1,618 leg-executions.Only windows-2025 is censored, and that is measured rather than assumed. Largest ubuntu step failure in 535 executions is 14:21; largest windows-2022 failure in 542 is 25:48. Neither leg has ever touched its cap, so those maxima are true. windows-2025 was killed at the cap seven times, three of them push runs on
main, earliest 2026-08-04 — three days before the item was filed.The kills are slowness, not a wedge — checked, not assumed. Run
31149117314:pytest finished green and the step was killed 4.85 seconds later. That settles a question
ci.ymlpreviously left open, and it has a consequence that is easy to miss: the killed rows are genuine population members, so "max passing" is censored, not conservative.Two corrections to what the file asserted
ci.ymlsaid both Windows legs were "sized on windows-2025 as the worse of the pair". False for the job cap — windows-2022 has the worse setup (4:05 vs 2:32). The step cap sizes on W25, the job cap on W22.step_timeout— a bound the runner enforces exactly (largest overrun in 1,618 executions: 8 seconds). The band now applies to measured job overhead only.The 28:00 re-derive trigger is retired for 40:00 (step) and 50:00 (job). It had become universally exceeded — including by the median — which makes it a permanently-tripped alarm, indistinguishable from no alarm.
Sized against the post-#1027 population: PR #253 moves the web console suite inside
Tests (pytest), so sizing on today's population would let it land and re-break the cap on contact.What this does NOT fix
All three legs' medians are rising monotonically — Windows at +1:37 to +2:06 per day, five to six times ubuntu's rate. At that rate the new trigger fires within days and 55:00 is marginal within about a week. This buys a working merge gate, not a stable one. The real fix is #320, not a larger integer, and re-deriving this weekly is the failure mode to avoid rather than the plan.
#1096 is therefore marked partly shipped rather than closed.
Collateral, in the same act
Two nesting figures and a web-console maximum in #344, a
2 x step_timeoutfigure in #1096's own body, two now-staleci.yml:442citations, and an open ADR 0158 checkbox that asked exactly the question this measurement answers. The superseded tables are retained with their pools rather than deleted, because a number without its pool cannot be rechecked. The "43 of 43" figure is deliberately not rewritten — it is correct for its own 80-run pool and #1096 says so explicitly.Provenance and verification
Derived by an eight-agent workflow: three independent measurement passes, one derivation, three adversarial verifiers on distinct lenses, one synthesis. Two of the three verifiers refuted the first proposal. These numbers are the corrected ones — raised from 48/60 after the sizing anchor was shown to be pre-#1027 and ubuntu shown to be negative.
Run locally:
ci.ymlparses; backlog hygiene 19 passed; the repo-wide link gate reports 5,359 links / 347 files, all resolve; the workflow linter passes in pre-commit.