backlog: file #1096, the windows-2025 Tests (pytest) cap is exceeded on main - #278
Merged
Merged
Conversation
…on main ci.yml sets step_timeout 36 for both Windows legs and, in the same comment block, names its own re-derivation trigger: re-derive if a windows-2025 Tests (pytest) step is ever seen above 28:00. Measured 2026-08-07 over all 80 ci.yml runs created that day, jobs fetched with filter=all so an attempt killed at the cap is not hidden behind its passing re-run, timing the STEP and filtering on step conclusion: windows-2025 Tests (pytest) max passing 35:13 n=65 steps / 43 success cap 36:00 -> 47 seconds of margin, 1.022x passing steps above the 28:00 trigger: 43 of 43 killed at 36:0x: 5, two of them push runs on main (7ecff8a, b78214f) So the trigger is not merely fired, it is universally exceeded, and the cap is failing green suites. ci.yml already records this shape once at 26:00 (PR #119, killed at 26:07 with zero tests failing, diagnosed there as coin-flipping against the cap) and already admits 36:00 failed its own spread rule when it was chosen, needing roughly 37:00 to hold. Measured the step rather than the job deliberately: ci.yml records at least three sessions substituting job durations while triaging this same cap, and a job-duration reading of this pool gives 33-40 minutes, which invites the wrong conclusion that the cap is comfortable. job_timeout must be re-derived in the same act. The job carries two step_timeout-gated pytest steps and its 46 is sized against the observed sum, not 2 x step_timeout, so raising step_timeout alone moves the kill to the job cap, where no step conclusion is reported and the instrument used to measure this is lost. Ledger mechanics, stated separately because they are independent claims: - Number allocated with scripts/coord/alloc.ps1, not grepped. Row added in this same commit at rank 6, ranks 6..108 renumbered to 7..109. - The renumber was bounded to the live table (lines 180..287). The file also holds a superseded 134-row 2026-07-10 table; an assertion checks it comes out byte-identical, and it does. - The four Distribution census lines were re-derived, not carried forward: Value 7 -> 8, Difficulty 3 -> 40, P1 -> 9, quick win -> 34, sum 108 -> 109. - The "Every one of the 102 open items" sentence was left ALONE on purpose. It is a dated claim about the 2026-08-03 re-scoring pass, not a live census, and bumping it would assert that pass scored items that did not exist yet. Prior passes appear to have bumped it; that looks like the actual source of its drift. MY FILING IS CORRECT AND THE CENSUS IS STILL WRONG -- these are separate claims and this commit only makes the first. Verified in both directions with parse_items imported from backlog_status_check.py rather than a hand-rolled scan, which surfaces PRE-EXISTING drift this commit does not touch: 52 open items carry no row, 11 rows name an item that is not open, and parse_items counts 150 open items against a census asserting 109. Those two sets cancel on a total, which is why a matching sum never caught it. That reconcile is its own pass and needs 52 items scored. Gates run locally: backlog_status_check.py --min-items 150 OK (366 items, each declaring exactly one status), ledger_check.py --ci OK.
… wrong instrument #1084 was filed the same day, for the same leg and the same cap, and neither item referenced the other. It is not a harmless duplicate: the two CONTRADICT each other on the central action. #1084 says "Do NOT simply raise the cap"; #1096 says the cap must be re-derived. Left alone, whichever is worked first leaves the survivor standing as an open item asserting the work was wrong. The collision was invisible from this worktree. #1084 is not on main -- it lives only on origin/claude/gate-3d-remedy-1057 (PR #261, open) -- so reading docs/BACKLOG.md here shows no conflict, and backlog_status_check cannot see one either. The cross-reference is therefore labelled FORWARD-LOOKING in the item, per the rule that a citation which does not resolve yet must say so. The contradiction resolves, and #1084 is not wrong in spirit -- its premise is wrong. Its "~92% of step_timeout, about 17% headroom" comes from comparing JOB wall time across three PRs (33m09s / 37m06s / 38m20s) against a STEP cap. On the step the figure is 35:13 against 36:00 = 97.8% of budget, 2.2% headroom. That is the same job-for-step substitution ci.yml already records at least three earlier sessions making while triaging this very cap. That changes what "protect the diagnostic" implies, which is the whole disagreement. #1084 is right that step_timeout sits under job_timeout so a deadlock below pytest surfaces as a step failure, and right that raising a cap to buy a green tick would trade that away. But at 2.2% headroom, against a distribution where EVERY passing run exceeds the file's own 28:00 re-derive trigger, the cap fires on a healthy suite as readily as on a deadlock -- so re-deriving it RESTORES the diagnostic rather than surrendering it. Also absorbed from #1084 so closing it loses nothing: its three-wrong-answers triage, the step-under-job rationale, its fix directions (margin reporting is the cheapest and should likely happen regardless; splitting the Windows leg; the measured 51% cut on tests/test_worktree_gate_remedy_families.py from removing 30 of 36 git spawns), and its #1000 negative-control requirement. Added stronger evidence than either item had alone: a SAME-COMMIT flip. Run 31192717684 (fix-1006-absence-mutation, head 27d7746) failed at 36:07 on attempt 1 and passed at 33:14 on attempt 2. #1084 inferred a flip from three DIFFERENT PRs, which is not identical content; this is. The 43-of-43 figure is NOT rewritten to 45. An independent re-derivation later the same day reproduced 35:13 to the second and the same five kills, with the pool grown to n=68 and 45 of 45 above the trigger. It is recorded as a second measurement with its own pool rather than folded in, because a number edited away from the pool that produced it is exactly the rot ci.yml warns about. Recommended disposition stated in the item: keep #1096, close #1084 as merged into it, correct #1084's numbers rather than carrying them forward. There is no allocation collision -- 1084 and 1096 are different numbers, only the subject collided. Gates: backlog_status_check.py --min-items 150 OK (366 items, each declaring exactly one status), ledger_check.py --ci OK. Live table still 109 rows with contiguous ranks; #1096 still parses open via parse_items.
Plain APPEND conflict in docs/BACKLOG.md: my side adds ## 1096., main's side adds ## 1097. (landed via PR #282). Neither touches the other's text and the overlapping-heading-number count between the sides is ZERO, measured. Kept both, ascending: 1096 then 1097. THIS CONFLICT WAS ENGINEERED TO LAND HERE RATHER THAN ON #282. Both PRs append to the same file tail, so whichever merged second was always going to conflict. #282 was armed FIRST and deliberately: the ledger gate keys on which worktree ALLOCATED a number, #1097 belongs to nice-payne-4dcee0, and that session has now stopped -- so a conflict landing on #282 would have been unresolvable by anyone still running. #1096 is allocated to THIS worktree, so the conflict is resolvable here. Ordering was the whole point; it was not luck. Verified after resolution: ## 1094. x1, ## 1095. x1, ## 1096. x1, ## 1097. x1, conflict markers 0, CRLF preserved. My #1084 reconciliation section survived intact, as did #1096's table row and the census line reading 109. Live table still 109 rows with contiguous ranks. backlog_status_check.py --min-items 150 OK at 368 items, each declaring exactly one status. LEDGER DRIFT NOTE, stated rather than quietly absorbed: #1097 arrived with no table row, so the gap between parse_items' open count and the census table grows by one. That is the pre-existing drift already recorded in #1096's own commit (52 open items with no row, 11 rows naming a closed item, the two sets cancelling on a total). This merge does not fix it and does not make it worse by any mechanism of mine -- it simply carries it forward, which is what a correct filing does. The reconcile pass remains outstanding and needs 52 items scored.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Files one item. No code, no workflow change -- the fix itself is deliberately not in this PR.
What was measured
ci.ymlsetsstep_timeout: 36for both Windows legs and, in the same comment block, names its ownre-derivation trigger: "RE-DERIVE IF a windows-2025
Tests (pytest)step is ever seen above 28:00 --that is the trigger, not a calendar reminder."
pushruns onmain(7ecff8ae,b78214f5)The trigger is not merely fired, it is universally exceeded, and the cap is failing green suites --
including on
main.The step was measured, not the job, on purpose.
ci.ymlrecords at least three sessionssubstituting job durations while triaging this same cap; a job-duration reading of this pool gives
33-40 minutes and invites the wrong conclusion that the cap is comfortable.
Two failures on this leg today are DIFFERENT causes
Kept separate in the item rather than merged into one story: #253 is a genuine timeout with no test
failing, while #256 is an assertion failure in
test_connscale_cpu_probe.py/test_connscale_smoke.pywhose own comment already records this leg failing twice in one job on 2026-07-30. Only the first belongs
to #1096.
Ledger mechanics
scripts/coord/alloc.ps1, not grepped. Row added in the same commit at rank 6; ranks6..108 renumbered to 7..109.
2026-07-10 table; an assertion checks it comes out byte-identical, and it does.
sum 108 -> 109.
about the 2026-08-03 re-scoring pass, not a live census, and bumping it would assert that pass scored
items that did not exist yet. Prior passes appear to have bumped it, which looks like the source of the
drift it is known for.
This PR does not fix the census, and says so
Verified in both directions with
parse_itemsimported frombacklog_status_check.pyrather than ahand-rolled scan. That surfaces pre-existing drift this PR does not touch: 52 open items carry no
row, 11 rows name an item that is not open, and
parse_itemscounts 150 open items against acensus asserting 109. Those two sets cancel on a total, which is why a matching sum never caught it.
That reconcile is its own pass and needs 52 items scored.
Verification
A docs-only PR is the blind mode here, not the cheap one: the doc-drift guards live in pytest gated on
code == 'true'and.mdis in thenoncodeallowlist, so they skip pre-merge and fire on the push tomainafterwards. Run locally instead, against the worktree venv:pytest tests/test_asvs_file_surface_doc_drift.py ... tests/test_backlog_status_check.py-- 152 passed,89 skipped (all skips are
docs/security/THREAT-MODEL.md, vault-only)backlog_status_check.py --min-items 150-- OK, 366 items each declaring exactly one statusledger_check.py --ci-- OKAuto-merge deliberately NOT armed.
docs/BACKLOG.mdis the one writer the coordinator playbook saysto serialise, and this renumbers 103 row lines -- landing it ahead of merge trains #274/#275 risks turning
them DIRTY. It gets armed once those land.