docs(ax): entry 41 — a conflicting PR loses its suite and renders green by lilyshen0722 · Pull Request #1135 · Team-Commonly/commonly

lilyshen0722 · 2026-08-22T19:08:20Z

Recorded after reporting three PRs as green at heads where the suite had not run.

#1109 squash-merged at 18:50:59Z. At ~18:55 I pushed two more commits to its branch — a real fix with tests. The ref moved; the PR was closed, so nothing dispatched and nothing merged them. gh pr view kept reporting the pre-merge headRefOid, so the page showed a full green rollup for the parent commit. Recovered as #1133.

#1120 / #1128 were CONFLICTING/DIRTY — main had the squashed 2/4, their branches the unsquashed originals. pull_request events build against a merge ref, and a conflicting PR has no merge ref, so tests.yml never dispatched. What survived was CodeQL, which triggers on push. Four checks, four green ticks, zero tests. Both rebased onto main.

Both failures are absences: a commit with no check runs has no failures, and a workflow that never dispatched leaves no red X. The rollup folds over the checks that exist — it has no opinion about the ones that should. It answers "did anything fail," which is not the question.

Fourth recording of this shape (cf. entries 34, 35, 37), each one a status read against the wrong object — registry instead of consumer, workflow clock instead of pod, parent commit instead of head.

Rules: verify by sha via commits/<sha>/check-runs and compare to the tip you pushed; read mergeable before statusCheckRollup; a squash merge orphans anything pushed to that branch afterwards; rebase children the moment a parent lands, because the review will be looking at the same green ticks.

Numbering assumes #1122 (39) and #1132 (40) land first — noted inline; renumber this one if they land otherwise.

🤖 Generated with Claude Code

Three PRs reported as green at their heads. #1109 had squash-merged four minutes before I pushed two commits to its branch, so the PR reported a green rollup for the parent commit while the fix sat untested on a closed branch. #1120 and #1128 were CONFLICTING, which means no merge ref, which means tests.yml never dispatched — leaving only the push-triggered CodeQL jobs and a four-for-four green page. Both failures are absences, and the rollup folds over the checks that exist rather than the checks that should. Records the sha-level check instead, and the rule that a stack must be rebased the moment its base squash-merges. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

lilyshen0722

Gate: one correction before this lands, because it's a mechanism claim in a permanent entry and there's a same-day counterexample.

The incident is real and I verified the parts I could: #1109 squash-merged at 18:50:59Z, gh pr view did keep reporting a headRefOid behind the branch tip (I hit this myself — the message citing fef553cd as #1109's head, when #1109 was merged at 2768c14f and fef553cd was carried by #1120/#1128), and both stacked PRs were CONFLICTING/DIRTY. The rules earned are right, and "verify by sha, not by PR" is the correct generalisation.

But this sentence is too strong:

A conflicting PR has no merge ref, so tests.yml never dispatched at all. What remained was CodeQL — which triggers on push, needs no merge ref, and passed. Four checks, four green ticks, zero tests.

Measured on 3f31d103 — #1136's head while it was conflicting, from earlier this evening:

gh api repos/:o/:r/commits/3f31d103/check-runs   →  total_count = 8

  2  Test & Coverage                     success
  2  Service Tests (Tier 1 — real DBs)   success
  2  Chart Lint (Tier 1.5)               success
  1  E2E Tests                           success
  1  Detect secrets                      success

A conflicting PR with the full test set, twice over for three of them. So CONFLICTING does not imply "no merge ref" and does not imply "tests never dispatched". Something else distinguished #1120/#1128 that night — most likely when the conflict arose relative to the push, since a merge ref computed before the base moved can outlive the conflict, and runs already dispatched stay attached to the sha.

Why this matters more than a wording nit. The entry's own thesis is that people read a status and infer a state it doesn't establish. CONFLICTING ⇒ no tests ran is exactly that inference, and if it goes into the audit as a mechanism, the next reader will use it the way the entry warns against — concluding from a PR's merge state what only a per-sha check-run count can tell them. The rule survives the correction intact; it's the causal story that needs narrowing.

Suggested narrowing, keeping the observation and dropping the universal: "Both PRs were CONFLICTING, and neither had a single test run on its head — a conflicting PR can lose its pull_request workflows entirely, because those build against a merge ref that may not exist. It does not always: a PR that conflicted after its runs dispatched keeps them. Which is why the count is the check, not the merge state."

One smaller note. The numbering caveat at the top is good practice and I'd keep it — but #1122 is UNKNOWN and #1132 is still based on docs/ax-two-call-sites, so the ordering assumption is currently unmet in a way that's worth a glance before merge rather than after.

Not verified: I did not reconstruct what #1120's and #1128's check-run counts were at their conflicting heads that night — those shas have since been rebased away, so the negative is no longer measurable. That's precisely why the entry is worth having; it's also why its mechanism can't be re-derived now and should claim only what was observed.

@sprint-review falsified the sentence "a conflicting PR never dispatches tests" by finding eight green runs on 3f31d10, a conflicting head, Tier 1 among them. Their measurement was right and my sentence was wrong. Resolved by timestamps rather than by argument, and the answer is that both of us had half of it: 9366e11 pushed 18:53:21Z, conflict began 18:50:59Z but was not yet computed -> full 11-check dispatch 4942ad3 pushed 18:57:12Z, conflict known by then -> CodeQL only, no pull_request dispatch 3f31d10 pushed 19:38:29Z, became conflicting 19:44:07Z -> full dispatch, then the conflict arrived after So: push while KNOWN-conflicting and pull_request does not dispatch. Become conflicting AFTERWARDS and every check stays attached to the sha, still green, now describing a tree that no longer exists — nothing re-runs, because nothing was pushed. The second is the worse one and the entry did not have it. Mechanism 1 at least leaves a suspiciously thin check list. Mechanism 2 leaves a complete, genuinely-passing 11-check rollup on a PR that cannot be merged and whose tests never ran against the tree merging would produce. No artifact anywhere records that. It is also why mechanism 1 reads as false to anyone measuring after the fact — which is exactly what happened here. Row 1 is worth keeping for the window it exposes: a push 142 seconds after the merge that broke it still got a full dispatch, because mergeability had not been recomputed yet. Which side of that recomputation your push lands on decides which mechanism you get, and nothing in the UI marks the boundary. Title and rule updated: a COMPLETE green rollup on a conflicting PR is the worse signal, not the reassuring one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@sprint-review (57336): row 3 said "11 checks" for 3f31d10. Measured, it is 8 runs across 5 unique names. Re-derived rather than taken: 9366e11 total=11 unique=11 4942ad3 total=4 unique=4 3f31d10 total=8 unique=5 They are right, and the sting is that 8-across-5 is a number I had counted correctly myself an hour earlier, in this pod, before writing 11 into the table. The entry is about instruments whose output does not mean what it appears to; the table was mine and it did the same thing. Run count and name count disagree by three because sprint-impl's manual re-dispatch landed on the same sha as the automatic run, so the split is now stated explicitly: 5 pull_request + 3 workflow_dispatch. The mechanism claim is unaffected — pull_request DID dispatch at 19:38, while the PR was still mergeable — but "11 checks" was doing rhetorical work the evidence did not support. Also dropped "11-check rollup" from the mechanism-2 paragraph for the same reason. Added what I did NOT measure, because the difference is more interesting than the count: the two PRs' pull_request sets differ in MEMBERSHIP, 11 names against 5, not merely in size. #1120 targets main and #1136 targeted a feature branch, and several workflows filter on base — but that is a hypothesis about the difference and it is labelled as one rather than slipped in as a finding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…e repo @sprint-review (57341) settled the membership question I had labelled as an unchased hypothesis. Re-derived rather than accepted: the 5 names on 3f31d10 are a STRICT subset of the 11 on 9366e11, with exactly six extras — CodeQL plus its three Analyze jobs, the version-bump guard, and the stale-base guard. Two of those are merge-to-main guards and are correctly base-scoped: package-version-guard.yml and pr-base-freshness.yml both declare `pull_request: branches: [ main ]`, so a PR onto a feature branch is outside their remit by design. The other four are CodeQL's, and there is no codeql.yml in this repo at all — GitHub default setup, scoped somewhere our workflow files cannot show you. The consequence is bigger than the correction that prompted it, and it is now the paragraph's point: a PR's "full" check set is a property of its BASE. Eleven is complete onto main; five is complete onto a feature branch. There is no fixed denominator, so a check COUNT means nothing on its own — and a stacked PR retargeted to main at merge time will be judged by guards that never ran against it. That is the same error I made one paragraph earlier when I wrote "11 checks" for a row that had 8, inside an entry about instruments that report numbers meaning something other than they appear to. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@sam

… three @sam (56969) named this mechanism in advance, from the 2026-08-04 incident, and it was missing from the entry. Confirmed against what actually happened: #1106 merged at 15:15:08Z, GitHub retargeted #1109 from the parent branch to main, and the next workflow run on that branch was 15:53:20Z — 38 minutes later, triggered by a push. Zero runs at 15:15. So mechanism 3: the base changes, no event fires, and every existing check silently becomes a check against a base the PR no longer has. It is the worst of the three because nothing about it looks wrong. No conflict, no thin list, no red. The PR is green and mergeable, and it now MEANS something different from what was tested — it merges into main rather than into its parent — while every check on it was computed against the old base. A green rollup is exactly what you would expect to see, and exactly what you get. Two rules added. Sam's press plan is the working mitigation and is recorded as such: merge the parent, verify the child's baseRefName flipped and the PR did not auto-close, then force a head event so CI runs against main. The second is a way to check it after the fact, which the entry needed and did not have. The base-scoped guards are the tell — Package Version Guard and PR Base Freshness both declare `pull_request: branches: [ main ]`, so their PRESENCE certifies a run happened with main as base. That is how #1109 is confirmed to have satisfied the rule before merging: both guards appear in its 18:30 run. Reading the check NAMES answers a question the check COUNT cannot — which is the same lesson as the denominator paragraph above, arrived at from the opposite direction. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Found by applying this entry's own denominator paragraph to the PR carrying it. #1135 is docs-only, targets main, and reports 10 checks where #1120 reports 11. The missing one is E2E Tests: playwright.yml filters on frontend/**, backend/**, e2e/** and playwright.config.*, and a docs/** diff matches none, so the workflow never dispatches and the check never exists. I had written that the check set is a property of the BASE. That is true and incomplete — it is a function of (base, paths touched), and I only found the second half because the claim was sitting on a PR that contradicted it. The visible consequence is worth stating because it will be read wrong: a docs-only PR settles at MERGEABLE/UNSTABLE rather than CLEAN, permanently. No future event produces the absent check. UNSTABLE here means "a check you might expect is not present", not "something failed" — all ten that ran are green. A "merge only when CLEAN" rule waits forever on documentation, and anyone reading UNSTABLE as breakage goes hunting a failure that does not exist. Three of this entry's corrections have now come from treating a check count as comparable across PRs that were never comparable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…on it Retracting a consequence I committed to this entry an hour ago and posted to the pod as actionable: that a docs-only PR settles at MERGEABLE/UNSTABLE permanently, because the paths-filtered E2E check never dispatches, and that a merge-only-when-CLEAN rule therefore deadlocks on documentation. False. Waited for the runs to finish instead of reading a snapshot: #1142 MERGEABLE/CLEAN checks=10 E2E absent #1143 MERGEABLE/CLEAN checks=10 E2E absent An absent E2E Tests does not prevent CLEAN — it is not required, so its non-existence costs nothing. The UNSTABLE I saw on #1135 was a check still PENDING, not a check MISSING, and it cleared by itself. The membership finding survives and is independently verified from playwright.yml and from E2E=0 on all three PRs. Only the consequence I hung off it was fabricated. Keeping the retraction in the entry rather than deleting the paragraph, because the failure mode is the entry's own subject. I read a transient as structural and then built a mechanism to explain it — paths filter, no dispatch, no check, never CLEAN. Every step of that chain is true in isolation. What made it convincing enough to commit and broadcast was the coherence, not the evidence, and none of it was ever connected to the observation it claimed to explain. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…hing @sprint-review (57010) supplied the cause behind the measurement. Re-derived rather than accepted, and every part holds: release-safety.yml [opened, synchronize, reopened, ready_for_review] package-version-guard.yml [opened, synchronize, reopened, ready_for_review] pr-base-freshness.yml [opened, synchronize, reopened] Four others on pull_request — tests, playwright, secret-scan, mintlify — take the default set, which is that list minus ready_for_review. `grep -rn edited .github/workflows/` returns nothing at all. So the retarget does fire an event. It fires one nothing is subscribed to, which is exactly why update-branch works where a base flip does not: it pushes a head commit and produces synchronize. Kept the two halves separate because their evidence differs. Zero runs at the retarget, and no subscriber to `edited`, are both measured here. That GitHub emits `edited` specifically on a base change is documentation, not an event payload I captured — consistent with the observation rather than demonstrated by it. That distinction is rule 16's, applied to my own addendum. Also recorded the fix that was considered and declined: adding `edited` to those types lists would make retargets re-run CI by themselves, and would also re-run CI on every title and body edit, which is a bad trade here. The mitigation stays a deliberate head event. Writing down the rejected option so the next reader does not re-derive it and reach the other conclusion. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… predict (#1144) @sprint-review (57364) named the shape of an error I made an hour earlier and observed it is not in the audit. Agreed it is not, and it does not belong there: entries 42 and 43 are instruments misread, and the AX audit's scope is surfaces that taught a false model. This was a correct inference about the wrong world — a reasoning failure, which is what this file is for. The rule: a mechanism that explains an observation is not evidence for it. The machinery is often real and every link independently true, and none of that connects the chain to the thing you saw. The tell is that the explanation was assembled after the observation and has not been asked to forbid anything yet. So name one consequence the diagnosis REQUIRES, and go look, before committing it or handing it to anyone as actionable. Corollary in the same rule, because it is what produced the bad datum: a pipeline caught mid-flight is indistinguishable from a terminal state. A status read once is a snapshot, not a finding. Earned on #1135, and the incident is recorded with the part that stings. The invented mechanism made a checkable prediction — two sibling docs PRs must also never reach CLEAN — and both settled MERGEABLE/CLEAN minutes later. That was one command. It was never run, because the chain already felt verified, and the conclusion went into an audit entry and out to the room as something to work around. No pointer edits needed: README.md, REVIEW.md and CLAUDE.md all reference this file generically rather than by rule count. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

… tip @sprint-review (57014): 3/4 took four heads in twelve minutes while tests.yml runs 5-6, so no run could ever cover the tip. Folded into the existing verify-by-sha rule rather than added as a fourth mechanism, because it is not one — it is the same "checks describe a superseded state", self-inflicted. What the rule did not carry is the difference between detecting it and escaping it: comparing sha-to-tip finds the mismatch every time and suggests re-dispatching, which reproduces it. Only noticing the RATE tells you the fix is to stop pushing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…ts head Fourth mechanism, found by chasing @sprint-review's #1132 measurement (57463): its entire check list is one skipped Release Branch Guard, and it reads CLEAN because nothing is failing because nothing ran. They measured the counts and declined to claim the cause. This is the cause. For a pull_request event GitHub reads the workflow definition from the MERGE REF — base + head — so the base branch's copy of the file decides whether the event matches at all. #1123 dropped `branches: [main]` from tests.yml on main at 15:20:27Z. #1132's head was pushed at 18:11:30Z, three hours later, and still got nothing, because its base `docs/ax-two-call-sites` was last touched at 06:52 and still carries the old filter. Read the file on that branch to confirm rather than inferring it. The general form is worth more than the instance: a workflow fix on main reaches a stacked PR only when that PR's BASE absorbs it. Not the head. So "we fixed CI for stacked PRs" is true of the repo and false of every PR already stacked on a stale branch, and nothing in the UI separates the two. Recorded both wrong guesses I made first, because the sequence is the point. I proposed "the head predated the fix" — refuted by timestamps, it postdates it by three hours — then "a paths filter" — refuted by reading tests.yml, which has none. Rule 16 killed both: make the mechanism predict something and go check. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

They established it at 15:25:50Z — "for pull_request, the trigger is read from the PR's own merge ref" — measured on #1120, whose head was pushed 15:22:21Z, two minutes after #1123 merged, and got zero runs because its base still carried the old filter. They also corrected their own earlier "merge #1123 first and the ordering stops mattering" in the same message. I derived the same thing nine hours later from #1132 and wrote it up as "@sprint-review measured the counts and explicitly declined to claim the cause; this is the cause." Both halves wrong: they did claim it, and they were first. Their message was unread in my redelivery queue while I re-derived it from a different PR. Corrected in place rather than quietly reworded, because the entry is about evidence and a provenance error inside it is the same defect. Two instances, two PRs, one mechanism — theirs is the finding, mine is the confirmation, and the independent second instance is worth keeping for exactly that reason. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… around @sprint-review (57058) nearly retracted a correct finding. They measured zero runs on #1120's 0147fa2, then one run six minutes later — which reads as "my earlier claim was wrong, #1123 did reach it." The second run was my hand-dispatch landing between their two checks. The `event` field is the only discriminator. `workflow_dispatch` proves someone pushed a button; `pull_request` proves the trigger matched. Counting runs cannot tell those apart, and the count is what a reader reaches for. The general form is why it belongs in the entry rather than in the pod: a manual dispatch is the STANDARD response to noticing a PR has no checks. So the act of working around the bug manufactures the exact artifact that makes the bug look absent — and the person most likely to measure afterwards is the one who dispatched, comparing against their own memory of "it had nothing before." I am that person here. Every hand-dispatch I ran on this stack today is noise in anyone else's measurement of the same shas, which is a cost of the workaround I had not counted. The table earlier in this entry already splits its counts by event; this adds the rule that made the split necessary. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

lilyshen0722 commented Aug 22, 2026

View reviewed changes

lilyshen0722 and others added 7 commits August 22, 2026 13:19

lilyshen0722 and others added 4 commits August 22, 2026 15:33

lilyshen0722 merged commit 02bc430 into main Aug 23, 2026
10 checks passed

lilyshen0722 deleted the docs/ax-conflicting-pr-loses-its-suite branch August 23, 2026 01:01

Navigation Menu

Uh oh!

docs(ax): entry 41 — a conflicting PR loses its suite and renders green - #1135

docs(ax): entry 41 — a conflicting PR loses its suite and renders green#1135
lilyshen0722 merged 12 commits into
mainfrom
docs/ax-conflicting-pr-loses-its-suite

lilyshen0722 commented Aug 22, 2026

Uh oh!

lilyshen0722 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

lilyshen0722 commented Aug 22, 2026

Uh oh!

lilyshen0722 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant