ci: Change host shard weight measurement to use representative runs - #5850
Draft
backspace wants to merge 2 commits into
Draft
ci: Change host shard weight measurement to use representative runs#5850backspace wants to merge 2 commits into
backspace wants to merge 2 commits into
Conversation
`test-module-timings.json` is regenerated on every push to main and consumed by pull_request runs, and those two shapes differ by about 1.5x per test: Percy is always on for a non-PR event (~1.23x) and the index cache is deliberately skipped, so every realm indexes from scratch (the remaining ~1.21x, visible as `Import cached realm index` taking 0s on all 20 main shards against 27s on a PR run). A uniform factor would not matter, since greedy packing is scale- invariant. This one is uneven — fixture-heavy acceptance modules pay far more of it than unit tests, per-module ratios running 0.86x to 3.40x — so the packer mis-assigns rather than mis-scales. On run 32406440696 it predicted a 1s spread across 20 shards and got 239s, with the slowest shard 19% over prediction. Regenerating from that run's own report drops the predicted slowest shard from 362s to 306s. Take the report from the run of the PR that produced the merge, and only when that run was Percy-free, index-cached and fully green. Anything else leaves the file alone: stale-but-consistent weights balance better than fresh ones measured in the wrong environment, which is also why there is no fallback to this run's own report. Over the last 25 PR-derived merges, 3 would have qualified — Percy is the binding filter, since most merged PRs touch UI. Weights move slowly and the existing drift gate already suppresses balance-equivalent rewrites, so roughly daily updates are enough; if that proves too sparse, the same check could scan back over recent merges for the newest qualifying run instead of only considering this one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Drop this commit before merging. The updater only runs on push-to-main, so a PR run would prove nothing about the new report selection. This makes the job run on this branch too, replays #5834's merge commit (whose PR run was Percy-free, index-cached and green — one the new logic should accept), and stops before pushing, printing the change it would have committed instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Host Test Results 1 files 1 suites 1h 57m 44s ⏱️ Results for commit 1e3831a. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test-module-timings.jsonis used to balance host tests across shards, but it’s regenerated by runs onmain, which differ timing-wise from PR runs because of Percy and index caching. This changes the balancing to use non-Percy PR runs when available.Claude explanation
`test-module-timings.json` is regenerated on every push to main and consumed by pull_request runs, and those two shapes differ by about 1.5x per test: Percy is always on for a non-PR event (~1.23x) and the index cache is deliberately skipped, so every realm indexes from scratch (the remaining ~1.21x, visible as `Import cached realm index` taking 0s on all 20 main shards against 27s on a PR run).A uniform factor would not matter, since greedy packing is scale- invariant. This one is uneven — fixture-heavy acceptance modules pay far more of it than unit tests, per-module ratios running 0.86x to 3.40x — so the packer mis-assigns rather than mis-scales. On run 32406440696 it predicted a 1s spread across 20 shards and got 239s, with the slowest shard 19% over prediction. Regenerating from that run's own report drops the predicted slowest shard from 362s to 306s.
Take the report from the run of the PR that produced the merge, and only when that run was Percy-free, index-cached and fully green. Anything else leaves the file alone: stale-but-consistent weights balance better than fresh ones measured in the wrong environment, which is also why there is no fallback to this run's own report.
Over the last 25 PR-derived merges, 3 would have qualified — Percy is the binding filter, since most merged PRs touch UI. Weights move slowly and the existing drift gate already suppresses balance-equivalent rewrites, so roughly daily updates are enough; if that proves too sparse, the same check could scan back over recent merges for the newest qualifying run instead of only considering this one.