Cache the apt archives, and verify the cache is used (#386) - #410
Open
ChronicallyJD wants to merge 1 commit into
Open
Cache the apt archives, and verify the cache is used (#386)#410ChronicallyJD wants to merge 1 commit into
ChronicallyJD wants to merge 1 commit into
Conversation
…oads anyway The measurement on commandprompt#386: across the last 80 runs, 352 successes of this step at a median of 25s, and six failures all pinned at 606 to 608s. Nothing in between. The step either finishes in half a minute or hits the 600s bound. The stall is azure.archive.ubuntu.com serving the LLVM toolchain at around 30 kB/s. That toolchain is 91 MB of the 102 MB and is a hard dependency: $ apt-cache depends postgresql-server-dev-18 Depends: clang-21 Depends: llvm-21-dev so --no-install-recommends cannot drop it, and there is no reason to want to. It is a legitimate dependency. The problem is downloading it repeatedly across a mirror that intermittently will not serve it. apt now writes its archives to a cached directory, keyed on the PG major, the runner OS and arch, and ImageOS. ImageOS is in the key because a new runner image can bring a different LLVM major, and a key that ignored it would serve debs for the wrong one. The second half matters more than the first. A cache that populates and never hits looks identical in the log to one that works: the job is simply still slow, and nothing reports it. So on a reported hit, the step now fails if apt fetched anything over the network. That is the same defect shape as commandprompt#396, where a suite was exempt from the registration check and never invoked. Two things deliberately not done, both from the issue: Raising the bound. At 30 kB/s the fetch needs roughly 50 minutes, which trades a red check for a runner held most of an hour. Pinning a different mirror. The runner chooses from its own mirrorlist, and betting on a different one is the same bet we are losing now. A cache miss degrades to exactly today's behaviour, and a stale entry is re-fetched by apt rather than used, so the failure mode is a slow job and not a wrong one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #386, implementing the plan from the issue.
What this does
apt writes its archives to
~/apt-archives, whichactions/cachekeys on the PG major,runner OS and arch, and
ImageOS. Four install steps acrossci.ymlandnightly.yml.ImageOSis in the key deliberately: a new runner image can bring a different LLVM major,and a key that ignored it would serve debs for the wrong one.
The half I care about more
On a reported cache hit, the step fails if apt fetched anything over the network.
A cache that populates and never hits looks identical in the log to one that works. The job
is simply still slow and nothing says so. That is the same shape as #396, where the suite
had the exemption and not the invocation, and it is the failure I would otherwise be
shipping here.
What I did not do
Raise the bound. At 30 kB/s the fetch needs about 50 minutes, so that trades a red check
for a runner held most of an hour.
Pin a mirror. The runner picks from its own mirrorlist. Betting on a different one is
the bet we are already losing.
Drop the toolchain.
apt-cache depends postgresql-server-dev-18showsDepends: clang-21andDepends: llvm-21-dev, hard, so no apt flag removes them. jd's viewand mine is that there is no reason to want to: it is a legitimate dependency of
postgresql-server-dev-N.What I could not verify, and how it fails
I cannot run GitHub Actions locally, so the first CI run on this PR is the test. I have
validated what I can: both workflows still parse as YAML, all four install steps route
their archives and tee their output, and no package was dropped from any list.
That last one is not rhetorical. My first patch attempt inserted the tee mid-list:
which would have silently stopped installing
lcovandpython3-pipin the coverage job.Caught by reading the generated file rather than trusting the script. Fixed, and every
package list is asserted intact in the diff.
The failure modes are benign by construction. A miss degrades to exactly today's behaviour.
A stale entry is re-fetched by apt rather than used, because apt validates what is in the
archive directory. So the downside is a slow job, not a wrong one.
Worth watching the first few runs for the hit rate rather than assuming it.