ansible: Drop unnecessary updates/upgrades - #2353
Open
mmlb wants to merge 8 commits into
Open
Conversation
mmlb
force-pushed
the
manny/psql-1451-do-apt-get-dist-upgrade-instead-of-apt-get-upgrade
branch
3 times, most recently
from
August 10, 2026 19:57
5444465 to
f224629
Compare
Having swap setup in stage1 uses up 10% of the disk while we need as much space as possible (for fetching, building, etc). We are extremely sensitive to cache clean up ordering so that we don't go over, but thats a big pain that can be allievated by moving swap to the end of the run. We run stage2 on beefy machines that don't need swap so lets use the space better.
It turns out that most of the code deleted here was actually not doing
anything useful. For example, switch_mirror modifies
/etc/apt/sources.list in place but the file doesn't have any mirrors
configured there so it's really a no-op! Here's the contents from an
instance I just fired up[^1]:
ubuntu@ip-172-31-26-227:~$ tail -n+1 /etc/apt/sources.list /etc/apt/sources.list.d/*
==> /etc/apt/sources.list <==
# Ubuntu sources have moved to the /etc/apt/sources.list.d/ubuntu.sources
# file, which uses the deb822 format. Use deb822-formatted .sources files
# to manage package sources in the /etc/apt/sources.list.d/ directory.
# See the sources.list(5) manual page for details.
==> /etc/apt/sources.list.d/ubuntu.sources <==
## Note, this file is written by cloud-init on first boot of an instance
## modifications made here will not survive a re-bundle.
--- 8< ---
Types: deb
URIs: http://us-east-2.ec2.archive.ubuntu.com/ubuntu/
Suites: noble noble-updates noble-backports
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
Types: deb
URIs: http://us-east-2.ec2.archive.ubuntu.com/ubuntu/
Suites: noble noble-updates noble-backports
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
## Ubuntu security updates. Aside from URIs and Suites,
## this should mirror your choices in the previous section.
Types: deb
URIs: http://security.ubuntu.com/ubuntu
Suites: noble-security
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
We get fallback handling by apt itself by adding multiple mirrors in
URIs, apt tries first (the regional) and falls back to global ubuntu
repos if there's an issue. We also setup a temporary apt config that
forces the APT_OPTIONS for all apt calls so we don't need to do it in
every call site.
I ended up dropping the in-repo sources file since they are strictly
worse than what we get from AWS in the build. There's no change in
suites or components between old and new, just that we get ubuntu
upstream as a fallback. Well technically there's a slight difference
since we are basing off of cloud-init generated files and they can
theoretically change under us but I'll gamble that it'll be fine or
better off. Besides, one day we'll be on NixOS as the ultimate "make
sure we know everything in the instance" ;).
I also got rid of the `add-apt-repository --yes universe` because
universe is already enabled.
[^1]: AMI=ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-20260604
We are wasting a bunch of time and brain power keeping track of package installs, so lets minimize them while keeping the grouping/messages intact. I did move the package installs from setup_apparmor && setup_grub into update_install_packages because why not just install everything in one place and leave the _setup_ to somewhere else. I moved them out of because package installation should all be done early and ASAP IMO so it can fail fast if its going to fail. Also I moved ec2-hibinit-agent, ec2-instance-connect, hibagent here since this is only run for AMIs and the comment isn't currently accurate. Either through `export DEBIAN_FRONTEND=noninteractive`, `disable_services` or maybe it was treated as a bug and is now fixed in the package I'm not sure but I ran this out of AWS and it was fine.
Lets be a little cleaner about setup/update/cleanup, also drop the wrapper functions to plain old apt-get calls since they no longer do anything useful. I created setup_apt because I want to move the Install-Recommends=False into it later and configure all apt calls to use it seamlessly, this way we will have a better/more visible list of packages being installed.
No yum handling necessary and commented out code should be deleted.
There's no point run updates/upgrades unnecessarily while also not
ensuring that we start with latest and end with latest. So I've dropped
any unecessary updates and upgrades from middle of runs. Now we update
and upgrade after start up ({surrogate,qemu}-bootstrap-nix, chroot was
already good) and then once at the end (qemu-bootstrap-nix and
nix-provision). Ansible's apt_repository module does update when
necessary and has not changed.
dist-upgrade ensures new packages are installed and unnecessary are removed while upgrade only upgrades packages. Using dist-upgrade gets us closer to inteded state according to metadata files.
mmlb
force-pushed
the
manny/psql-1451-do-apt-get-dist-upgrade-instead-of-apt-get-upgrade
branch
from
August 24, 2026 15:42
f224629 to
ebe2013
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What kind of change does this PR introduce?
Maintenance
What is the current behavior?
We run apt-get update almost every time we install a package in ansible and we run an apt-get upgrade in the middle of the AMI creation process. Both update and upgrade are already being done before and after ansible runs which is better. Have a good known state early and update/upgrade at the end to ensure we have latest versions. Could have been done in ansible too but since we already do pre/post ansible apt calls I decided to just leave them there.
What is the new behavior?
We don't waste time with unnecessary apt updates during ansible which could slow down the run and/or introduce flakiness if the remote is being updated too. No need to do the upgrade either since it was already being done post ansible.