Loaded OCI images are left orphaned in Docker daemon after test execution

[bkost-fernride]

Problem

When running container_structure_test targets with Bazel using the docker driver, the Bazel rule compiles a wrapper launcher script that loads the built OCI image layout into the local Docker daemon under a dynamic unique tag: --default-image-tag "cst.oci.local/$DIGEST:$DIGEST". However, after test execution this image tag is not cleaned up from the Docker daemon.

This means that on developer machines and stateful runners (e.g. self-hosted GPU runners) this results in large amounts of disk space being used for leftover docker images.

> docker images
     i Info →   U  In Use
IMAGE                                                                                   ID             DISK USAGE   CONTENT SIZE
cst.oci.local/sha256-0f22b2b3ab798590eba542fd0b97c7eea9d75f4d22e265f8a6cba473e3d5d24f   d460401f0e96       12.2GB             0B
cst.oci.local/sha256-19138e79d02af90ad4da61e5fc41b6fdb109956356dbf8be349ae2a095953041   bec76fe59b4b       12.2GB             0B
cst.oci.local/sha256-25ae02e0a51277d9a52238dbace09e82001804908971974cdcf921328f6dd31d   9d892964272a       12.2GB             0B
cst.oci.local/sha256-300734190c063608de30f95c297603b0c6be1527953c6eec9d067f1983ddd28b   8cc6a7358615       12.2GB             0B
cst.oci.local/sha256-483f8a0141c030e23cb1ea331b48271fd5c75913b2911fe4b360946bc278d496   abc7645425fb       12.2GB             0B
cst.oci.local/sha256-4e4d63413de21ce825d8ce1ac8eabd1766e7b577124501b60cdfffd44d723c3f   4c1bad1d8747       12.2GB             0B
... <plenty more>
cst.oci.local/sha256-d0e9b5def972bfa86bf1d38f07a8541f3b1d2ffabcc3851095c679e33898ae38   cf9f322bd3b7       11.3GB             0B
cst.oci.local/sha256-d5df6e7bb4968ac1ab923a6073ffe9fbb984b60ffc19756cdaf5f5f6497385da   dcc785d69393       12.2GB             0B
cst.oci.local/sha256-dd00cdb24084dae5b00f81f04a40913917ff03d7943cd220b16e5e35fa92c167   2c6ca4cb8d2b       12.2GB             0B
cst.oci.local/sha256-ee066855ba10357ee64115f3b5df1c2fbcb1f6fef0bab1eb04e1bb382b7d75ce   1435fcd64b84       12.2GB             0B
cst.oci.local/sha256-f42d968b033ac86f3ee0233d3e1200bf7d6d49e318d67c307d76f731fe615c2e   e729997dda2b       12.2GB             0B
cst.oci.local/sha256-fccd3bb94eafab0006c3f56dcf3626df3c4142c758ad9a6c060a4a29d0e9d54d   ad5ffafb4631       12.2GB             0B
cst.oci.local/sha256-fd1526406fe8aac0bf3f8783b6b31c0348282be5d093a6ee941219adb6236aab   f6fa3e41e9c6       12.2GB             0B
cst.oci.local/sha256-fdb5bdbffc96f4b276c9668a99881b1aad14c34d77927243a1d24fa7d52893d1   24f3206d5feb       12.2GB             0B

Current workaround we are using

We're patching the bazel/container_structure_test.bzl with the following:

--- bazel/container_structure_test.bzl
+++ bazel/container_structure_test.bzl
@@ -37,9 +37,18 @@
 # When the image points to a folder, we can read the index.json file inside
 if [[ -d "$image" ]]; then
   readonly DIGEST=$("$jq" -r '.manifests[0].digest | sub(":"; "-")' "$image/index.json")
-  exec "$st" test --driver {driver} {fixed_args} --default-image-tag "cst.oci.local/$DIGEST:$DIGEST" $@
+  readonly TEST_TAG="cst.oci.local/$DIGEST:$DIGEST"
+
+  # Register a cleanup trap to remove the loaded image from Docker
+  cleanup() {{
+    docker rmi "$TEST_TAG" >/dev/null 2>&1 || true
+  }}
+  trap cleanup EXIT
+
+  "$st" test --driver {driver} {fixed_args} --default-image-tag "$TEST_TAG" "$@"
+  exit "$?"
 else
-  exec "$st" test --driver {driver} {fixed_args} $@
+  exec "$st" test --driver {driver} {fixed_args} "$@"
 fi
 """

effectively ensuring the created docker image is removed after the test completes