Await Docker image cleanup to ensure GHES stale image cleanup succeeds - #1753
Await Docker image cleanup to ensure GHES stale image cleanup succeeds#1753jeffwidman wants to merge 2 commits into
Conversation
| jest.spyOn(docker, 'listImages').mockResolvedValue([oldImage]) | ||
| jest | ||
| .spyOn(docker, 'getImage') | ||
| .mockReturnValue({remove} as unknown as Docker.Image) |
There was a problem hiding this comment.
I'm a little torn on this mock... yes, this test ensures the function was called as expected, but due to the mocks it doesn't ensure it behaves under the covers as expected...
Ultimately I decided to drop this test, it doesn't do much and the integration test does enough (especially since we removed the artificial sleep on it... it should now fail if we're not properly awaiting)
The test mocked Dockerode end to end, so it only verified that cleanup awaited a promise supplied by the mock. It did not exercise Dockerode's promise implementation or prove that an image was removed. The Docker-backed integration test already verifies cleanup through the real client and daemon, including the resulting image state immediately after cleanup resolves. Keep that as the authoritative coverage instead.
| core.info(`Skipping current image ${imageInfo.Id}`) | ||
| continue | ||
| } | ||
| const imageInfoList = await docker.listImages(options) |
There was a problem hiding this comment.
Copilot flagged this:
Now that we await docker.listImages(options) directly, a rejection here propagates up and rejects cleanupOldImageVersions. In run(), since the updater images are cleaned via Promise.all(...), a listImages failure for one repo will reject the whole batch and skip cleanup of the remaining images (and the subsequent proxy-image cleanup) — landing in the outer try/catch.
The old callback-based code silently ignored the err argument, so a listing failure for one repo didn't stop the others. To preserve that isolation, consider scoping the listing failure per repo, e.g.:
let imageInfoList: Docker.ImageInfo[]
try {
imageInfoList = await docker.listImages(options)
} catch (error: unknown) {
if (error instanceof Error) {
core.info(`Unable to list images for ${repo} -- ${error.message}`)
}
return
}That keeps one repo's listing failure from short-circuiting cleanup of the others, while still awaiting removals as intended. Alternatively, Promise.allSettled in run() would achieve similar isolation across images.
There was a problem hiding this comment.
I'd spotted that error propagation change originally and saw it as a feature not a bug! Since it'll stop swallowing errors...
But I hadn't considered there might be a way to both report the errors and continue for non-error ones. I'll poke at this.
Why this is needed
The cleanup post-action previously started Docker image discovery through Dockerode’s callback API, but
cleanupOldImageVersionsreturned before that callback and the image removals completed. This allowed the post-action to report success and the Node process to exit while cleanup was still in flight.In production, that means stale Dependabot updater and proxy images may be left behind. This is especially important for persistent self-hosted and GHES runners, where Docker state survives between jobs: old image versions can accumulate over time, consume disk space, and eventually interfere with later Dependabot jobs. Awaiting both image listing and every removal makes completion of the post-action mean that cleanup has actually finished.
The change also speeds up the integration tests by removing two 200 ms timing delays. That test improvement is secondary; the main benefit is making production image cleanup reliable rather than best-effort timing-dependent work.
Summary
cleanupOldImageVersionsremains pending until image removal completesTesting
npx eslint src/cleanup.ts __tests__/cleanup.test.ts __tests__/cleanup-integration.test.tsnpx tsc --noEmitSKIP_INTEGRATION_TESTS=true npx jest __tests__/cleanup.test.ts --runInBandnpx jest __tests__/cleanup-integration.test.ts --runInBandnpm run packageThe full local non-integration suite still hits the existing five-second timeout in two
container-service.test.tscases; the cleanup unit and integration suites pass.