Speed up CI service health checks with 1s start-interval probes - #6588
Draft
cnkk wants to merge 4 commits into
Draft
Speed up CI service health checks with 1s start-interval probes#6588cnkk wants to merge 4 commits into
cnkk wants to merge 4 commits into
Conversation
…obes Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
… hydration stall Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
cnkk
marked this pull request as draft
August 10, 2026 14:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes to cut "Initialize containers" time in Elixir CI, based on profiling real runs (details below).
1. Fine-grained health-check probing. Since #6571 made the service health checks real, the step went from ~10s to a 31-50s median. Postgres is ready in ~1-2s but with `--health-interval 10s` Docker runs its first probe only at t=10s, so the healthy flag can't flip earlier and the runner detects it at ~14s. `--health-start-interval 1s` (Docker Engine 25+; runners are on 28) probes every second during the start period so the flag flips at actual readiness:
```yaml
options: >-
--health-cmd "pg_isready -h 127.0.0.1"
--health-start-period 30s
--health-start-interval 1s
--health-interval 10s
--health-timeout 5s
--health-retries 5
```
Postgres also gets `-h 127.0.0.1` so the probe checks TCP instead of the unix socket, since the official image's temporary init-phase server listens only on the socket. Steady-state cadence (10s interval, 5 retries) is unchanged.
2. Non-alpine ClickHouse image. A 6-variant profiling matrix on `blacksmith-4vcpu-ubuntu-2404` showed the alpine image, served from the Docker container cache, pays ~29s of lazy layer hydration on first use per VM (~10s before the entrypoint even executes), while disabling system log tables and tmpfs on `/var/lib/clickhouse` had zero effect on boot. The non-alpine `clickhouse/clickhouse-server:25.11.5.8` bypasses the cache (17s registry pull, overlapped with the postgres wait) and starts in ~0.6s, netting roughly 12s per service-backed job. Caveat: if the container cache later picks this image up, the hydration cost may return; the durable fix is platform-side.