Skip to content

Speed up CI service health checks with 1s start-interval probes - #6588

Draft
cnkk wants to merge 4 commits into
masterfrom
ci/faster-service-health-checks
Draft

Speed up CI service health checks with 1s start-interval probes#6588
cnkk wants to merge 4 commits into
masterfrom
ci/faster-service-health-checks

Conversation

@cnkk

@cnkk cnkk commented Aug 10, 2026

Copy link
Copy Markdown
Member

Two changes to cut "Initialize containers" time in Elixir CI, based on profiling real runs (details below).

1. Fine-grained health-check probing. Since #6571 made the service health checks real, the step went from ~10s to a 31-50s median. Postgres is ready in ~1-2s but with `--health-interval 10s` Docker runs its first probe only at t=10s, so the healthy flag can't flip earlier and the runner detects it at ~14s. `--health-start-interval 1s` (Docker Engine 25+; runners are on 28) probes every second during the start period so the flag flips at actual readiness:

```yaml
options: >-
--health-cmd "pg_isready -h 127.0.0.1"
--health-start-period 30s
--health-start-interval 1s
--health-interval 10s
--health-timeout 5s
--health-retries 5
```

Postgres also gets `-h 127.0.0.1` so the probe checks TCP instead of the unix socket, since the official image's temporary init-phase server listens only on the socket. Steady-state cadence (10s interval, 5 retries) is unchanged.

2. Non-alpine ClickHouse image. A 6-variant profiling matrix on `blacksmith-4vcpu-ubuntu-2404` showed the alpine image, served from the Docker container cache, pays ~29s of lazy layer hydration on first use per VM (~10s before the entrypoint even executes), while disabling system log tables and tmpfs on `/var/lib/clickhouse` had zero effect on boot. The non-alpine `clickhouse/clickhouse-server:25.11.5.8` bypasses the cache (17s registry pull, overlapped with the postgres wait) and starts in ~0.6s, netting roughly 12s per service-backed job. Caveat: if the container cache later picks this image up, the hydration cost may return; the durable fix is platform-side.

cnkk and others added 4 commits August 10, 2026 11:57
…obes

Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
… hydration stall

Co-authored-by: Codesmith <codesmith-bot@users.noreply.github.com>
@cnkk
cnkk marked this pull request as draft August 10, 2026 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant