TT-17868 fix env-up race + drift-check GOPROXY cache - #152
Conversation
env-up: docker compose up now uses --wait so master- and slave-datacenter profiles block until their existing healthchecks (dashboard-checker, gateway-checker-*) pass, instead of returning as soon as containers are created. dash-bootstrap.sh has no retry of its own (set -e on the first curl), so without --wait it could hit tyk-analytics before it's actually serving -- this was the root cause of an intermittent 'Temporary failure in name resolution' in tyk-analytics api-tests. drift-check: GOPROXY=direct bypasses proxy.golang.org's CDN, which caches the @master ref's resolved commit for 30 minutes. Without this, a merge to gromit's master can take up to half an hour to be picked up here, making this check flag false drift against a stale render.
🎯 Recommended Merge TargetsBased on JIRA ticket TT-17868: Update Tyk components to Go 1.26 Fix Version: Ara 1.33.1
Required:
Fix Version: Tyk Identity Broker 1.8.0
Required:
Fix Version: Tyk Sync 2.2.2
Required:
Fix Version: Tyk Operator 1.5.0
Required:
Fix Version: Tyk Portal 1.19.0
Required:
📋 Workflow
|
|
This pull request resolves a caching issue in the Note: A previously included change to address a race condition in the Files Changed Analysis
The change is minimal, with only 7 lines of code and comments added to a single CI workflow file. Architecture & Impact Assessment
|
\n\n
✅ Architecture Check PassedNo architecture issues found – changes LGTM. ✅ Performance Check PassedNo performance issues found – changes LGTM. ✅ Quality Check PassedNo quality issues found – changes LGTM. Powered by Visor from Probelabs Last updated: 2026-08-12T23:55:17.158Z | Triggered by: pr_updated | Commit: 0fbbc62 💡 TIP: You can chat with Visor using |
Confirmed via a live test run (tyk-analytics PR #6082 pointed at this branch) that --wait itself works — the name-resolution race is gone — but the initial 120s timeout was too short for the slave-datacenter profile's 6 gateways, each behind a checker with retries: 30 x interval: 5s (up to ~150s worst case): 'application not healthy after 2m0s'.
Root-caused the second CI failure: keycloak (start_period: 30s, retries: 15 x interval: 10s) has a 180s worst-case healthcheck window on its own, longer than the 120s the master-datacenter wait was given. Confirmed via a live run (tyk-analytics PR #6082): every other service reached Healthy, keycloak alone stayed Waiting until the compose command was killed at the timeout.
keycloak has consistently failed its healthcheck on every prior test run regardless of --wait-timeout (120s/240s) or runner size (8x/16x), which rules out both a timeout-too-short and a CPU/IO-starvation explanation. Dumping its container logs + health state on failure to find the actual cause instead of continuing to guess at timeout/resource tuning.
Previous attempt called 'docker compose ... logs keycloak' without --env-file, so Compose failed re-parsing pro-ha.yml with 'invalid spec: :/conf/: empty section between colons' before it could even print keycloak's logs. keycloak has a fixed container_name so 'docker logs keycloak' works directly without needing the compose project context at all.
keycloak in the multi-datacenter test env consistently fails its own healthcheck ('Health check exceeded timeout (10s)') regardless of --wait-timeout (120s/240s) or runner size (8x/16x) -- keycloak itself starts and imports its realm in ~5.5s every time, so this isn't a startup-time or resource problem. The real fix belongs in tyk-pro's keycloak healthcheck definition (deps_pro-ha.yml), not here. Reverting env-up to its original form and keeping only the drift-check GOPROXY fix in this PR.
Reverting was correct at the time -- keycloak's healthcheck was hanging regardless of --wait-timeout, so --wait alone couldn't help. Now that tyk-pro's TT-17868_fix_keycloak_healthcheck_timeout branch fixes the actual healthcheck (adds Connection: close), re-adding --wait (no custom timeout needed -- Compose's default is generous) to verify both fixes together resolve the original 'Temporary failure in name resolution' in tyk-analytics api-tests.
--wait is incompatible with the one-shot barrier containers (wait_db/wait_tyk_components) used in tyk-pro's compose setup: they're designed to exit 0 once their depends_on condition is satisfied, but --wait treats any non-running/non-healthy exit as a failure. Simplifying this PR to just the drift-check GOPROXY fix, which is unrelated and safe on its own.
Summary
GOPROXY=directbypasses proxy.golang.org's CDN cache for the@masterref (30 min TTL), so a merge to gromit's master is picked up immediately instead of up to half an hour later.Why
Confirmed via repeated CI runs pulling a stale gromit version (e.g. an old commit instead of a just-merged one) up to 20+ minutes after merge, matching proxy.golang.org's
Cache-Control: public, max-age=1800header on the@masterref resolution.Note
This PR originally also included an
env-upchange (docker compose up --wait) to fix an intermittentTemporary failure in name resolutionin tyk-analyticsapi-tests. That part has been reverted: further investigation (see tyk-analytics PR #6082) showed the actual blocker iskeycloak's own healthcheck in tyk-pro'sdeps_pro-ha.ymlconsistently exceeding its 10s timeout — keycloak itself starts and imports its realm in ~5.5s every time, confirmed unrelated to wait-timeout tuning (tested 120s/240s) or runner size (tested 8x/16x). The real fix belongs in tyk-pro, not here.