Skip to content

TT-17868 fix env-up race + drift-check GOPROXY cache - #152

Open
rafalgolarz wants to merge 8 commits into
mainfrom
TT-17868_fix_env_up_wait_and_goproxy_cache
Open

TT-17868 fix env-up race + drift-check GOPROXY cache#152
rafalgolarz wants to merge 8 commits into
mainfrom
TT-17868_fix_env_up_wait_and_goproxy_cache

Conversation

@rafalgolarz

@rafalgolarz rafalgolarz commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • drift-check: GOPROXY=direct bypasses proxy.golang.org's CDN cache for the @master ref (30 min TTL), so a merge to gromit's master is picked up immediately instead of up to half an hour later.

Why

Confirmed via repeated CI runs pulling a stale gromit version (e.g. an old commit instead of a just-merged one) up to 20+ minutes after merge, matching proxy.golang.org's Cache-Control: public, max-age=1800 header on the @master ref resolution.

Note

This PR originally also included an env-up change (docker compose up --wait) to fix an intermittent Temporary failure in name resolution in tyk-analytics api-tests. That part has been reverted: further investigation (see tyk-analytics PR #6082) showed the actual blocker is keycloak's own healthcheck in tyk-pro's deps_pro-ha.yml consistently exceeding its 10s timeout — keycloak itself starts and imports its realm in ~5.5s every time, confirmed unrelated to wait-timeout tuning (tested 120s/240s) or runner size (tested 8x/16x). The real fix belongs in tyk-pro, not here.

env-up: docker compose up now uses --wait so master- and slave-datacenter profiles block until their existing healthchecks (dashboard-checker, gateway-checker-*) pass, instead of returning as soon as containers are created. dash-bootstrap.sh has no retry of its own (set -e on the first curl), so without --wait it could hit tyk-analytics before it's actually serving -- this was the root cause of an intermittent 'Temporary failure in name resolution' in tyk-analytics api-tests.

drift-check: GOPROXY=direct bypasses proxy.golang.org's CDN, which caches the @master ref's resolved commit for 30 minutes. Without this, a merge to gromit's master can take up to half an hour to be picked up here, making this check flag false drift against a stale render.
@rafalgolarz
rafalgolarz requested a review from a team August 12, 2026 19:29
@github-actions

Copy link
Copy Markdown

🎯 Recommended Merge Targets

Based on JIRA ticket TT-17868: Update Tyk components to Go 1.26

Fix Version: Ara 1.33.1

⚠️ Warning: Expected release branches not found in repository

Required:

  • master - No matching release branches found. Fix will be included in future releases.

Fix Version: Tyk Identity Broker 1.8.0

⚠️ Warning: Expected release branches not found in repository

Required:

  • master - No matching release branches found. Fix will be included in future releases.

Fix Version: Tyk Sync 2.2.2

⚠️ Warning: Expected release branches not found in repository

Required:

  • master - No matching release branches found. Fix will be included in future releases.

Fix Version: Tyk Operator 1.5.0

⚠️ Warning: Expected release branches not found in repository

Required:

  • master - No matching release branches found. Fix will be included in future releases.

Fix Version: Tyk Portal 1.19.0

⚠️ Warning: Expected release branches not found in repository

Required:

  • master - No matching release branches found. Fix will be included in future releases.

📋 Workflow

  1. Merge this PR to master first

@rafalgolarz rafalgolarz self-assigned this Aug 12, 2026
@probelabs

probelabs Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

This pull request resolves a caching issue in the drift-check workflow that could cause it to run against a stale version of the gromit tool. By setting the GOPROXY=direct environment variable, the workflow now bypasses the Go module proxy's 30-minute cache for the @master reference, ensuring that the latest commit is fetched directly from the source repository. This prevents false-positive drift alerts that occurred when a recently merged change to gromit was not immediately available to the CI job.

Note: A previously included change to address a race condition in the env-up action has been reverted from this PR, as the root cause was identified in an external repository.

Files Changed Analysis

  • .github/workflows/drift-check.yml: Modified to add an env block with GOPROXY: direct to the 'Build gromit from master' step. This is the only file changed.

The change is minimal, with only 7 lines of code and comments added to a single CI workflow file.

Architecture & Impact Assessment

  • What this PR accomplishes: It increases the accuracy and reliability of the drift-check CI workflow by ensuring it always uses the live master version of the gromit tool, eliminating false negatives caused by CDN caching delays.
  • Key technical changes introduced: The GOPROXY environment variable is set to direct for a go install command, altering how Go modules are fetched for this specific step.
  • Affected system components: The impact is strictly limited to the drift-check GitHub Actions workflow. There are no changes to production code or other parts of the CI/CD pipeline.

drift-check Dependency Fetching Flow

This diagram illustrates how GOPROXY=direct bypasses the CDN cache to fetch the latest tool version immediately.

sequenceDiagram
    participant CI as drift-check job
    participant GoProxy as proxy.golang.org
    participant GitHub as gromit repository

    group "Before Change (Potential 30-min Delay)"
        CI->>GoProxy: go install gromit@master
        alt "Cache Hit (within 30 min TTL)"
            GoProxy-->>CI: Returns cached (old) commit
        else Cache Miss
            GoProxy->>GitHub: Resolve @master
            GitHub-->>GoProxy: Latest commit
            GoProxy-->>CI: Returns latest commit
        end
    end

    group "After Change (GOPROXY=direct)"
        CI->>GitHub: go install gromit@master
        GitHub-->>CI: Returns latest commit directly
    end
Loading

Scope Discovery & Context Expansion

  • The scope of this change is narrow, affecting only the dependency fetching mechanism for the gromit tool within the drift-check workflow.
  • This change highlights a tight coupling between this repository's CI and the master branch of the TykTechnologies/gromit repository, where even short delays in propagation are problematic.
  • No other files, actions, or workflows are directly impacted. The solution is self-contained within the workflow file.
Metadata
  • Review Effort: 1 / 5
  • Primary Label: chore

Powered by Visor from Probelabs

Last updated: 2026-08-12T23:55:32.560Z | Triggered by: pr_updated | Commit: 0fbbc62

💡 TIP: You can chat with Visor using /visor ask <your question>

@probelabs

probelabs Bot commented Aug 12, 2026

Copy link
Copy Markdown
Contributor
\n\n

✅ Architecture Check Passed

No architecture issues found – changes LGTM.

✅ Performance Check Passed

No performance issues found – changes LGTM.

✅ Quality Check Passed

No quality issues found – changes LGTM.


Powered by Visor from Probelabs

Last updated: 2026-08-12T23:55:17.158Z | Triggered by: pr_updated | Commit: 0fbbc62

💡 TIP: You can chat with Visor using /visor ask <your question>

Confirmed via a live test run (tyk-analytics PR #6082 pointed at this
branch) that --wait itself works — the name-resolution race is gone —
but the initial 120s timeout was too short for the slave-datacenter
profile's 6 gateways, each behind a checker with retries: 30 x
interval: 5s (up to ~150s worst case): 'application not healthy after
2m0s'.
Root-caused the second CI failure: keycloak (start_period: 30s,
retries: 15 x interval: 10s) has a 180s worst-case healthcheck
window on its own, longer than the 120s the master-datacenter wait
was given. Confirmed via a live run (tyk-analytics PR #6082): every
other service reached Healthy, keycloak alone stayed Waiting until
the compose command was killed at the timeout.
keycloak has consistently failed its healthcheck on every prior test
run regardless of --wait-timeout (120s/240s) or runner size (8x/16x),
which rules out both a timeout-too-short and a CPU/IO-starvation
explanation. Dumping its container logs + health state on failure to
find the actual cause instead of continuing to guess at
timeout/resource tuning.
Previous attempt called 'docker compose ... logs keycloak' without
--env-file, so Compose failed re-parsing pro-ha.yml with
'invalid spec: :/conf/: empty section between colons' before it
could even print keycloak's logs. keycloak has a fixed container_name
so 'docker logs keycloak' works directly without needing the compose
project context at all.
keycloak in the multi-datacenter test env consistently fails its own healthcheck ('Health check exceeded timeout (10s)') regardless of --wait-timeout (120s/240s) or runner size (8x/16x) -- keycloak itself starts and imports its realm in ~5.5s every time, so this isn't a startup-time or resource problem. The real fix belongs in tyk-pro's keycloak healthcheck definition (deps_pro-ha.yml), not here. Reverting env-up to its original form and keeping only the drift-check GOPROXY fix in this PR.
Reverting was correct at the time -- keycloak's healthcheck was
hanging regardless of --wait-timeout, so --wait alone couldn't help.
Now that tyk-pro's TT-17868_fix_keycloak_healthcheck_timeout branch
fixes the actual healthcheck (adds Connection: close), re-adding
--wait (no custom timeout needed -- Compose's default is generous)
to verify both fixes together resolve the original
'Temporary failure in name resolution' in tyk-analytics api-tests.
--wait is incompatible with the one-shot barrier containers
(wait_db/wait_tyk_components) used in tyk-pro's compose setup: they're
designed to exit 0 once their depends_on condition is satisfied, but
--wait treats any non-running/non-healthy exit as a failure. Simplifying
this PR to just the drift-check GOPROXY fix, which is unrelated and
safe on its own.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant