Skip to content

[DO NOT MERGE] [REFERENCE] optimize: Tweak Integration Tests Flow for Speed + Cost #23408

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
kalverra wants to merge 4 commits into from
Draft

[DO NOT MERGE] [REFERENCE] optimize: Tweak Integration Tests Flow for Speed + Cost #23408

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
81798b3
chore(wasm): replace old wasmtest with wasmbuild
kalverra Aug 17, 2026
c44432f
ci: optimize test runners, caching, and setup-go
kalverra Aug 17, 2026
c9d774b
test(cre): add vault config propagation sync and auth mock tests
kalverra Aug 17, 2026
d114281
chore(deps): update module versions and fix linter rules
kalverra Aug 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ GitHub Actions in the Chainlink Go monorepo.
- Prefer runs-on runners when ubuntu-latest is insufficient.
- Minimize YAML and shell in workflows.
- Resolve smartcontractkit/.github from a local clone. Ask the user for the path if you cannot find it.
- Use `echo "key=value" | tee -a "$GITHUB_OUTPUT"` instead of `echo "key=value" >> "${GITHUB_OUTPUT}"`
</rules>

<docs>
Expand Down
11 changes: 3 additions & 8 deletions .github/workflows/ccip-system-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@

echo "matrix=$tests" | tee -a "${GITHUB_OUTPUT}"

run-ccip-tests:

Check failure on line 127 in .github/workflows/ccip-system-tests.yaml

View workflow job for this annotation

GitHub Actions / Validate Github Action Workflows 

[actionlint] reported by reviewdog 🐶
could not parse as YAML: did not find expected key [syntax-check]

Raw Output:
e:.github/workflows/ccip-system-tests.yaml:127:4: could not parse as YAML: did not find expected key [syntax-check]
name: ${{ matrix.tests.test_name }}
permissions:
contents: read
Expand Down Expand Up @@ -172,15 +172,10 @@

- name: Set up Go
id: setup-go
uses: smartcontractkit/.github/actions/ctf-setup-go@fa1d48a33e24f9b3b9f8c52e99a578a4597cb2a5 # v0.4.0
uses: actions/setup-go@v7
with:
go_mod_path: integration-tests/go.mod
cache_key_id: integration-tests-v1
cache_builds: true
cache_restore_only: "true"
should_tidy: false
no_cache: false
gati_token: ${{ steps.github-token.outputs.access-token }}
go-version-file: integration-tests/go.mod
cache: false # Using pre-compiled binaries, only need go setup for gotestsum

- name: Restore Pre-Compiled Test Binaries from S3 Cache
uses: actions/cache/restore@v6
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/ci-core.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -203,7 +203,7 @@
with:
method: chat.postMessage
token: ${{ secrets.QA_SLACK_API_KEY }}
payload: |

Check failure on line 206 in .github/workflows/ci-core.yml

View workflow job for this annotation

GitHub Actions / Validate Github Action Workflows 

[actionlint] reported by reviewdog 🐶
property "job_id" is not defined in object type {action: string; action_path: string; action_ref: string; action_repository: string; action_status: string; actor: string; actor_id: string; api_url: string; artifact_cache_size_limit: number; base_ref: string; env: string; event: object; event_name: string; event_path: string; graphql_url: string; head_ref: string; job: string; output: string; path: string; ref: string; ref_name: string; ref_protected: bool; ref_type: string; repository: string; repository_id: string; repository_owner: string; repository_owner_id: string; repository_visibility: string; repositoryurl: string; retention_days: number; run_attempt: string; run_id: string; run_number: string; secret_source: string; server_url: string; sha: string; state: string; step_summary: string; token: string; triggering_actor: string; workflow: string; workflow_ref: string; workflow_sha: string; workspace: string} [expression]

Raw Output:
e:.github/workflows/ci-core.yml:206:374: property "job_id" is not defined in object type {action: string; action_path: string; action_ref: string; action_repository: string; action_status: string; actor: string; actor_id: string; api_url: string; artifact_cache_size_limit: number; base_ref: string; env: string; event: object; event_name: string; event_path: string; graphql_url: string; head_ref: string; job: string; output: string; path: string; ref: string; ref_name: string; ref_protected: bool; ref_type: string; repository: string; repository_id: string; repository_owner: string; repository_owner_id: string; repository_visibility: string; repositoryurl: string; retention_days: number; run_attempt: string; run_id: string; run_number: string; secret_source: string; server_url: string; sha: string; state: string; step_summary: string; token: string; triggering_actor: string; workflow: string; workflow_ref: string; workflow_sha: string; workspace: string} [expression]
channel: ${{ secrets.SLACK_TEAM_CORE_CHANNEL_ID}}
text: ":red-warn: golangci-lint: module `${{ matrix.modules }}` has ${{ steps.golang-lint.outputs.golang-report-issue-count }} issues across ${{ steps.golang-lint.outputs.golang-report-file-count }} files\n<${{ format('https://github.com/{0}/actions/runs/{1}/jobs/{2}', github.repository, github.run_id, github.job_id) }}|View Run> - <${{ steps.golang-lint.outputs.golang-report-artifact-url }}|Download Report>\n*By severity*: ${{ steps.golang-lint.outputs.golang-report-per-severity-count }}\n*By linter*: ${{ steps.golang-lint.outputs.golang-report-per-source-count }}"

Expand Down Expand Up @@ -346,7 +346,7 @@
id: wasm-key
shell: bash
run: |
echo "key=${{ runner.os }}-${{ runner.arch }}-wasmcache-${{ hashFiles('go.mod', 'go.sum', 'core/internal/testutils/wasmtest/**', 'core/capabilities/compute/**', 'core/services/workflows/**') }}" >> "$GITHUB_OUTPUT"
echo "key=${{ runner.os }}-${{ runner.arch }}-wasmcache-${{ hashFiles('go.mod', 'go.sum', 'core/capabilities/compute/**', 'core/services/workflows/**', 'system-tests/lib/**') }}" | tee -a "$GITHUB_OUTPUT"

- name: Restore WASM test binaries cache
if: ${{ matrix.type.should-run == 'true' }}
Expand Down Expand Up @@ -592,7 +592,7 @@
# Not a scheduled event - no frequencies to set. They default to false.
exit 0
fi

curent_day=$(date +%u)
if [ "$curent_day" -ge 6 ]; then
# Weekend
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/cre-mixed-env-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@
})')
echo "matrix=$matrix" | tee -a "${GITHUB_OUTPUT}"

run-mixed-env-tests:

Check failure on line 142 in .github/workflows/cre-mixed-env-tests.yaml

View workflow job for this annotation

GitHub Actions / Validate Github Action Workflows 

[actionlint] reported by reviewdog 🐶
could not parse as YAML: did not find expected key [syntax-check]

Raw Output:
e:.github/workflows/cre-mixed-env-tests.yaml:142:4: could not parse as YAML: did not find expected key [syntax-check]
name: ${{ matrix.tests.test_name }} (mixed-env)
permissions:
contents: read
Expand Down Expand Up @@ -173,14 +173,14 @@
persist-credentials: false

- name: Set up Go
id: setup-go
uses: smartcontractkit/.github/actions/ctf-setup-go@fa1d48a33e24f9b3b9f8c52e99a578a4597cb2a5 # v0.4.0

uses: actions/setup-go@v7
with:
go_mod_path: system-tests/tests/go.mod
go-version-file: system-tests/tests/go.mod
# No Go build/restore: mixed-env runs the pre-compiled cre-smoke.test binary
# (restored below), and the non-determinism gate reuses that same binary — so
# nothing is compiled here and cache restore would only waste ~30-60s per run.
no_cache: true
cache: false

- name: Restore Pre-Compiled Test Binaries from S3 Cache
uses: actions/cache/restore@v6
Expand Down
9 changes: 4 additions & 5 deletions .github/workflows/cre-regression-system-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@

echo "matrix=$tests" | tee -a "${GITHUB_OUTPUT}"

run-system-tests:

Check failure on line 112 in .github/workflows/cre-regression-system-tests.yaml

View workflow job for this annotation

GitHub Actions / Validate Github Action Workflows 

[actionlint] reported by reviewdog 🐶
could not parse as YAML: did not find expected key [syntax-check]

Raw Output:
e:.github/workflows/cre-regression-system-tests.yaml:112:4: could not parse as YAML: did not find expected key [syntax-check]
name: ${{ matrix.tests.test_name }} ${{ matrix.tests.topology != '' && format(' ({0})', matrix.tests.topology) || '' }}
permissions:
contents: read
Expand Down Expand Up @@ -158,11 +158,10 @@
persist-credentials: false

- name: Set up Go
id: setup-go
uses: smartcontractkit/.github/actions/ctf-setup-go@fa1d48a33e24f9b3b9f8c52e99a578a4597cb2a5 # v0.4.0
uses: actions/setup-go@v7
with:
go_mod_path: system-tests/tests/go.mod
no_cache: true
go-version-file: system-tests/tests/go.mod
cache: false

- name: Restore Pre-Compiled Test Binaries from S3 Cache
uses: actions/cache/restore@v6
Expand Down Expand Up @@ -268,7 +267,7 @@

exit_code="$?"
if [ "$exit_code" -eq 0 ]; then
echo "tests_result=✅ Tests passed" >> "$GITHUB_OUTPUT"
echo "tests_result=✅ Tests passed" | tee -a "$GITHUB_OUTPUT"
fi

echo "⚠️⚠️⚠️ Add 'skip-e2e-regression' label to skip this step if necessary ⚠️⚠️⚠️"
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/cre-system-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@
--attempt='${{ github.run_attempt }}' \
--github-output

run-system-tests:

Check failure on line 102 in .github/workflows/cre-system-tests.yaml

View workflow job for this annotation

GitHub Actions / Validate Github Action Workflows 

[actionlint] reported by reviewdog 🐶
could not parse as YAML: did not find expected key [syntax-check]

Raw Output:
e:.github/workflows/cre-system-tests.yaml:102:4: could not parse as YAML: did not find expected key [syntax-check]
name: ${{ matrix.tests.test_name }}
permissions:
contents: read
Expand Down Expand Up @@ -138,11 +138,11 @@
persist-credentials: false

- name: Set up Go
id: setup-go
uses: smartcontractkit/.github/actions/ctf-setup-go@fa1d48a33e24f9b3b9f8c52e99a578a4597cb2a5 # v0.4.0

uses: actions/setup-go@v7
with:
go_mod_path: system-tests/tests/go.mod
no_cache: true
go-version-file: system-tests/tests/go.mod
cache: false

- name: Restore Pre-Compiled Test Binaries from S3 Cache
uses: actions/cache/restore@v6
Expand Down
18 changes: 10 additions & 8 deletions .github/workflows/integration-in-memory-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,14 +186,17 @@ jobs:

- name: Set up Go for compilation
id: setup-go
uses: smartcontractkit/.github/actions/ctf-setup-go@fa1d48a33e24f9b3b9f8c52e99a578a4597cb2a5 # v0.4.0
uses: actions/setup-go@v7
with:
go_mod_path: integration-tests/go.mod
cache_key_id: integration-tests-v1
cache_builds: true
cache_restore_only: ${{ github.ref_name != 'develop' }}
should_tidy: false
gati_token: ${{ steps.github-token.outputs.access-token }}
go-version-file: integration-tests/go.mod
cache: true
cache-dependency-path: |
system-tests/tests/go.sum
integration-tests/go.sum

- name: Configure Go private modules
shell: bash
run: go env -w GOPRIVATE=github.com/smartcontractkit/*

- name: Compile E2E test binaries and tools
shell: bash
Expand All @@ -202,7 +205,6 @@ jobs:
# Restored mod cache contains shallow VCS clones that can be in an
# inconsistent state after tar restore, breaking fetches of new
# pseudo-versions ("shallow file has changed since we read it").
# Drop them; only private repos re-clone, public zips stay cached.
rm -rf "$(go env GOMODCACHE)/cache/vcs"
mkdir -p integration-tests/bin
(cd integration-tests && go test -c -ldflags="-s -w" -o ./bin/ccip-inmemory.test ./smoke/ccip)
Expand Down
52 changes: 39 additions & 13 deletions .github/workflows/integration-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -512,7 +512,7 @@
id-token: write
contents: read
if: needs.test-setup.outputs.cre-should-run == 'true' && needs.test-setup.outputs.cre-run-mixed-env == 'true'
uses: ./.github/workflows/cre-mixed-env-tests.yaml

Check failure on line 515 in .github/workflows/integration-tests.yml

View workflow job for this annotation

GitHub Actions / Validate Github Action Workflows 

[actionlint] reported by reviewdog 🐶
error while parsing reusable workflow "./.github/workflows/cre-mixed-env-tests.yaml": yaml: line 142: did not find expected key [expression]

Raw Output:
e:.github/workflows/integration-tests.yml:515:11: error while parsing reusable workflow "./.github/workflows/cre-mixed-env-tests.yaml": yaml: line 142: did not find expected key [expression]
with:
ecr: "sdlc"
chainlink_image_repository_path: ${{ inputs.ecr_name || 'chainlink-integration-tests' }}
Expand Down Expand Up @@ -541,7 +541,7 @@
}}
needs:
- test-setup
runs-on: runs-on=${{ github.run_id }}-compile/cpu=32/ram=64/family=c7i+c8i/spot=co/volume=100GB/extras=s3-cache
runs-on: runs-on=${{ github.run_id }}-compile/cpu=32/ram=64/family=c8in+c6in/spot=co/volume=100gb:gp3:500mbs:4000iops/extras=s3-cache
environment:
name: integration
deployment: false
Expand Down Expand Up @@ -571,19 +571,33 @@

- name: Set up Go for compilation
id: setup-go
uses: smartcontractkit/.github/actions/ctf-setup-go@fa1d48a33e24f9b3b9f8c52e99a578a4597cb2a5 # v0.4.0
uses: actions/setup-go@v7
with:
# Key the cache on go.sum hash; restore-only on PRs to eliminate upload overhead and cache thrashing.
go_mod_path: ./system-tests/tests/go.mod
cache_key_id: integration-tests-v1
cache_builds: true
cache_restore_only: ${{ github.ref_name != 'develop' }}
should_tidy: false
no_cache: false
gati_token: ${{ steps.github-token.outputs.access-token }}

# No cache-hit short-circuit: ctf-setup-go@v0.4.0 exposes no `cache-hit` output,
# and this job's purpose is to (re)populate the shared build cache regardless.
go-version-file: system-tests/tests/go.mod
cache: true
cache-dependency-path: |
system-tests/tests/go.sum
integration-tests/go.sum

- name: Configure Go private modules
shell: bash
run: go env -w GOPRIVATE=github.com/smartcontractkit/*

- name: Compute WASM cache key
id: wasm-key
shell: bash
run: |
echo "key=${{ runner.os }}-${{ runner.arch }}-wasmcache-${{ hashFiles('go.mod', 'go.sum', 'system-tests/tests/go.mod', 'system-tests/tests/go.sum', 'system-tests/lib/**', 'core/capabilities/compute/**', 'core/services/workflows/**') }}" | tee -a "$GITHUB_OUTPUT"

- name: Restore WASM test binaries cache
id: wasm-cache-restore
uses: actions/cache/restore@v6
with:
path: .wasm-cache/
key: ${{ steps.wasm-key.outputs.key }}
restore-keys: |
${{ runner.os }}-${{ runner.arch }}-wasmcache-

- name: Compile E2E tests
shell: bash
env:
Expand All @@ -594,6 +608,11 @@
mkdir -p "$GITHUB_WORKSPACE/.gotmp"
export GOTMPDIR="$GITHUB_WORKSPACE/.gotmp"

# Restored mod cache contains shallow VCS clones that can be in an
# inconsistent state after tar restore, breaking fetches of new
# pseudo-versions ("shallow file has changed since we read it").
rm -rf "$(go env GOMODCACHE)/cache/vcs"

mkdir -p "$GITHUB_WORKSPACE/system-tests/tests/bin"
mkdir -p "$GITHUB_WORKSPACE/integration-tests/bin"

Expand Down Expand Up @@ -641,6 +660,13 @@

rm -rf "$GITHUB_WORKSPACE/.gotmp"

- name: Save WASM test binaries cache
if: steps.wasm-cache-restore.outputs.cache-hit != 'true'
uses: actions/cache/save@v6
with:
path: .wasm-cache/
key: ${{ steps.wasm-key.outputs.key }}

- name: Cache Pre-Compiled CRE Test Binaries
if: needs.test-setup.outputs.cre-should-run == 'true'
uses: actions/cache/save@v6
Expand All @@ -665,7 +691,7 @@
id-token: write
contents: read
if: needs.test-setup.outputs.ccip-should-run == 'true'
uses: ./.github/workflows/ccip-system-tests.yaml

Check failure on line 694 in .github/workflows/integration-tests.yml

View workflow job for this annotation

GitHub Actions / Validate Github Action Workflows 

[actionlint] reported by reviewdog 🐶
error while parsing reusable workflow "./.github/workflows/ccip-system-tests.yaml": yaml: line 127: did not find expected key [expression]

Raw Output:
e:.github/workflows/integration-tests.yml:694:11: error while parsing reusable workflow "./.github/workflows/ccip-system-tests.yaml": yaml: line 127: did not find expected key [expression]
with:
ecr: "sdlc"
chainlink_image_repository_path: ${{ inputs.ecr_name || 'chainlink-integration-tests' }}
Expand Down
2 changes: 1 addition & 1 deletion .golangci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,6 @@ formatters:
- standard
- default
- prefix(github.com/smartcontractkit/)
- prefix(github.com/smartcontractkit/chainlink)
- prefix(github.com/smartcontractkit/chainlink/)

@Tofel Tofel Aug 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rationale?

@jmank88 jmank88 Aug 17, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is deliberate for historical reasons. Please don't undo

Suggested change
- prefix(github.com/smartcontractkit/chainlink/)
- prefix(github.com/smartcontractkit/chainlink)

exclusions:
generated: lax
Loading
Loading