Skip to content

fix(ci): build app on 16vcpu runner to stop cold-cache OOM kills #5944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
waleedlatif1 merged 1 commit into from
Jul 24, 2026
Merged

fix(ci): build app on 16vcpu runner to stop cold-cache OOM kills #5944

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 10 additions & 3 deletions .github/workflows/test-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -197,11 +197,18 @@ jobs:
# harmless), but the Turbo cache gets its own key: with a shared key, only
# the last committer's new entries survive each run, so the test and build
# Turbo entries would evict each other nondeterministically.
# Same next build as the app image, so it needs the same larger runner in
# GitHub mode: it peaks ~21.9 GB and SIGKILLs on the free 16 GB ones.
# Same next build as the app image, so it needs a large runner. Peak RSS
# tracks Turbo/Turbopack cache warmth, and it is the COLD case that sizes the
# runner: warm peaks ~12 GB, partial ~28 GB, and a cold full rebuild peaked
# 51 GB. The 8vcpu tier only has 30.4 GB, so cold-cache runs OOM-killed the VM
# (oom_count 1, memory p100 ~30.5 GB, ~99% of the tier) — the job burned 8-15
# min and died, while warm runs finished under 8 min. NODE_OPTIONS'
# --max-old-space-size only caps Node's JS heap; the bulk is native Turbopack
# worker memory, so the cap cannot prevent this. 16vcpu = 60.8 GB fits the
# 51 GB cold peak with headroom. Keep the test job on 8vcpu; its memory is fine.
build:
name: Build App
runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-8vcpu-ubuntu-2404' || 'linux-x64-8-core' }}
runs-on: ${{ (vars.CI_PROVIDER == '' || vars.CI_PROVIDER == 'blacksmith') && 'blacksmith-16vcpu-ubuntu-2404' || 'linux-x64-8-core' }}

@greptile-apps greptile-apps Bot Jul 24, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 GitHub fallback remains undersized

When CI_PROVIDER selects GitHub or contains an unrecognized value, this conditional still assigns the documented 32 GB linux-x64-8-core runner to a build whose cold-cache peak is documented as 51 GB, causing the fallback Build App job to remain vulnerable to an OOM termination.

@waleedlatif1 waleedlatif1 Jul 24, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid, and the finding stands — flagging rather than silently resolving.

The fallback is latent, not live: it only activates when vars.CI_PROVIDER is set to a value other than empty or blacksmith, which is not the case today (every run in the last 7d went to Blacksmith). So this did not affect the OOM failures this PR fixes.

But the concern is real, and arguably worse than it looks: the GitHub fallback is the escape hatch you would reach for precisely when Blacksmith is unavailable, so it failing under exactly the cold-cache conditions that motivated this PR defeats its purpose. linux-x64-8-core is 32 GB, no better than the 30.4 GB tier that was OOM-killing.

Not fixed here because picking the right label needs information not in the repo: linux-x64-8-core is the only larger-runner label used anywhere in these workflows, and whether a bigger x64 label is provisioned in the org runner group is not visible from the workflow files. Tracking as a follow-up rather than guessing a label that may not resolve — an unresolvable runs-on label hangs the job in queued indefinitely, which is a worse failure mode than the latent OOM.

timeout-minutes: 15

steps:
Expand Down
Loading