Skip to content

fix(ibmcloud): remove PID limits to prevent EAGAIN during heavy builds#856

Open
adrianriobo wants to merge 1 commit into
redhat-developer:mainfrom
adrianriobo:fix-ibm-limits
Open

fix(ibmcloud): remove PID limits to prevent EAGAIN during heavy builds#856
adrianriobo wants to merge 1 commit into
redhat-developer:mainfrom
adrianriobo:fix-ibm-limits

Conversation

@adrianriobo

Copy link
Copy Markdown
Collaborator

Problem

On ppc64le (and s390x) runners, building wheel packages fails with:

```
ccache: error: Failed to fork: Resource temporarily unavailable
gcc: error: posix_spawn: Operation not permitted
```

Root cause: Pillow compiles 60+ C extension files. Each file spawns a
`ccache → gcc → cc1 → as` process chain (4 processes per file). Running
two concurrent jobs exhausts the container's cgroup `pids.max`, which
Podman inherits from the system default (~2048 on RHEL 9).

Three enforcement layers were all unset, each independently capable of
triggering the failure:

Layer Before After
Podman `pids_limit` in `containers.conf` system default `-1` (unlimited)
Docker executor `--docker-pids-limit` not set (inherits Podman default) `-1` (unlimited)
Systemd `TasksMax` on `gitlab-runner.service` systemd default (512–4096) `infinity`

Changes

  • `pkg/integrations/gitlab/snippet-linux.sh`: adds a `pids_limit = -1`
    block in `/etc/containers/containers.conf` (idempotent, covers all DNS
    code-path branches); adds `--docker-pids-limit -1` to `gitlab-runner register`
    so the Docker executor explicitly removes the pids cgroup on each job container.
  • `pkg/provider/ibmcloud/action/ibm-power/cloud-config`: adds
    `TasksMax=infinity` to the `gitlab-runner.service.d` drop-in.
  • `pkg/provider/ibmcloud/action/ibm-z/cloud-config`: same fix applied to
    IBM Z runners to prevent the same failure class on s390x.

Test plan

  • Re-provision a ppc64le runner with these changes and trigger a Pillow wheel build
  • Confirm no EAGAIN / posix_spawn errors during C extension compilation
  • Verify s390x runner is unaffected (TasksMax=infinity is a no-op on non-constrained hosts)

🤖 Generated with Claude Code

On ppc64le (and s390x) runners, compiling Pillow's 60+ C extension files
causes ccache->gcc->cc1->as process chains that exhaust the container's
cgroup pids.max, producing EAGAIN on fork and posix_spawn failures.

Three enforcement layers are now cleared on every GitLab runner setup:

- snippet-linux.sh: sets pids_limit = -1 in /etc/containers/containers.conf
  so Podman does not apply a default ceiling to job containers.
- snippet-linux.sh: passes --docker-pids-limit -1 to gitlab-runner register
  so the Docker executor explicitly removes the pids cgroup on each job
  container it spawns.
- ibm-power/cloud-config, ibm-z/cloud-config: adds TasksMax=infinity to
  the gitlab-runner.service.d drop-in so the runner daemon's own systemd
  task limit does not cap its child process tree.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 8, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Enterprise

Run ID: 5aee5ff5-a1cc-41eb-8e54-005c5446dad8

📥 Commits

Reviewing files that changed from the base of the PR and between 20fee32 and 36effd9.

📒 Files selected for processing (3)
  • pkg/integrations/gitlab/snippet-linux.sh
  • pkg/provider/ibmcloud/action/ibm-power/cloud-config
  • pkg/provider/ibmcloud/action/ibm-z/cloud-config

📝 Walkthrough

Summary by CodeRabbit

  • Bug Fixes
    • Removed process/task limits for GitLab Runner jobs in supported environments, helping prevent failures in process-heavy builds and tooling.
    • Improved runner registration and container execution so jobs can run without inheriting restrictive PID limits.
    • Updated service settings to allow unlimited tasks, reducing issues caused by default system limits.

Walkthrough

Configuration changes remove per-container PID limits and systemd task limits for GitLab runner infrastructure. The gitlab snippet script sets containers.conf pids_limit to -1 and adds a --docker-pids-limit flag to runner registration. IBM Power and IBM Z cloud-config templates add TasksMax=infinity to the runner's systemd service override.

Changes

PID and Task Limit Removal

Layer / File(s) Summary
Podman and Docker executor PID limit removal
pkg/integrations/gitlab/snippet-linux.sh
Sets pids_limit = -1 in /etc/containers/containers.conf, creating the [containers] section if missing, and adds --docker-pids-limit -1 to the gitlab-runner register command.
Systemd TasksMax override for GitLab runner service
pkg/provider/ibmcloud/action/ibm-power/cloud-config, pkg/provider/ibmcloud/action/ibm-z/cloud-config
Adds TasksMax=infinity with explanatory comments to the GitLab runner systemd service override configuration on both IBM Power and IBM Z templates.

Estimated code review effort: 1 (Trivial) | ~5 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: removing PID limits to avoid EAGAIN during heavy builds.
Description check ✅ Passed The description matches the changeset and explains the PID limit fixes across runner, container, and systemd layers.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

Warning

Review ran into problems

🔥 Problems

Git: Failed to clone repository. Please run the @coderabbitai full review command to re-trigger a full review. If the issue persists, set path_filters to include or exclude specific files.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant