fix(ci): clean up orphaned apt/dpkg processes between Playwright install retries - #1265
Merged
Conversation
…all retries The outer `timeout 10m` wrapping npx playwright install --with-deps kills the top-level node process when it fires, but the underlying apt-get runs as root via sudo in a separate process tree that timeout doesn't reach. When an attempt genuinely times out (a slow mirror, not a full stall), the orphaned apt-get process keeps running in the background holding the dpkg lock -- so the very next retry attempt fails immediately with "Could not get lock /var/lib/dpkg/lock-frontend. It is held by process <pid> (apt-get)", burning all remaining retries in seconds with no chance of success. Same underlying class of bug as the nick-fields/retry EPERM issue fixed previously: killing a process that has escalated privilege via sudo doesn't reliably reach its children. Before each retry, kill any lingering apt/apt-get/dpkg processes and clear stale lock files so the next attempt starts clean instead of immediately failing on a lock held by the previous attempt's orphan. Observed on PR #1262: both WebKit shards' first install attempt hit the full 10-minute timeout (mirror slowness, not a hang), then both retries failed in ~17s each with the dpkg lock error above.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
Contributor
|
| Severity | Count |
|---|---|
| 🔴 Critical | 0 |
| 🟠 High | 2 |
| 🟡 Medium | 5 |
| 🟢 Low | 2 |
| Total | 13 |
📎 Artifacts
- SBOM (CycloneDX JSON) and Grype results available in workflow artifacts
Generated by Supply Chain Verification workflow • View Details
Contributor
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR #1262's re-run surfaced a new bug in the apt-reliability hardening from #1263, on both WebKit shards (shard 1, shard 2):
The outer
timeout 10mwrappingnpx playwright install --with-depskills the top-level node process when it fires, but the underlyingapt-getruns as root viasudoin a separate process tree thattimeoutdoesn't reach. When attempt 1 genuinely times out, the orphanedapt-getprocess keeps running in the background holding the dpkg lock — so attempt 2 fails immediately on the lock, and so does attempt 3, burning all remaining retries in seconds with zero chance of success.This is the same underlying class of bug as the
nick-fields/retryEPERMissue fixed in #1259: killing a process that has escalated privilege viasudodoesn't reliably reach its children.Fix
Before each retry (all 10 install sites), clean up any lingering apt/apt-get/dpkg processes and clear stale lock files so the next attempt starts clean instead of immediately failing on a lock held by the previous attempt's orphan:
Test plan
actionlintclean (including shellcheck on the embedded script)lefthook run pre-commitclean