Skip to content

feat(cli): --debug-cli reports active handles that block process exit#1651

Open
sai-ray wants to merge 12 commits into
mainfrom
sai/debug-cli-handle-tracker
Open

feat(cli): --debug-cli reports active handles that block process exit#1651
sai-ray wants to merge 12 commits into
mainfrom
sai/debug-cli-handle-tracker

Conversation

@sai-ray

@sai-ray sai-ray commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

In some CI environments a cdk command finishes its work but the process never exits, and the job hangs until the runner kills it (issue #1217). It's intermittent and environment-specific, so there's no way today to see what is keeping the process alive.

This adds that visibility behind the existing --debug-cli flag. If the process is still alive shortly after the work is done, the CLI prints the handles holding it open and where each was created. It's a diagnostic only and on a normal run nothing is printed.

How it works

One module, lib/cli/debug-handles.ts, built on Node's async_hooks. LeakedHandleTracker watches every async resource as it's created (recording its type and creation stack) and forgets it when it's cleaned up. At exit, report() lists the ones still keeping the process alive, skipping handles that were garbage collected or .unref()'d since those don't block exit.

Wiring in cli.ts is two lines: enable tracking at startup when --debug-cli is set, and schedule the report behind a short, .unref()'d grace timer:

if (argv.debugCli) {
  setTimeout(() => {
    void reportLeakedHandles(ioHelper);
  }, HANDLE_DUMP_GRACE_MS).unref();
}

The grace timer is the key bit: a handle open the instant work finishes might just be closing on a clean shutdown. The timer lets the normal exit path finish first, and because it's .unref()'d it never fires on a clean run. The report only shows up when the process is genuinely still alive after the window, which is likely the hang.

What the user sees

4 handles still keeping the CLI process alive:

# Timeout (timer from setTimeout or setInterval)
  created in scheduleRefresh()
  call stack:
    setInterval(() => this.refresh(), 60_000)

# FSEVENTWRAP (file-system watcher)
  created in watchConfig()
  call stack:
    return fs.watch(configPath, onChange)

# TLSWRAP (open TLS connection)
  call stack:
    return socket.addRequest(req, connectOpts)

# TCPWRAP (open network connection)
  (no application stack frames)

Each entry shows the type, a plain-language description, the function it was created in, and the line of code (greyed). Handles opened entirely inside Node show (no application stack frames), they're still listed so nothing is hidden.

Tests

test/cli/debug-handles.test.ts uses real resources (only the IoHost is faked): a leaked timer and a real leaked TCP connection are reported, .unref()'d handles are excluded, and promise noise is filtered out.

Checklist

  • Unit tests added
  • This change contains a major version upgrade for a dependency and I confirm all breaking changes are addressed
    • Release notes for the new version:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@github-actions

github-actions Bot commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@codecov-commenter

codecov-commenter commented Jun 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.69767% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.09%. Comparing base (9f04d41) to head (0e259af).
⚠️ Report is 38 commits behind head on main.

Files with missing lines Patch % Lines
packages/aws-cdk/lib/cli/debug-handles.ts 94.64% 14 Missing and 1 partial ⚠️
packages/aws-cdk/lib/cli/cli.ts 38.09% 13 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1651      +/-   ##
==========================================
+ Coverage   88.98%   89.09%   +0.10%     
==========================================
  Files          77       78       +1     
  Lines       11354    11734     +380     
  Branches     1591     1631      +40     
==========================================
+ Hits        10103    10454     +351     
- Misses       1221     1249      +28     
- Partials       30       31       +1     
Flag Coverage Δ
suite.unit 89.09% <90.69%> (+0.10%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sai-ray sai-ray changed the title feat(cli): report async handles keeping the process alive with --debug-cli feat(cli): --debug-cli reports active handles that block process exit Jun 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Total lines changed 1252 is greater than 1000. Please consider breaking this PR down.

@rix0rrr

rix0rrr commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

What the user sees

Can you confirm this with a packaged CLI? Because we bundle all the source, and I'm pretty sure source locations are going to end up as cli.js:1:35891.

I don't think we have to go to source maps (though we can), but we should at least have the trace of all functions.

@sai-ray

sai-ray commented Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

What the user sees

Can you confirm this with a packaged CLI? Because we bundle all the source, and I'm pretty sure source locations are going to end up as cli.js:1:35891.

I don't think we have to go to source maps (though we can), but we should at least have the trace of all functions.

In the bundle the locations just came out as index.js. Function names survive bundling fine, so updated the report to lead with the function and shows the line of code that opened the handle:

# Timeout (timer from setTimeout or setInterval)
  created in scheduleRefresh()
  call stack:
    setInterval(() => this.refresh(), 60_000)

I'd like to get the actual file and line too and was curious how you'd approach it. From what I found, shipping a source map would do it since node-backpack already passes a sourcemap option through to esbuild, it just isn't exposed on the pack command in use, and the tracker would switch to reading error.stack instead of CallSites since that path is sourcemap aware. Any ideas on the cleanest way, or something simpler?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants