Flaky CLI E2E tests: PTY `exit` resolves before stdout drains

[tobyhede]

Symptom

CLI E2E tests intermittently fail with the PTY output buffer only partially captured. Example: runner-aware-help.e2e.test.ts > --help — runner-aware Usage + Examples > with npm_config_user_agent='pnpm/9.0.0' asserts the captured output contains pnpm dlx stash init, but on a failing run the captured output was only the truncated env banner — the actual help text had not yet landed in the buffer when the assertion read it.

Mechanism

In packages/cli/tests/helpers/pty.ts, the exit promise resolves directly on pty.onExit:

const exit = new Promise<{ exitCode: number; signal?: number }>((res) => {
  pty.onExit((e) => res(e))
})

Tests then do:

const { exitCode } = await r.exit
expect(exitCode).toBe(0)
expect(r.output).toContain(`Usage: ${label} stash`)

r.output is a getter over the raw buffer that is accumulated in pty.onData. onExit is not ordered after the final onData — node-pty (and the underlying OS pipe) can deliver the exit event while trailing stdout is still in flight. On a slower/loaded runner the trailing stdout (the actual help text) has not been appended to raw when the assertion reads r.output, so the assertion sees a truncated buffer and fails.

Proof it is a nondeterministic flake (not a regression)

The same commit re-run produced Node 24 fail → pass and Node 22 pass → fail — the failure hops between matrix nodes on identical code. This is a classic timing race, not a logic regression.

Blast radius

Tests using the unsafe pattern (await r.exit then read r.output/r.raw, without waitFor):

Test file	output reads	waitFor uses	Exposure
smoke.e2e.test.ts	16	0	High
runner-aware-help.e2e.test.ts	5	0	High (observed flake)
doctor.e2e.test.ts	4	0	High
database-url.e2e.test.ts	5	1	Partial
auth-login-cancel.e2e.test.ts	1	1	Partial

The helper already exposes a safe waitFor(match) that polls onData until text appears; the racy tests simply do not use it before asserting on post-exit output.

History

• Flaky test introduced by 0560049 — test(cli): E2E coverage for runner-aware help rendering (Dan Draper, 2026-05-01).
• Latent helper race predates it: a4532a56 — test(cli): add node-pty E2E suite for the stash CLI (Dan Draper, 2026-04-30). Every helper version since resolves exit on onExit without draining the data stream first.

Proposed fix

Fix the helper once so all suites benefit: on onExit, capture the exit event but resolve the exit promise only after the onData stream has gone idle (no data for a short settle window, e.g. ~50ms, with a hard cap), so buffered/in-flight stdout lands in raw before any test reads r.output. A bare setImmediate/microtask is insufficient because OS pipe data may still be in flight after the exit event. This preserves the { exitCode, signal? } return shape and requires no changes to individual tests. Test-infra-only; no changeset (does not affect published output).