`opencode web` permanently hangs (event loop dead, HTTP timeouts) — main thread deadlocks inside `@parcel/watcher` native subscribe during multi-project attach (Linux)

[panta82]

Description

The following summary was written by Fable, after multi-hour investigation.

---

Summary

opencode web intermittently but repeatedly becomes permanently unresponsive on Linux when a web client (re)connects and several projects bootstrap concurrently. The process stays "online" but never answers HTTP again. Live diagnosis of the hung process shows the main JS thread parked in an infinite futex wait inside a native N-API call, with no @parcel/watcher thread alive and no inotify instance created despite watcher backend … backend=inotify having been logged. Disabling the file watcher (OPENCODE_EXPERIMENTAL_DISABLE_FILEWATCHER=1) fully eliminates the hang; nothing else we tried did (see isolation matrix).

We first chased "project copy refresh" because its started line (without done) was always the last log line — that turned out to be a red herring: the refresh is a forked fiber that never gets scheduled again because the event loop is dead.

Environment

• opencode 1.17.20 (npm opencode-ai, bun-compiled opencode.exe ELF, glibc)
• Linux x86_64, kernel 6.1 (Debian 12)
• Run as: opencode web --port 4096 --hostname 127.0.0.1 --print-logs under PM2, cwd = a small stub dir
• Global DB ~437MB, ~170 sessions across projects
• Workspaces the client attaches on connect (5–7 instances bootstrap within ~5s):
  • a ~48GB umbrella directory containing many nested git repos
  • a monorepo (git, with ~10 linked git worktrees)
  • several small git repos
  • two stale paths that no longer exist on disk (client still requests them; server logs FileSystem.realPath ENOENT for those — mentioning for completeness)
• OPENCODE_EXPERIMENTAL_FILEWATCHER not set — so the only parcel-watcher subscriptions should be the per-project .git directory watches (packages/core/src/filesystem/watcher.ts, the location.vcs?.type === "git" branch). The monorepo's .git is large (packfiles + many worktree admin dirs).

Symptoms

• HTTP to the port times out forever (3–5s client timeouts); ss -ltn shows the accept queue growing (seen 18–161)
• PM2 shows the process online; restarts reproduce the hang on the next attach storm (our external health-check watchdog created a restart death-spiral)
• Log signature — everything normal, then silence mid-bootstrap:

INFO "watcher backend" directory=/srv/service-b platform=linux backend=inotify
INFO "project copy refresh started" projectID=global
INFO "project copy refresh done"    projectID=global updated=[] removed=[]
INFO "watcher backend" directory=/srv/umbrella           platform=linux backend=inotify
INFO "watcher backend" directory=/srv/umbrella/monorepo  platform=linux backend=inotify
INFO "watcher backend" directory=/srv/site-a             platform=linux backend=inotify
INFO "project copy refresh started" projectID=<umbrella-id>
INFO "project copy refresh started" projectID=<monorepo-id>
INFO "project copy refresh started" projectID=<site-a-id>
### nothing is ever logged again; HTTP dead from this moment ###

Live diagnostics of the hung process

The process is not busy — it is dead idle:

• ~0% CPU across all threads; /proc/PID/io byte counters unchanged over 10s

• Every thread in futex_wait / epoll_wait (/proc/PID/task/*/wchan)

• Main thread blocked in an infinite futex wait (/proc/PID/task/<pid>/syscall):

  202 0x335128d8 0x189 0x0 0x0 0x0 0xffffffff ...
  

  = FUTEX_WAIT_BITSET|FUTEX_PRIVATE_FLAG|FUTEX_CLOCK_REALTIME, expected val 0, timeout = NULL, futex address on [heap].

• A stack scan of the main thread (dump stack words, resolve those pointing into the binary's .text) finds napi_typeof + 37 — i.e. the main thread is blocked inside a native N-API addon call that never returns.

• Crucially, in the hung state the process has zero inotify file descriptors (/proc/PID/fdinfo) and no watcher/backend thread in /proc/PID/task — only Bun's pool threads, all idle — even though watcher backend … backend=inotify was logged for four directories. So the native subscribe died somewhere between "backend chosen" and "backend thread running": the requester waits forever on a start/handshake signal that never comes.

This reproduces identically with the fff file-picker disabled, which removes all other native modules from the picture (in that configuration the only N-API addon in play is @parcel/watcher 2.5.1).

Isolation matrix

Same host, same DB, same client attach set, one variable at a time:

fff (OPENCODE_DISABLE_FFF)	parcel watcher (OPENCODE_EXPERIMENTAL_DISABLE_FILEWATCHER)	Result
enabled	enabled	hangs (idle futex deadlock as above)
disabled	enabled	still hangs — identical signature
disabled	disabled	healthy: all workspaces attach in ~1.1s each, HTTP 3–70ms, stable under soak
enabled	disabled	healthy (heavy CPU/IO from fff indexing the 48GB tree, but never hangs)

So the watcher subscribe path is both necessary and sufficient for the hang.

Why the existing 10s guard doesn't help

packages/core/src/filesystem/watcher.ts wraps the subscription promise in Effect.timeout(SUBSCRIBE_TIMEOUT_MS):

const pending = w.subscribe(directory, callback, { ignore, backend })
return Effect.promise(() => pending).pipe(..., Effect.timeout(SUBSCRIBE_TIMEOUT_MS), ...)

but w.subscribe(...) performs its backend setup synchronously in the native call on the calling (main JS) thread before a promise ever materializes. When that setup deadlocks, the timeout never gets a chance to run — nothing does; the event loop is gone. A timeout can only protect this if the subscribe is dispatched off the main thread (or the native call is made genuinely async).

Suspected mechanism

@parcel/watcher's Linux backend start involves a shared backend singleton and a condvar handshake with the backend thread. With several projects bootstrapping concurrently (multiple subscribe calls close together, watched dirs including a very large .git), the requester ends up waiting forever on the started-signal while no backend thread exists. The classic watcher/debouncer deadlock (parcel-bundler/watcher#187) was fixed in 2.x, so this appears to be a different variant — possibly specific to Bun's NAPI threading. Related-looking reports: #23190 (watcher hang on huge subtrees, closed not-planned), #23009 (FileWatcher subscribe timeouts at startup).

Reproduction sketch

Not minimal, but reliably reproduced here (~every few attach storms):

1. Global DB with 5+ projects, at least one being a multi-GB tree and one a git repo with several linked worktrees / large .git
2. opencode web on Linux
3. Connect a web client that has all those workspaces open, so instances bootstrap concurrently
4. Within seconds of the watcher backend lines, the server goes permanently silent; HTTP accept queue grows

Workaround

OPENCODE_EXPERIMENTAL_DISABLE_FILEWATCHER=1 — with it, the same attach storm completes in seconds and the server is stable. (Cost: no live file-change events.)

Ask

• Run w.subscribe() off the main JS thread, or otherwise make the guard timeout actually effective, so a wedged native subscribe degrades to "no watcher for that dir" instead of killing the whole server
• Consider surfacing watcher subscribe failures/timeouts in health output — the current failure mode (process alive, event loop dead, last log line pointing at an unrelated subsystem) is very hard to diagnose

Plugins

No response

OpenCode version

1.17.20

Steps to reproduce

No response

Screenshot and/or share link

No response

Operating System

Linux 6.1.0-31-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.128-1 (2025-02-07) x86_64 GNU/Linux

Terminal

Wezterm, remote session

[github-actions]

This issue might be a duplicate of or closely related to existing issues. Please check:

• Opencode hangs at startup if a .git repo is present and inotify user instances run out #16610: Same symptom (opencode hangs on Linux with @parcel/watcher / inotify, same last-log-line pattern of watcher backend lines followed by silence). The root cause domain is identical — the native inotify/parcel-watcher subscribe path blocking the main JS thread on Linux. The mechanism here (concurrent subscribes during multi-project attach causing a futex deadlock) appears to be a distinct variant, but worth confirming whether the issues are related before pursuing separate fixes.