Skip to content

fix(envd): bind HTTP listener before sync init chain to eliminate startup race - #3611

Open
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/envd-listener-first-startup
Open

fix(envd): bind HTTP listener before sync init chain to eliminate startup race#3611
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/envd-listener-first-startup

Conversation

@AdaAibaby

Copy link
Copy Markdown
Contributor

Problem

Closes #3610

run() in packages/envd/main.go calls s.ListenAndServe() as the last statement, after a long synchronous init chain. Any blocking call in that chain — most concretely writeCgroupProp writing memory.high/memory.max to cgroupv2 under host memory pressure — leaves port :49983 returning ECONNREFUSED for the entire init duration while the process appears alive.

Verified on dev (main branch, Linux 6.8, cgroupv2):

# 200ms synthetic init delay, current behaviour
[0–200ms]  ECONNREFUSED — process alive but no listener
[201ms]    port open

# same delay, with this fix
[0ms]      port open — kernel queues SYNs during init
[201ms]    port open — Serve() starts accepting

Changes

packages/envd/main.go (+19 / -1)

  • Call net.Listen(tcp, addr) at the top of run(), before any blocking initializer.
    Incoming connections are queued in the kernel accept backlog (SO_BACKLOG) and dispatched once s.Serve(ln) starts — HTTP semantics are unchanged.
  • Replace s.ListenAndServe() with s.Serve(ln).
  • Add defer ln.Close() so the port is released on any early-return error path.
  • Add stderr phase logs (envd startup: creating cgroup manager / ready) around the most likely blocking point, so a hung init appears in logs rather than a silent stall.

packages/envd/pkg/version.go

  • Bump 0.7.00.7.1 (behaviour change per project convention).

Why this is safe

net.Listener + Server.Serve is exactly what ListenAndServe does internally — this splits it into two steps with no semantic difference. Connections that arrive while init is still running are held in the kernel backlog and processed normally once Serve starts accepting.

Testing

  • All existing tests pass (the two TestIsPathOnNetworkMount / TestCreateWatcherOnNetworkMount failures are pre-existing: bindfs not installed in dev).
  • Manually confirmed on dev: envd: HTTP listener bound on :49983 appears as the first stderr line, port visible in ss -tlnp within 50 ms of process start, before envd startup: cgroup manager ready.

…rtup race

Move net.Listen to the top of run() so port :49983 is reachable from
process start. Incoming connections are queued in the kernel accept
backlog and dispatched once s.Serve(ln) begins — no change in HTTP
semantics.

Previously s.ListenAndServe() was called last, after a long synchronous
chain (logger → filesystem service → cgroup manager → process service →
api → port scanner/forwarder). Any blocking call in that chain — most
plausibly writeCgroupProp writing memory.high/memory.max under host
memory pressure — left the port returning ECONNREFUSED for the entire
init duration while the process appeared alive. Orchestrators and health
checks that TCP-dial :49983 could not distinguish this from a crash.

Also add stderr phase logs around createCgroupManager (the most likely
blocking point) so a hung init surfaces in logs rather than a silent
stall.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(envd): HTTP listener bound after blocking init chain — connect-refused during startup race

2 participants