fix(envd): bind HTTP listener before sync init chain to eliminate startup race - #3611
Open
AdaAibaby wants to merge 1 commit into
Open
fix(envd): bind HTTP listener before sync init chain to eliminate startup race#3611AdaAibaby wants to merge 1 commit into
AdaAibaby wants to merge 1 commit into
Conversation
…rtup race Move net.Listen to the top of run() so port :49983 is reachable from process start. Incoming connections are queued in the kernel accept backlog and dispatched once s.Serve(ln) begins — no change in HTTP semantics. Previously s.ListenAndServe() was called last, after a long synchronous chain (logger → filesystem service → cgroup manager → process service → api → port scanner/forwarder). Any blocking call in that chain — most plausibly writeCgroupProp writing memory.high/memory.max under host memory pressure — left the port returning ECONNREFUSED for the entire init duration while the process appeared alive. Orchestrators and health checks that TCP-dial :49983 could not distinguish this from a crash. Also add stderr phase logs around createCgroupManager (the most likely blocking point) so a hung init surfaces in logs rather than a silent stall.
AdaAibaby
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
August 31, 2026 06:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Closes #3610
run()inpackages/envd/main.gocallss.ListenAndServe()as the last statement, after a long synchronous init chain. Any blocking call in that chain — most concretelywriteCgroupPropwritingmemory.high/memory.maxto cgroupv2 under host memory pressure — leaves port:49983returningECONNREFUSEDfor the entire init duration while the process appears alive.Verified on dev (main branch, Linux 6.8, cgroupv2):
Changes
packages/envd/main.go(+19 / -1)net.Listen(tcp, addr)at the top ofrun(), before any blocking initializer.Incoming connections are queued in the kernel accept backlog (
SO_BACKLOG) and dispatched onces.Serve(ln)starts — HTTP semantics are unchanged.s.ListenAndServe()withs.Serve(ln).defer ln.Close()so the port is released on any early-return error path.envd startup: creating cgroup manager/ready) around the most likely blocking point, so a hung init appears in logs rather than a silent stall.packages/envd/pkg/version.go0.7.0→0.7.1(behaviour change per project convention).Why this is safe
net.Listener+Server.Serveis exactly whatListenAndServedoes internally — this splits it into two steps with no semantic difference. Connections that arrive while init is still running are held in the kernel backlog and processed normally onceServestarts accepting.Testing
TestIsPathOnNetworkMount/TestCreateWatcherOnNetworkMountfailures are pre-existing:bindfsnot installed in dev).envd: HTTP listener bound on :49983appears as the first stderr line, port visible inss -tlnpwithin 50 ms of process start, beforeenvd startup: cgroup manager ready.