fix(log_lib): run_with_log hangs forever when an orphan holds the child's pipe

[tigist-far]

Summary

run_with_log can hang forever. It reads the child's output until EOF and only then calls proc.wait(), on the assumption — stated in the code — that "Stream processing already waited for process completion". That assumption breaks whenever a grandchild outlives the child: EOF requires every write end of the pipe to be closed, including the ones the grandchild inherited. The reader stays blocked, proc.wait() is never reached, and the function never returns a returncode. There is no timeout on that path.

A watchdog thread now waits for the child, gives the readers a grace period to drain, and only if they are still blocked kills the leftover process group to force EOF.

Why it matters

Observed in production on a two-node training job. The ranks aborted on an RDMA transport error and torchrun exited, but five orphaned multiprocessing.spawn children kept the stdout pipe open:

5 × spawn_main   PPid=1   fd1,fd2 -> pipe:[2925536000]
ray::head        fd35    -> pipe:[2925536000]   wchan=pipe_read

The Ray task running the command never finished, so the managed-jobs controller reported JobStatus.RUNNING for two hours — #RECOVERIES 0 — while 16 GPUs sat idle. Autorecovery cannot fire on a job that never reports failure.

Why the process group

kill_children_processes cannot reach these processes: it walks the process tree, and orphans are reparented to init, so they have left it. The group still reaches them — start_new_session=True makes the child a session and group leader, so the group id is simply its pid, and neither reparenting nor reaping the leader moves the orphans out of that group.

Using proc.pid directly also avoids a race: a short-lived child is already a zombie by the time the watchdog starts, and os.getpgid() on it can fail. (My first attempt did exactly that, and silently degraded to a no-op.)

Why the trigger is narrow

The kill fires only when the child has exited and the readers are still blocked after the grace period. That distinction is load-bearing: an unconditional group-kill after exit would break commands that deliberately leave daemons in the group, ray start among them. When the narrow condition does hold, the caller was already hung forever, so forcing EOF cannot lose anything that was still working.

Testing

New test: a child that exits while a background grandchild holds its stdout.

	Result
With the fix	run_with_log returns the child's exit code after the grace period
Without the fix	hangs; killed at 90 s
Full test_log_lib.py	13 passed in 4.9 s

yapf (pinned 0.32.0), isort, whitespace hooks clean.