Skip to content

Avoid throwing when workers shutdown on Windows#240

Merged
Drvi merged 2 commits into
mainfrom
td-windows-worker-termination
Jul 17, 2026
Merged

Avoid throwing when workers shutdown on Windows#240
Drvi merged 2 commits into
mainfrom
td-windows-worker-termination

Conversation

@Drvi

@Drvi Drvi commented Jul 14, 2026

Copy link
Copy Markdown
Member

When porting a private package to Julia 1.12, I consistently ran into issues when all the tests pass but the test run failed with an error pointing to worker termination:

ERROR: LoadError: TaskFailedException
    nested task error: TaskFailedException
    Stacktrace:
      [1] #wait#582
        @ .\task.jl:363 [inlined]
      [2] wait
        @ .\task.jl:360 [inlined]
      [3] fetch
        @ .\task.jl:554 [inlined]
      [4] wait
        @ .\packages\ReTestItems\rFUty\src\workers.jl:145 [inlined]
      [5] close(w::ReTestItems.Workers.Worker)
        @ ReTestItems.Workers .\packages\ReTestItems\rFUty\src\workers.jl:140
      [6] manage_worker(worker::ReTestItems.Workers.Worker, proj_name::String, testitems::ReTestItems.TestItems, testitem::TestItem, cfg::ReTestItems._Config; worker_num::Int64)
        @ ReTestItems .\packages\ReTestItems\rFUty\src\ReTestItems.jl:753
        ...
        nested task error: IOError: kill: permission denied (EACCES)
        Stacktrace:
         [1] kill(p::Base.Process, signum::Int64)
           @ Base .\process.jl:627
         [2] terminate!(w::ReTestItems.Workers.Worker, from::Symbol)
           @ ReTestItems.Workers .\packages\ReTestItems\rFUty\src\workers.jl:87
         [3] process_responses(w::ReTestItems.Workers.Worker, ev::Base.Event)
           @ ReTestItems.Workers .\packages\ReTestItems\rFUty\src\workers.jl:272
         [4] (::ReTestItems.Workers.var"#18#19"{ReTestItems.Workers.Worker, Base.Event})()
           @ ReTestItems.Workers .\packages\ReTestItems\rFUty\src\workers.jl:206
        caused by: EOFError: read end of file
        Stacktrace:
         [1] read(this::Sockets.TCPSocket, ::Type{UInt8})
           @ Base .\stream.jl:1012
         [2] deserialize
           @ .\stdlib\v1.12\Serialization\src\Serialization.jl:851 [inlined]
         [3] deserialize(s::Sockets.TCPSocket)
           @ Serialization .\stdlib\v1.12\Serialization\src\Serialization.jl:838
         [4] process_responses(w::ReTestItems.Workers.Worker, ev::Base.Event)
           @ ReTestItems.Workers .\packages\ReTestItems\rFUty\src\workers.jl:254
         [5] (::ReTestItems.Workers.var"#18#19"{ReTestItems.Workers.Worker, Base.Event})()
           @ ReTestItems.Workers .\packages\ReTestItems\rFUty\src\workers.jl:206
...and 4 more exceptions.
Stacktrace:
  [1] sync_end(c::Channel{Any})
  ...

The issue seems to be that when a worker shuts down, the managing task gets an EOFError when listening for messages to deserialize from the worker, and then in a catch block it tries to terminate! the already-shut-down worker, but it seems that marking the worker as done is racy (i.e. its process_running(p) can still return true at this point), so we end up trying to kill an exited worker, which results in an access violation error on Windows.

With this patch, the issue no longer reproduces.
I also fixed a junit_xml.jl tests since some of the error messages changed on Julia 1.12, and I bumped the CI Julia version from '~1.12.0-rc1' to '1.12'

@Drvi
Drvi requested a review from nickrobinson251 July 15, 2026 09:31

@quinnj quinnj left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure what the CI failure is, but this LGTM

@Drvi
Drvi enabled auto-merge (squash) July 17, 2026 12:56
@Drvi
Drvi merged commit 1f43385 into main Jul 17, 2026
16 of 19 checks passed
@Drvi
Drvi deleted the td-windows-worker-termination branch July 17, 2026 13:19
@Drvi
Drvi restored the td-windows-worker-termination branch July 17, 2026 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants