Unix socket support by dometto · Pull Request #9337 · dask/distributed

dometto · 2026-07-27T13:07:33Z

Resolves #3630

This PR adds support for Unix Domain Sockets...

as a protocol/backend in comms
as an optional way to serve the dashboard server

The rationale for this is twofold:

improved security
improved performance

Test this branch with:

the provided tests
or by creating a cluster with the new protocol:

    cluster = LocalCluster(
        n_workers=2,
        protocol="uds",
    )
   # do something with the cluster here

New UDS backend

This is the main part of this PR. We implement a new backend UDS which subclasses the standard TCP classes. When setting protocol on a cluster/worker/scheduler to uds://, this backend is used. When a worker or scheduler is given an absolute path, it will use that absolute path to create the unix socket. If instead any other value is passed in (e.g. the default localhost), the backend will create the socket at a new temporary path.

Benefits

Performance

@mrocklin tested this:

I just benchmarked this on
a macbook as a consideration for Frisky and got a 1.5x speedup (15 GB/s ->
25 GB/s) and 3x latency reduction (18us -> 6us) compared to TCP.  Maybe
still not worth doing there though just due to code complexity and I've yet
to find a workload that's bound by network on localhost there.  Still
pretty cool though.

Security

UDS support is desireable for usecases where:

Multi-processing is preferable to multi-threading (so no inproc)
ucx is not available (e.g. not on Linux)
We care about other users (e.g. on a shared VM) not being able to access our data and processes (so no tcp).
- within a single node, enabling TLS is not always feasible (in dask-labextension for instance), or desired (more overhead).

Dashboard listening on UDS

When the dashboard_url is set to an absolute path, listen on UDS instead of TCP. This can help prevent new CVE's of the form GHSA-c336-7962-wfj2

Todo

Tests added / passed
Passes pixi run lint
Make unix socket directory configurable
Fold tests into general test_comms.py?
Use unix:// instead of uds://?

dometto · 2026-07-27T13:09:19Z

@jacobtomlinson could you let me know if you would be interested in merging this PR (after feedback/refactoring of course)? Thanks!

jacobtomlinson

This generally seems like a positive change. I agree that I like unix:// more. I would be apprehensive to make this the default without broader community testing, but I agree that it's a better default in the long run.

Could I ask you to make sure CI is happy (unrelated failures are fine) and add docs?

github-actions · 2026-07-27T13:36:33Z

Unit Test Results

See test report for an extended history of previous test failures. This is useful for diagnosing flaky tests.

40 files ± 0 40 suites ±0 14h 52m 43s ⏱️ + 5m 23s
4 178 tests + 19 3 998 ✅ + 20 178 💤 ± 0 2 ❌ - 1
81 320 runs +382 76 982 ✅ +285 4 336 💤 +99 2 ❌ - 2

For more details on these failures, see this check.

Results for commit 588d58f. ± Comparison against base commit 40fcd99.

♻️ This comment has been updated with latest results.

Make get_uds_path() respect XDG_RUNTIME_DIR and dask.config['temporary-directory']

… filename

…in service definition

dometto · 2026-07-29T16:20:26Z

            retries_left = 3

            if os.path.isabs(http_address["address"]):  # unix socket
+                try: # remove any old sockets


Does this make sense or should we do a check on whether the server is bound to unix socket again in stop, and remove there?

dometto · 2026-07-29T16:31:44Z

I added some docs here and here.

It's pretty short, but the existing documentation on protocols is also pretty basic.

For the dashboard_address which now also supports UDS I couldn't find any docs apart from the function description for LocalCluster().

Let me know if there's anywhere else where I should add any more extensive documentation!

dometto · 2026-07-29T16:32:16Z

The remaining linting errors are pre-existing.

This is easier than removing on stop. It technically introduces a race condition, but so would removing on stop, I think.

On Linux we need to unset XDG_RUNTIME_DIR when we're not testing it.

dometto · 2026-07-30T13:02:02Z

    Set kernel-level TCP timeout on the stream.
    """
-    if comm.closed():
+    if comm.closed() or (not WINDOWS and comm.socket.family is socket.AF_UNIX):


I don't like that we have to check for unix socket stuff in tcp.py -- that should all belong to the subclass.

To refactor this we could change set_tcp_timeout to a function in the TCP class, create a new UDS subclass of TCP in uds.py, and override set_tcp_timeout there.

An example of this approach is here (see also my other comment): dometto@5e60065

Let me know if you'd like me to implement that.

dometto · 2026-07-30T13:03:22Z

        raise CommClosedError()

-    return unparse_host_port(*comm.socket.getsockname()[:2])
+    if not WINDOWS and comm.socket.family is socket.AF_UNIX:


I don't like that we have to check for unix socket stuff in tcp.py -- that should all belong to the subclass.

Refactoring this is slightly more involved than for set_tcp_tmeout, as this function is called from inside _handle_stream.

To work around this, we would have to implement _handle_stream entirely in UDSListener, instead of making it subclass TCPListener. That is perhaps cleaner anyway.

An example of this approach is again here: dometto@5e60065

Let me know if you'd like me to implement that.

Basically dometto@5e60065 removes all subclassing from uds.py except for the UDS Comm, which subclasses TCP -- in order not to have to duplicate the code for reading and writing streams.

To make the UDS code entirely independent from TCP another option would be to create a new mixin class for IOStream reading and writing, and include that in both UDS and TCP. Not sure if that's worth it.

dometto · 2026-07-31T07:15:03Z

@jacobtomlinson I think the remaining test failures are probably unrelated.

dometto added 2 commits July 23, 2026 15:59

Add UDS support

723ad80

Also run dashboard on unix domain socket

791466a

dometto requested a review from fjetter as a code owner July 27, 2026 13:07

dometto changed the title ~~Unix socket~~ Unix socket support Jul 27, 2026

jacobtomlinson reviewed Jul 27, 2026

View reviewed changes

dometto commented Jul 28, 2026

View reviewed changes

Comment thread distributed/node.py Outdated

dometto added 16 commits July 28, 2026 09:39

Pass linter

455252d

Move integration test to test_comms.py

819c378

uds:// -> unix://

14b6670

Add some docs

5bb2eea

Skip tests and don't register UDS backend on Windows

ebdd90d

Don't check for unix socket type on Windows

de2a7ec

Move get_uds_path() to distributed.utils

959bd1f

Make get_uds_path() respect XDG_RUNTIME_DIR and dask.config['temporary-directory']

Dashboard: allow passing in 'unix://' as dashboard address for random…

5c8451b

… filename

Set dashboard unix socket permissions to 600

bd52b6e

Fix get_uds_path

34f1ba2

Add unix to --protocol help

0db0f6a

fix uds path

805dc71

Remove unnecessarily added comments

c2ce79b

Add test and support for starting scheduler on a random unix socket

66fb708

refactor format_dashboard_link logic: allow setting unix socket path …

d875699

…in service definition

Add test for get_uds_path()

e5fc50d

dometto force-pushed the unix_socket branch from 0b09a58 to e5fc50d Compare July 29, 2026 15:53

Restore wrongly deleted line

d2e191a

dometto commented Jul 29, 2026

View reviewed changes

dometto requested a review from jacobtomlinson July 30, 2026 10:42

dometto added 4 commits July 30, 2026 14:51

Remove dashboard socket before binding.

42a2561

This is easier than removing on stop. It technically introduces a race condition, but so would removing on stop, I think.

Fix test_get_uds_path on ubuntu.

0bf2f6e

On Linux we need to unset XDG_RUNTIME_DIR when we're not testing it.

Fix resolver tests

a5431bc

Fix get_uds_path method signature

558101d

dometto force-pushed the unix_socket branch from e4830a7 to 4796a46 Compare July 30, 2026 12:51

Fix UDS backend: ensure we return a filesystem path on host lookup

04a14d6

dometto force-pushed the unix_socket branch from 4796a46 to 04a14d6 Compare July 30, 2026 12:58

dometto commented Jul 30, 2026

View reviewed changes

dometto force-pushed the unix_socket branch 2 times, most recently from 8b9f840 to 0ca84de Compare July 30, 2026 14:10

Skip all UDS tests on Windows

588d58f

dometto force-pushed the unix_socket branch from 0ca84de to 588d58f Compare July 30, 2026 18:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Unix socket support - #9337

Unix socket support#9337
dometto wants to merge 25 commits into
dask:mainfrom
dometto:unix_socket

dometto commented Jul 27, 2026 •

edited

Loading

Uh oh!

dometto commented Jul 27, 2026

Uh oh!

jacobtomlinson left a comment

Uh oh!

github-actions Bot commented Jul 27, 2026 •

edited

Loading

Uh oh!

Uh oh!

dometto Jul 29, 2026

Uh oh!

dometto commented Jul 29, 2026 •

edited

Loading

Uh oh!

dometto commented Jul 29, 2026

Uh oh!

dometto Jul 30, 2026 •

edited

Loading

Uh oh!

dometto Jul 30, 2026 •

edited

Loading

Uh oh!

dometto Jul 31, 2026

Uh oh!

dometto commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

dometto commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New UDS backend

Benefits

Performance

Security

Dashboard listening on UDS

Todo

Uh oh!

dometto commented Jul 27, 2026

Uh oh!

jacobtomlinson left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Unit Test Results

Uh oh!

Uh oh!

dometto Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

dometto commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dometto commented Jul 29, 2026

Uh oh!

dometto Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dometto Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dometto Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

dometto commented Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dometto commented Jul 27, 2026 •

edited

Loading

github-actions Bot commented Jul 27, 2026 •

edited

Loading

dometto commented Jul 29, 2026 •

edited

Loading

dometto Jul 30, 2026 •

edited

Loading

dometto Jul 30, 2026 •

edited

Loading