[Feature] Supporting dstack-inside-dstack for runs

[peterschmidt85]

Problem

Runs may need to call dstack themselves, for example to inspect runs, submit
another run, or attach to it. This currently requires the dstack server to
accept direct connections from the run.

UX

Add dstack: true to a run configuration.

type: task
image: dstackai/dstack
dstack: true
env:
  - DSTACK_TOKEN
commands:
  - dstack ps

With dstack: true, dstack exposes the server managing the run through
/run/dstack/server.sock inside each job. It sets DSTACK_SERVER_URL to the
corresponding http+unix URL and DSTACK_PROJECT to the current project,
allowing the CLI and API clients to use that connection.

Authentication remains explicit. dstack does not inject DSTACK_TOKEN; it can
be passed through the existing environment or secret mechanisms.

The CLI also accepts DSTACK_SERVER_URL, DSTACK_PROJECT, and DSTACK_TOKEN
as a complete configuration without requiring ~/.dstack/config.yml.

dstack cannot be combined with inactivity_duration on dev environments,
since the persistent server connection counts as activity.

Implementation

Before starting commands, the server opens an SSH connection to each job and
creates a remote Unix-socket forward. Connections to
/run/dstack/server.sock inside the job are forwarded to the HTTP port of the
server managing the run. DSTACK_SERVER_URL points clients to this socket.

This uses the existing job SSH path, so the server does not need a public URL
or gateway. Health checks use the same socket path end to end; if the SSH
connection or socket becomes stale, the server closes and recreates the
tunnel. The tunnel is removed when the job terminates.

With multiple server replicas, a replica probes the socket and adopts an
existing healthy one instead of overwriting it, so jobs keep a single stable
server owner.

[jvstme]

So if there are multiple dstack server replicas, each replica will overwrite /run/dstack/server.sock once (at JobServerConnectionsPool.ensure), and then the socket will point to the replica that overwrote it last?

I anticipate a few issues with that design:

1. Disproportionate distribution of requests among dstack server replicas. For example, the server replica that starts last (e.g., as a result of scaling or rolling update) will gradually overtake /run/dstack/server.sock in all existing jobs, and all requests from all jobs will be handled exclusively by that single replica.
2. Worse availability.
   1. If the server replica behind /run/dstack/server.sock stops or fails, the job will be unable to request the server for some period of time (potentially longer than 30s if the job was locked) until another server replica detects the missing or unhealthy /run/dstack/server.sock and takes over.
   2. The server will also be unavailable for jobs during the overwrite — requests issued between rm -f {remote_socket} and self._tunnel.aopen() will fail.

Issue 2ii can be fixed with the existing design by not overwriting the socket if it already exists and is healthy.

Issues 1 and 2i would require revisiting the design. One alternative design would be to have each server replica create its own Unix socket in the job container (similar to gateways), and have a load balancer in front of the sockets. For example, the runner could act as a load balancer.

[peterschmidt85]

Good catch — addressed in 42c8f6e. Instead of unconditionally overwriting /run/dstack/server.sock on every open, a replica now probes it first and only (re)creates the reverse forward when it's missing or unreachable; otherwise it adopts the existing one. That keeps a single stable owner, so replicas no longer steal the socket from each other (issue 1) and there's no overwrite gap (issue 2ii). If the owner replica dies, another takes over on its next health check (issue 2i) — bounded, not instant.

Verified with two replicas on a shared Postgres and a real SSH fleet, with a job probing its socket once a second:

• Unfixed: start replica B, kill B's server → probe breaks (200 → FAIL), because B had stolen the socket.
• Fixed: same steps → probe holds at 200 (B adopted; replica A stays owner).
• Fixed failover: kill the owning replica entirely → the other takes over in ~33s (bounded by the 30s health-check throttle + tick).

This deliberately keeps the single-socket design and doesn't spread a single job's load across replicas or make failover instant — the per-replica-socket + load-balancer approach you sketched would, at the cost of a runner-side change. Happy to file that as a follow-up if we want those properties.

[jvstme]

That keeps a single stable owner, so replicas no longer steal the socket from each other (issue 1).

To be precise, issue 1 is about disproportionate distribution of requests, and I think it is still relevant. Depending on server replica startup timing and job startup timing, a single server replica can still occupy /run/dstack/server.sock in all or most jobs.

I don't mind filing issues 1 and 2i as a follow-up.

[peterschmidt85]

You're right, and thanks for the precision — issue 1 (even request distribution) is only partially addressed. Adopt stops the churn and gives a stable owner, but since the owner is whichever replica opens the socket first, startup timing can still concentrate most or all jobs on a single replica. So balanced distribution (issue 1) and instant failover (2i) remain open; the per-replica-socket + load-balancer design is the way to get them.

We'll create follow-up issues for 1 and 2i.