[Feature] Supporting dstack-inside-dstack for runs

mkdocs/docs/concepts/dev-environments.md

@@ -642,6 +642,28 @@ The `schedule` property can be combined with `max_duration` or `utilization_poli
 By default, `dstack` uses on-demand instances. However, you can change that
 via the [`spot_policy`](../reference/dstack.yml/dev-environment.md#spot_policy) property. It accepts `spot`, `on-demand`, and `auto`.
 
+### `dstack` inside `dstack`
+
+Set `dstack` to `true` when a dev environment needs to use the dstack CLI. dstack configures the
+server and current project automatically. To run authenticated commands, pass `DSTACK_TOKEN`
+explicitly.
+
+<div editor-title=".dstack.yml">
+
+```yaml
+type: dev-environment
+image: dstackai/dstack
+dstack: true
+env:
+  - DSTACK_TOKEN
+init:
+  - dstack ps
+```
+
+</div>
+
+> Besides inspecting runs, you can submit new runs with `dstack apply` and attach to them with `dstack attach`.
+
 --8<-- "docs/concepts/snippets/manage-fleets.ext"
 
 !!! info "Reference"

mkdocs/docs/concepts/services.md

@@ -1048,6 +1048,29 @@ resources:
   gpu: H100:1
 ```
 
+### `dstack` inside `dstack`
+
+Set `dstack` to `true` when a service needs to use the dstack CLI. dstack configures the server and
+current project automatically. To run authenticated commands, pass `DSTACK_TOKEN` explicitly.
+
+<div editor-title=".dstack.yml">
+
+```yaml
+type: service
+image: dstackai/dstack
+dstack: true
+port: 8000
+env:
+  - DSTACK_TOKEN
+commands:
+  - dstack ps
+  - python -m http.server 8000
+```
+
+</div>
+
+> Besides inspecting runs, you can submit new runs with `dstack apply` and attach to them with `dstack attach`.
+
 ### Environment variables
 
 <div editor-title=".dstack.yml">

mkdocs/docs/concepts/tasks.md

@@ -864,6 +864,27 @@ schedule:
 By default, `dstack` uses on-demand instances. However, you can change that
 via the [`spot_policy`](../reference/dstack.yml/task.md#spot_policy) property. It accepts `spot`, `on-demand`, and `auto`.
 
+### `dstack` inside `dstack`
+
+Set `dstack` to `true` when a task needs to use the dstack CLI. dstack configures the server and
+current project automatically. To run authenticated commands, pass `DSTACK_TOKEN` explicitly.
+
+<div editor-title=".dstack.yml">
+
+```yaml
+type: task
+image: dstackai/dstack
+dstack: true
+env:
+  - DSTACK_TOKEN
+commands:
+  - dstack ps
+```
+
+</div>
+
+> Besides inspecting runs, you can submit new runs with `dstack apply` and attach to them with `dstack attach`.
+
 --8<-- "docs/concepts/snippets/manage-fleets.ext"
 
 !!! info "Reference"

mkdocs/docs/guides/protips.md

@@ -188,6 +188,26 @@ Set `docker` to `true` to enable the `docker` CLI in your dev environment, e.g.,
                 optional: true
         ```
 
+## dstack in dstack
+
+Set `dstack` to `true` when a run needs to use the dstack CLI. dstack configures the server and current project automatically. To run authenticated commands, pass `DSTACK_TOKEN` explicitly.
+
+<div editor-title=".dstack.yml">
+
+```yaml
+type: task
+image: dstackai/dstack
+dstack: true
+env:
+  - DSTACK_TOKEN
+commands:
+  - dstack ps
+```
+
+</div>
+
+> Besides inspecting runs, you can submit new runs with `dstack apply` and attach to them with `dstack attach`.
+
 ## Fleets
 
 ### Creation policy

mkdocs/docs/reference/env.md

@@ -124,6 +124,7 @@ For more details on the options below, refer to the [server deployment](../guide
 - `DSTACK_DEFAULT_SERVICE_CLIENT_MAX_BODY_SIZE`{ #DSTACK_DEFAULT_SERVICE_CLIENT_MAX_BODY_SIZE } – Request body size limit for services running with a gateway, in bytes. Defaults to 64 MiB.
 - `DSTACK_SERVICE_CLIENT_TIMEOUT`{ #DSTACK_SERVICE_CLIENT_TIMEOUT } – Timeout in seconds for HTTP requests sent from the in-server proxy and gateways to service replicas. Defaults to 60.
 - `DSTACK_FORBID_SERVICES_WITHOUT_GATEWAY`{ #DSTACK_FORBID_SERVICES_WITHOUT_GATEWAY } – Forbids registering new services without a gateway if set to any value.
+- `DSTACK_FORBID_DSTACK_IN_RUNS`{ #DSTACK_FORBID_DSTACK_IN_RUNS } – Forbids submitting runs with `dstack: true` (dstack server access inside runs) if set to any value.
 - `DSTACK_SERVER_CODE_UPLOAD_LIMIT`{ #DSTACK_SERVER_CODE_UPLOAD_LIMIT } - The repo size limit when uploading diffs or local repos, in bytes. Set to `0` to disable size limits. Defaults to `2MiB`.
 - `DSTACK_SERVER_S3_BUCKET`{ #DSTACK_SERVER_S3_BUCKET } - The bucket that repo diffs will be uploaded to if set. If unset, diffs are uploaded to the database.
 - `DSTACK_SERVER_S3_BUCKET_REGION`{ #DSTACK_SERVER_S3_BUCKET_REGION } - The region of the S3 Bucket.
@@ -168,6 +169,17 @@ slows down processing and may cause CPU spikes due to frequent SSH-connection es
 
 The following environment variables are supported by the CLI.
 
+- `DSTACK_TOKEN`{ #DSTACK_TOKEN } – The user token used by the CLI. Set `DSTACK_TOKEN`,
+  `DSTACK_SERVER_URL`, and `DSTACK_PROJECT` together to use the CLI without a project in
+  `~/.dstack/config.yml`, or to override the configured server, project, and user.
+
+  ```shell
+  DSTACK_SERVER_URL=https://server.example.com \
+  DSTACK_PROJECT=main \
+  DSTACK_TOKEN=your-token \
+  dstack ps
+  ```
+
 - `DSTACK_CLI_LOG_LEVEL`{ #DSTACK_CLI_LOG_LEVEL } – Sets the logging level for CLI output to stdout. Defaults to `INFO`.
 
 Example:

pyproject.toml

@@ -13,6 +13,7 @@ classifiers = [
 dependencies = [
     "pyyaml",
     "requests",
+    "requests-unixsocket>=0.4.1",
     "typing-extensions>=4.0.0",
     "cryptography",
     "packaging",
@@ -187,7 +188,6 @@ server = [
     "aiorwlock",
     "aiocache",
     "httpx>=0.28.0",
-    "requests-unixsocket>=0.4.1",
     "jinja2",
     "watchfiles",
     "sqlalchemy[asyncio]>=2.0.0",

src/dstack/_internal/core/compatibility/runs.py

@@ -6,7 +6,11 @@
     IncludeExcludeDictType,
     IncludeExcludeSetType,
 )
-from dstack._internal.core.models.configurations import ServiceConfiguration
+from dstack._internal.core.models.configurations import (
+    DevEnvironmentConfiguration,
+    ServiceConfiguration,
+    TaskConfiguration,
+)
 from dstack._internal.core.models.routers import SGLangServiceRouterConfig
 from dstack._internal.core.models.runs import (
     DEFAULT_PROBE_UNTIL_READY,
@@ -89,6 +93,15 @@ def get_run_spec_excludes(run_spec: RunSpec) -> IncludeExcludeDictType:
     if run_spec.configuration.backend_options is None:
         configuration_excludes["backend_options"] = True
 
+    if (
+        isinstance(
+            run_spec.configuration,
+            (DevEnvironmentConfiguration, TaskConfiguration, ServiceConfiguration),
+        )
+        and not run_spec.configuration.dstack
+    ):
+        configuration_excludes["dstack"] = True
+
     if isinstance(run_spec.configuration, ServiceConfiguration):
         if run_spec.configuration.probes:
             probe_excludes: IncludeExcludeDictType = {}

src/dstack/_internal/core/consts.py

@@ -1,8 +1,16 @@
+from urllib.parse import quote
+
 # shim (runs on the host) HTTP API port
 DSTACK_SHIM_HTTP_PORT = 10998
 # runner (runs inside a container) HTTP API port
 DSTACK_RUNNER_HTTP_PORT = 10999
 # ssh server (runs alongside the runner inside a container) listen port
 DSTACK_RUNNER_SSH_PORT = 10022
+# Private socket created inside jobs that request access to the dstack server.
+DSTACK_RUN_SERVER_SOCKET_PATH = "/run/dstack/server.sock"
+DSTACK_RUN_SERVER_URL = f"http+unix://{quote(DSTACK_RUN_SERVER_SOCKET_PATH, safe='')}"
+DSTACK_PROJECT_ENV = "DSTACK_PROJECT"
+DSTACK_SERVER_URL_ENV = "DSTACK_SERVER_URL"
+DSTACK_TOKEN_ENV = "DSTACK_TOKEN"
 # legacy AWS, Azure, GCP, and OCI image for older GPUs
 DSTACK_OS_IMAGE_WITH_PROPRIETARY_NVIDIA_KERNEL_MODULES = "0.10"

src/dstack/_internal/core/models/configurations.py

@@ -693,6 +693,18 @@ def check_image_or_commands_present(cls, values):
         return values
 
 
+class ConfigurationWithDstackParams(CoreModel):
+    dstack: Annotated[
+        bool,
+        Field(
+            description=(
+                "Make the dstack server accessible inside the run. "
+                "No authentication credentials are provided"
+            )
+        ),
+    ] = False
+
+
 class DevEnvironmentConfigurationParams(CoreModel):
     ide: Annotated[
         Optional[Union[Literal["vscode"], Literal["cursor"], Literal["windsurf"], Literal["zed"]]],
@@ -762,6 +774,7 @@ class DevEnvironmentConfiguration(
     ProfileParams,
     BaseRunConfiguration,
     ConfigurationWithPortsParams,
+    ConfigurationWithDstackParams,
     DevEnvironmentConfigurationParams,
     generate_dual_core_model(DevEnvironmentConfigurationConfig),
 ):
@@ -773,6 +786,13 @@ def validate_entrypoint(cls, v: Optional[str]) -> Optional[str]:
             raise ValueError("entrypoint is not supported for dev-environment")
         return v
 
+    @root_validator
+    def validate_dstack_and_inactivity_duration(cls, values):
+        if values.get("dstack") and values.get("inactivity_duration") is not None:
+            # The persistent server connection counts as activity, so inactivity is never detected
+            raise ValueError("`dstack` is not supported together with `inactivity_duration`")
+        return values
+
 
 class TaskConfigurationParams(CoreModel):
     nodes: Annotated[int, Field(description="Number of nodes", ge=1)] = 1
@@ -793,6 +813,7 @@ class TaskConfiguration(
     BaseRunConfiguration,
     ConfigurationWithCommandsParams,
     ConfigurationWithPortsParams,
+    ConfigurationWithDstackParams,
     TaskConfigurationParams,
     generate_dual_core_model(TaskConfigurationConfig),
 ):
@@ -1338,6 +1359,7 @@ class ServiceConfiguration(
     ProfileParams,
     BaseRunConfiguration,
     ConfigurationWithCommandsParams,
+    ConfigurationWithDstackParams,
     ServiceConfigurationParams,
     generate_dual_core_model(ServiceConfigurationConfig),
 ):

src/dstack/_internal/core/services/api_client.py

@@ -1,15 +1,40 @@
+import os
 from typing import Optional, Tuple
 
 import dstack._internal.core.services.configs as configs
+from dstack._internal.core.consts import (
+    DSTACK_PROJECT_ENV,
+    DSTACK_SERVER_URL_ENV,
+    DSTACK_TOKEN_ENV,
+)
 from dstack._internal.core.errors import ConfigurationError
 from dstack.api.server import APIClient
 
 
 def get_api_client(project_name: Optional[str] = None) -> Tuple[APIClient, str]:
+    env_project_name = project_name or os.getenv(DSTACK_PROJECT_ENV)
+    server_url = os.getenv(DSTACK_SERVER_URL_ENV)
+    token = os.getenv(DSTACK_TOKEN_ENV)
+    if server_url is not None and token is not None:
+        if env_project_name is None:
+            raise ConfigurationError(
+                f"{DSTACK_SERVER_URL_ENV} and {DSTACK_TOKEN_ENV} are set,"
+                f" but the project is not specified."
+                f" Set {DSTACK_PROJECT_ENV} or use --project"
+            )
+        return APIClient(server_url, token), env_project_name
+
     config = configs.ConfigManager()
     project = config.get_project_config(project_name)
     if project is None:
+        if server_url is not None:
+            raise ConfigurationError(
+                f"{DSTACK_SERVER_URL_ENV} is set, but {DSTACK_TOKEN_ENV} is not set"
+            )
         if project_name is not None:
             raise ConfigurationError(f"Project {project_name} is not configured")
         raise ConfigurationError("No default project, specify project name")
+    if token is not None:
+        # DSTACK_TOKEN overrides the configured token
+        return APIClient(project.url, token), project.name
     return APIClient(project.url, project.token), project.name

src/dstack/_internal/server/app.py

@@ -53,6 +53,7 @@
 )
 from dstack._internal.server.services.config import ServerConfigManager
 from dstack._internal.server.services.gateways import gateway_connections_pool
+from dstack._internal.server.services.jobs.server_connection import job_server_connections_pool
 from dstack._internal.server.services.locking import advisory_lock_ctx
 from dstack._internal.server.services.projects import get_or_create_default_project
 from dstack._internal.server.services.proxy.deps import ServerProxyDependencyInjector
@@ -213,6 +214,7 @@ async def lifespan(app: FastAPI):
     if pipeline_manager is not None:
         await pipeline_manager.drain()
     await gateway_connections_pool.remove_all()
+    await job_server_connections_pool.remove_all()
     service_conn_pool = await get_injector_from_app(app).get_service_connection_pool()
     await service_conn_pool.remove_all()
     if settings.SERVER_SSH_POOL_ENABLED: