diff --git a/enterprise/custom-sandbox-image.mdx b/enterprise/custom-sandbox-image.mdx index 0c74aca5c..271f45a56 100644 --- a/enterprise/custom-sandbox-image.mdx +++ b/enterprise/custom-sandbox-image.mdx @@ -1,6 +1,6 @@ --- title: Custom Sandbox Images -description: Preload repos, dependencies, and tooling into a custom sandbox image to make your agents faster and more reliable. +description: Preload repos, dependencies, and tooling into custom sandbox images, and run multiple images side by side with warm runtime pools. icon: box --- @@ -8,12 +8,24 @@ Custom sandbox images let you prebake the repository, dependencies, compiled out your agents need. Instead of spending minutes provisioning a workspace on every run, your agents start on the actual task immediately. +This page covers two levels of customization: + +1. **[A single custom image](#configure-a-single-custom-image-admin-console)** that replaces the default + sandbox image for the whole installation. Configured in the Replicated Admin Console; no cluster access needed. +2. **[Multiple custom images](#run-multiple-custom-images-with-warm-runtime-pools)** running side by side, + each with its own warm pool, selectable per user. Configured through the Runtime API; requires `kubectl` access. + ## Why Use a Custom Image Custom images eliminate cold-start setup work (clone, install, transpile, and bootstrap) so agents spend their time on the actual task. They also reduce setup variance and lower sandbox memory requirements by keeping only what the agent needs. +With **multiple** custom images, different teams get different environments: a PHP image with Composer and +MySQL client for the web team, a JDK and Maven image for the Java services team, a data science image with +pinned Python packages for the analytics team. Each image is kept ready in its own warm pool so conversations +start in seconds regardless of which environment they use. + ## Build Your Own Custom Image The [OpenHands agent-server sandbox guide](https://docs.openhands.dev/sdk/guides/agent-server/docker-sandbox) @@ -26,7 +38,7 @@ Replicated VM deployment. 2. Keep the normal OpenHands entrypoint intact: extend the image, do not replace the entrypoint. 3. Add your repo, docs, tools, and verification wrappers. 4. Pre-run the expensive setup you do not want to repeat at task time. -5. Publish the image to a registry and point the Replicated installer at it. +5. Publish the image to a registry reachable from your OpenHands cluster. Do not override the entrypoint or replace the runtime contract of the base image. The installer @@ -36,12 +48,13 @@ Replicated VM deployment. ### Base Image ```dockerfile -FROM ghcr.io/openhands/agent-server:1.41.0-python +FROM ghcr.io/openhands/agent-server:1.46.0-python ``` -Pin a specific version tag to ensure reproducible builds. Check +This example matches OpenHands Enterprise 0.64.0. Pin a specific version tag to ensure reproducible +builds, and replace it with the tag expected by your installed release. Check [ghcr.io/openhands/agent-server](https://github.com/OpenHands/OpenHands/pkgs/container/agent-server) -for the latest available tags. +for available tags. ### Version Compatibility @@ -100,12 +113,12 @@ Good candidates for prebaking: If the repository or dependencies change frequently, include a `prepare-*` script in the image so the agent can refresh only the parts that need updating without a full rebuild. -## Configure the Replicated VM Installer +## Configure a Single Custom Image (Admin Console) Once your image is built and pushed to a registry, point the Replicated Admin Console at it. 1. Open the **Admin Console** at `https://admin.:30000`. -2. Navigate to **Config** and find the **Sandbox Image** section. +2. Navigate to **Config** and find the **Sandbox Configuration** section. 3. Set the following fields: | Field | Value | @@ -119,12 +132,351 @@ Once your image is built and pushed to a registry, point the Replicated Admin Co 4. Click **Save config** and then **Deploy** to apply the change. +This single image becomes both the default image for new conversations and the image kept ready in the +installer-managed warm pool. + This setting applies to the **sandbox / agent-server image** only (the image that runs inside each agent's isolated workspace). It does not replace the other OpenHands service images. +## Run Multiple Custom Images with Warm Runtime Pools + +To offer several sandbox images at once, configure **warm runtime pools** through the Runtime API. +Each configuration names one image and keeps a pool of pre-started sandbox pods ready for it. The +OpenHands application automatically exposes every configuration as a selectable sandbox, so users can +pick their environment without any redeployment. + +**Requirements:** + +- OpenHands Enterprise **0.64.0 or later**. +- `kubectl` access to the cluster. On a Replicated VM install, get a shell with + `sudo /var/lib/embedded-cluster/bin/openhands shell`; on a Helm install, use your normal kubeconfig. +- Custom images built and pushed as described above (all on the agent-server version your release expects). + +### How It Works + +- The installer-managed configuration remains the base configuration. Configurations saved through the + Runtime API are overlaid by name: a new name adds a pool, while an existing name overrides that + installer-managed entry. +- You manage database configurations with the admin REST endpoints + (`PUT` / `DELETE /api/admin/warm-runtime-configs/{name}`). Deleting an override reveals the + installer-managed entry again. +- A reconciler job runs **every minute** and creates or removes warm sandbox pods so each + configuration has `count` unclaimed pods ready. +- The OpenHands application polls the configuration list (cached for 60 seconds) and exposes each + configuration as a **sandbox spec**. Users choose their default in **Settings → Application → Default Sandbox**. +- When a conversation starts, the Runtime API hands it a matching warm pod in a few seconds. If no + warm pod is available, the sandbox cold-starts from the image instead (20+ seconds), and the + reconciler replenishes the pool. + +Changes take effect within about a minute, with no application restarts and no redeployments. + + + The installer-managed `v1_current` pool remains active when you add API-managed configurations. Do + not save a `v1_current` configuration unless you intentionally want to override the installer default. + + +### Step 1: Confirm the Admin Password + +The Runtime API's admin endpoints authenticate with an admin password. Replicated generates a durable +password, stores it in the `admin-password` secret, and injects it into the runtime-api pod. The helper +script in Step 2 uses that pod environment. If the value is empty, the script reports an error before a +save or delete. + +To set or rotate the password: + +1. Open the `Admin Console` and select `Config`. +2. In `Sandbox Configuration`, set `Runtime API Admin Password`. +3. Select `Save config`, then deploy the new configuration. + +The value persists across later deploys. Changing it automatically restarts runtime-api so the new +password takes effect. For a Helm installation, populate the chart's `admin-password` Secret before +using the admin endpoints and restart runtime-api after changing it. + +### Step 2: Save the Helper Script + +The Runtime API is not exposed outside the cluster by default. Download the maintained +[`warm-runtime-configs.sh`](https://github.com/OpenHands/runtime-api/blob/main/scripts/warm-runtime-configs.sh) +helper, which runs each API call inside the runtime-api pod with `kubectl exec`: + +```bash +curl -fsSLo warm-runtime-configs.sh \ + https://raw.githubusercontent.com/OpenHands/runtime-api/main/scripts/warm-runtime-configs.sh +chmod +x warm-runtime-configs.sh +./warm-runtime-configs.sh list +``` + + + The helper uses `DEFAULT_API_KEY` and `ADMIN_PASSWORD` from the runtime-api pod without printing or + copying either value. Listing authenticates with the regular API key. Saving and deleting use the + admin password via a challenge-response login that returns a 24-hour JWT. + + +List responses identify each configuration's `source`. When `v1_current` is not overridden, it appears +with `"source": "file"`. + +### Step 3: Start From the Installer's Default Configuration + +Do not write configurations from scratch. The environment in a warm runtime configuration is what its +sandbox pods actually boot with; the default configuration contains install-specific values (webhook +callback URL, CA bundles, workspace paths) that sandboxes need to function. Export the default from the +installer-managed ConfigMap and use it as your template: + +```bash +kubectl -n openhands get configmap warm-runtimes-config \ + -o jsonpath='{.data.warm-runtimes\.json}' \ + | jq '.configs[] | select(.name == "v1_current") | del(.name)' > default-config.json +``` + +(If the ConfigMap has a different name in your install, find it with +`kubectl -n openhands get configmap | grep warm-runtimes`.) + +The installer-managed `v1_current` entry remains live and follows Admin Console changes. Derive each +custom image configuration from the exported template, changing only the image and pool size: + +```bash +jq '.image = "ghcr.io/your-org/openhands-php:8.4-v1" | .count = 1' \ + default-config.json > php-web.json +./warm-runtime-configs.sh save php-web php-web.json + +./warm-runtime-configs.sh list +``` + +### Configuration Format + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `image` | string | Yes | Full image reference (e.g. `ghcr.io/your-org/openhands-php:8.4-v1`) | +| `working_dir` | string | Yes | Working directory inside the sandbox (copy from the default) | +| `command` | array | Yes | Agent-server start command (copy from the default) | +| `environment` | object | Yes | Environment variables the sandbox boots with (copy from the default) | +| `count` | integer | No | Warm pods to keep ready for this image. When omitted, uses the installer-wide **Warm Runtime Count** (default: `1` on Replicated installs) | +| `run_as_user` | integer | No | Copy from the installer default (currently `10001`) so warm pods match application start requests | +| `run_as_group` | integer | No | Copy from the installer default (currently `10001`) so warm pods match application start requests | +| `fs_group` | integer | No | Copy from the installer default (currently `10001`) so warm pods match application start requests | + +The configuration name comes from the URL path (the `save ` argument), not the body. Saving +creates or replaces a database entry. If an installer-managed entry has the same name, the database +entry overrides it. List responses also include a read-only `source` field: `file` for installer-managed +entries and `db` for API-managed entries and overrides. Do not add `source` to a saved configuration. + +The application uses the image reference as the sandbox spec ID. Give every selectable configuration a +distinct image reference; configurations that share an image reference cannot be selected independently, +even if their commands or environments differ. + + + Set `count` explicitly. Every warm pod reserves the full sandbox resource envelope (including 10Gi of + ephemeral storage by default) whether or not it is in use, so the sum of all pool sizes must fit your + node capacity. Pools that exceed capacity show up as `Pending` pods. Start with `count: 1` per image + and grow the pools that see real traffic. + + +### Step 4: Verify the Warm Pools + +The reconciler runs every minute. Watch it create the pods: + +```bash +# Warm (unclaimed) sandboxes: runtime deployments with no session_id label yet +kubectl -n openhands get deploy -l 'runtime_id,!session_id' \ + -o custom-columns='NAME:.metadata.name,READY:.status.readyReplicas,IMAGE:.spec.template.spec.containers[0].image' +``` + +You should see one `runtime-` deployment per warm pod, with your configured images. To see +the reconciler's own view (per-pool counts, pull failures, culling decisions), read the latest +reconciler job log: + +```bash +JOB=$(kubectl -n openhands get jobs --sort-by=.metadata.creationTimestamp -o name \ + | grep warm-runtimes | tail -1) +kubectl -n openhands logs "$JOB" +``` + +If a pod is stuck pulling your image, `kubectl -n openhands describe pod ` shows the pull +error. For private registries, either fill in the **Registry Server / Username / Password** fields in +the Admin Console's **Sandbox Configuration** section (they render an image pull secret that runtime +pods use), or add your own secret name to the runtime-api `RUNTIME_IMAGE_PULL_SECRETS` setting. + +### Step 5: Pick an Image and Start a Conversation + +Within a minute of saving configurations (the application caches the list for 60 seconds): + +- **Per user**: each user opens **Settings → Application** and picks an image in the **Default Sandbox** + dropdown (entries are the image references). Before a user picks an image, the application uses the + configuration named `v1_current`, or the first configuration if no `v1_current` exists. All of the + user's new conversations use their selected image. +- **Per conversation (API)**: start a sandbox for a specific image, then attach a conversation to it: + + ```bash + # 1. Start a sandbox from a specific spec (the spec id is the image reference) + curl -X POST "https://app./api/v1/sandboxes?sandbox_spec_id=ghcr.io/your-org/openhands-php:8.4-v1" \ + -H "Authorization: Bearer $API_KEY" + # 2. Create the conversation on that sandbox, using "id" from the response + curl -X POST "https://app./api/v1/app-conversations" \ + -H "Authorization: Bearer $API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"sandbox_id": ""}' + ``` + +To confirm a conversation claimed a warm pod rather than cold-starting, note that its sandbox was +ready in a few seconds, or check the cluster: the claimed runtime deployment now carries a +`session_id` label, and the reconciler creates a fresh warm pod to replace it within a minute. + +### How Warm Pods Are Claimed + +A conversation claims a warm pod only when the pod **exactly matches** the requested image, command, +working directory, environment (ignoring a fixed set of session-specific variables), and +`run_as_user` / `run_as_group` / `fs_group`. Because the application requests exactly what the +selected configuration declares, conversations started through OpenHands match automatically. + +Cold starts still happen when: + +- All warm pods for the image are already claimed (`count` too low for the traffic). +- The configuration changed in the last minute, so the old pods no longer match and replacements are + still starting. +- Warm pods cannot become ready (image pull failures, insufficient node resources). + +Cold-started conversations run the same image and work normally; they just take 20+ seconds to begin. + +### Updating and Deleting Configurations + +Update by saving the same name again. To roll out a new image version: + +```bash +jq '.image = "ghcr.io/your-org/openhands-php:8.4-v2"' php-web.json > php-web-v2.json +./warm-runtime-configs.sh save php-web php-web-v2.json +``` + +Within a minute the reconciler stops the old pods and starts pods on the new image. Delete a +database-only configuration to remove its pool: + +```bash +./warm-runtime-configs.sh delete php-web +``` + +If the deleted name overrides an installer-managed entry, the underlying entry becomes effective again +instead of disappearing. Confirm the result with `./warm-runtime-configs.sh list`; its `source` changes +from `db` to `file`. + +Keep superseded image tags available in your registry while conversations that used them can still +resume: a paused conversation resumes on its **original** image. Delete old tags only after the +conversations that used them are gone (by default, stopped sandboxes are cleaned up after 10 days). + +### After Upgrading OpenHands Enterprise + + + API-managed configurations are **frozen snapshots**; upgrades do not touch them. The installer-managed + `v1_current` entry updates automatically unless a database entry with that name overrides it. Each + release expects a specific agent-server version and may add or change sandbox environment variables. + After every OpenHands Enterprise upgrade: + + 1. Rebuild your custom images on the release's new agent-server base version. + 2. Re-export the default template (Step 3) from the refreshed ConfigMap. + 3. Re-derive and save each API-managed custom configuration from the new template. + 4. If you intentionally override `v1_current`, refresh or delete that override so the new + installer-managed entry can take effect. + + Skipping this leaves configurations pointing at the previous agent-server version, and new + conversations fail with a version mismatch error until the configurations are updated. + + +### Return an Entry to Installer Management + +Delete a same-named database override to restore the installer-managed entry on the next reconciler +cycle. For example, if `v1_current` was intentionally overridden: + +```bash +./warm-runtime-configs.sh delete v1_current +./warm-runtime-configs.sh list # v1_current now reports "source": "file" +``` + +Other API-managed configurations continue running. Delete them individually when you no longer want +their pools or images in the application's selector. + +### Troubleshooting + +| Symptom | Cause and fix | +|---|---| +| `HTTP 403: Admin functionality is disabled` | The runtime-api deployment has no admin password, or the configured value is empty. On Replicated installs, set `Runtime API Admin Password` per Step 1 and deploy. | +| `HTTP 401` on login | Wrong password, or the challenge expired. Challenges are single-use and expire after 5 minutes; the script fetches a fresh one per call. | +| `HTTP 401: ...provide a valid API key...` on list | The list endpoint authenticates with `X-API-Key`, not the admin JWT. Use the helper script. | +| Installer-managed default is missing from the list | The release does not include overlay support, or the overlay is not enabled. Upgrade OpenHands Enterprise and confirm that the list reports `source` before saving configurations. | +| Saved a config but the dropdown does not show it | The application caches the list for 60 seconds; wait a minute and reload. Also confirm with `./warm-runtime-configs.sh list`. | +| No warm pods appear | Read the latest reconciler job log (Step 4). Look for image pull errors or scheduling failures. | +| Warm pods `Pending` | Insufficient node resources. Every warm pod reserves the full sandbox resource envelope; lower the pool `count`s or add capacity. | +| Conversations cold-start despite warm pods | Pool exhausted or configuration recently changed; see [How Warm Pods Are Claimed](#how-warm-pods-are-claimed). | +| Sandbox fails at start with an agent-server version error | The custom image's base version does not match the release. Rebuild on the expected agent-server version (see [Base Image](#base-image)). | +| Conversations on a custom image start but never show agent output | The configuration's `environment` is missing install-specific values (webhook callback URL, CA bundles). Rebuild the configuration from the default template (Step 3). | + +### API Reference + +The endpoints below are served by the runtime-api service (in-cluster: `http://:5000`). + +**Admin authentication** (for save and delete): + +1. `GET /api/admin/challenge` returns `{challenge, salt, iterations}`. Challenges are single-use and + expire after 5 minutes. +2. Compute `PBKDF2-HMAC-SHA256(password, salt + challenge, iterations, dklen=32)` and hex-encode it. +3. `POST /api/admin/login` with `{"challenge": ..., "hash": ...}` returns `{"token": ...}`, a JWT + valid for 24 hours. +4. Send `Authorization: Bearer ` on admin requests. + +**List configurations** (regular API key, not admin): + +```http +GET /api/warm-runtime-configs +X-API-Key: {api-key} +``` + +Returns `200` with the effective configuration set: + +```json +{ + "configs": [ + {"name": "v1_current", "image": "...", "source": "file"}, + {"name": "php-web", "image": "...", "source": "db"} + ] +} +``` + +Installer-managed entries have `source: "file"`. API-managed entries have `source: "db"`; a database +entry with the same name replaces the file entry in this effective list. + +**Create or update a configuration** (admin): + +```http +PUT /api/admin/warm-runtime-configs/{name} +Authorization: Bearer {admin-jwt} +Content-Type: application/json + +{"image": "...", "working_dir": "...", "command": [...], "environment": {...}, "count": 1} +``` + +Returns `200` with the saved configuration. Creates or overwrites; the name in the URL is the identity. + +**Delete a configuration** (admin): + +```http +DELETE /api/admin/warm-runtime-configs/{name} +Authorization: Bearer {admin-jwt} +``` + +Returns `200` with a confirmation message, or `404` if no database configuration has that name. When +the deleted name also exists in the installer-managed file, that file entry becomes effective again. + ## Reference -- [OpenHands custom image example repo](https://github.com/OpenHands/openhands-custom-image): Dockerfile, benchmark scripts, and analysis tooling for the VS Code custom image example. -- [Agent-server sandbox guide](https://docs.openhands.dev/sdk/guides/agent-server/docker-sandbox): full SDK documentation on building and configuring custom sandbox images. + + + Full SDK documentation on building custom sandbox images + + + Dockerfile, benchmark scripts, and analysis tooling for the VS Code custom image example + + + How conversations, sandboxes, and their lifecycle fit together + + + Capacity planning, including headroom for warm pools + +