diff --git a/enterprise/custom-sandbox-image.mdx b/enterprise/custom-sandbox-image.mdx
index 37a4bbf9..78e962ac 100644
--- a/enterprise/custom-sandbox-image.mdx
+++ b/enterprise/custom-sandbox-image.mdx
@@ -148,15 +148,19 @@ pick their environment without any redeployment.
**Requirements:**
-- OpenHands Enterprise **0.28.0 or later**.
+- An OpenHands Enterprise release with warm runtime configuration overlays enabled.
- `kubectl` access to the cluster. On a Replicated VM install, get a shell with
`sudo /var/lib/embedded-cluster/bin/openhands shell`; on a Helm install, use your normal kubeconfig.
- Custom images built and pushed as described above (all on the agent-server version your release expects).
### How It Works
-- The Runtime API stores warm runtime configurations in its database. You manage them with the
- admin REST endpoints (`PUT` / `DELETE /api/admin/warm-runtime-configs/{name}`).
+- The installer-managed configuration remains the base configuration. Configurations saved through the
+ Runtime API are overlaid by name: a new name adds a pool, while an existing name overrides that
+ installer-managed entry.
+- You manage database configurations with the admin REST endpoints
+ (`PUT` / `DELETE /api/admin/warm-runtime-configs/{name}`). Deleting an override reveals the
+ installer-managed entry again.
- A reconciler job runs **every minute** and creates or removes warm sandbox pods so each
configuration has `count` unclaimed pods ready.
- The OpenHands application polls the configuration list (cached for 60 seconds) and exposes each
@@ -167,39 +171,30 @@ pick their environment without any redeployment.
Changes take effect within about a minute, with no application restarts and no redeployments.
-
- **The first configuration you save takes over warm pool management.** While the Runtime API database
- holds *any* configurations, the installer-managed default pool (from the Admin Console
- **Sandbox Configuration** section) is ignored entirely, and the Admin Console image settings stop
- affecting warm pools. Always re-declare the default image as one of your configurations
- (Step 3 below does this). To hand control back to the installer, delete **all** configurations
- (see [Reverting](#reverting-to-installer-managed-configuration)).
-
-
-### Step 1: Set the Admin Password
+
+ The installer-managed `v1_current` pool remains active when you add API-managed configurations. Do
+ not save a `v1_current` configuration unless you intentionally want to override the installer default.
-The Runtime API's admin endpoints authenticate with an admin password. On Replicated installs the
-`admin-password` secret exists but is **empty by default**, so set one before using the admin API:
+ If you configured multiple images on an earlier release, you may already have a `v1_current` database
+ entry that was needed to preserve the default pool. After upgrading, delete that override once to
+ return `v1_current` to installer management. Your other API-managed pools remain active.
+
-```bash
-ADMIN_PASSWORD=$(openssl rand -base64 24)
+### Step 1: Confirm the Admin Password
-kubectl -n openhands patch secret admin-password \
- -p '{"stringData":{"admin-password":"'"$ADMIN_PASSWORD"'"}}'
+The Runtime API's admin endpoints authenticate with an admin password. Replicated generates a durable
+password during installation and stores it in the `admin-password` secret. The helper script in Step 2
+reads that secret, so you do not need to copy the generated password.
-# Restart runtime-api to pick up the new password
-kubectl -n openhands rollout restart deployment -l app.kubernetes.io/name=runtime-api
-kubectl -n openhands rollout status deployment -l app.kubernetes.io/name=runtime-api
+To choose your own password:
-echo "Save this in your password manager: $ADMIN_PASSWORD"
-```
+1. Open the `Admin Console` and select `Config`.
+2. In `Sandbox Configuration`, set `Runtime API Admin Password`.
+3. Select `Save config`, then deploy the new configuration.
-
- **Re-apply the password after every Admin Console deploy.** The `admin-password` secret is rendered
- by the application chart, so any config change or upgrade deployed through the Admin Console resets
- it to empty. Your warm runtime configurations are stored in the database and survive deploys; only
- the password needs re-patching (followed by the `rollout restart` above).
-
+The value persists across later deploys. Changing it automatically restarts runtime-api so the new
+password takes effect. For a Helm installation, populate the chart's `admin-password` Secret before
+using the admin endpoints and restart runtime-api after changing it.
### Step 2: Save the Helper Script
@@ -324,6 +319,7 @@ esac
```bash
chmod +x warm-runtime-configs.sh
+./warm-runtime-configs.sh list
```
@@ -332,6 +328,10 @@ chmod +x warm-runtime-configs.sh
handles both.
+List responses identify each configuration's `source`. On an installation where `v1_current` is not
+overridden, it appears with `"source": "file"`. If no API configurations have been saved and the list
+is empty or omits `source`, upgrade OpenHands Enterprise before continuing.
+
### Step 3: Start From the Installer's Default Configuration
Do not write configurations from scratch. The environment in a warm runtime configuration is what its
@@ -348,16 +348,8 @@ kubectl -n openhands get configmap openhands-runtime-api-warm-runtimes \
(If the ConfigMap has a different name in your install, find it with
`kubectl -n openhands get configmap | grep warm-runtimes`.)
-First, re-declare the default image so its pool survives the takeover described above. Name it
-`v1_current`; the application treats the configuration with that name as the system default:
-
-```bash
-jq '.count = 1' default-config.json > v1_current.json
-./warm-runtime-configs.sh save v1_current v1_current.json
-```
-
-Then derive each custom image configuration from the same template, changing only the image and the
-pool size:
+The installer-managed `v1_current` entry remains live and follows Admin Console changes. Derive each
+custom image configuration from the exported template, changing only the image and pool size:
```bash
jq '.image = "ghcr.io/your-org/openhands-php:8.4-v1" | .count = 1' \
@@ -380,8 +372,10 @@ jq '.image = "ghcr.io/your-org/openhands-php:8.4-v1" | .count = 1' \
| `run_as_group` | integer | No | Group ID for the pod security context (default `10001`) |
| `fs_group` | integer | No | Filesystem group ID for the pod security context (default `10001`) |
-The configuration name comes from the URL path (the `save ` argument), not the body. Saving an
-existing name overwrites it.
+The configuration name comes from the URL path (the `save ` argument), not the body. Saving
+creates or replaces a database entry. If an installer-managed entry has the same name, the database
+entry overrides it. List responses also include a read-only `source` field: `file` for installer-managed
+entries and `db` for API-managed entries and overrides. Do not add `source` to a saved configuration.
Set `count` explicitly. Every warm pod reserves the full sandbox resource envelope (including 10Gi of
@@ -466,12 +460,16 @@ jq '.image = "ghcr.io/your-org/openhands-php:8.4-v2"' php-web.json > php-web-v2.
```
Within a minute the reconciler stops the old pods and starts pods on the new image. Delete a
-configuration to remove its pool:
+database-only configuration to remove its pool:
```bash
./warm-runtime-configs.sh delete php-web
```
+If the deleted name overrides an installer-managed entry, the underlying entry becomes effective again
+instead of disappearing. Confirm the result with `./warm-runtime-configs.sh list`; its `source` changes
+from `db` to `file`.
+
Keep superseded image tags available in your registry while conversations that used them can still
resume: a paused conversation resumes on its **original** image. Delete old tags only after the
conversations that used them are gone (by default, stopped sandboxes are cleaned up after 10 days).
@@ -479,38 +477,42 @@ conversations that used them are gone (by default, stopped sandboxes are cleaned
### After Upgrading OpenHands Enterprise
- Your saved configurations are **frozen snapshots**; upgrades do not touch them. Each release expects
- a specific agent-server version and may add or change sandbox environment variables, and only the
- installer-managed default template picks those up. After every OpenHands Enterprise upgrade:
+ API-managed configurations are **frozen snapshots**; upgrades do not touch them. The installer-managed
+ `v1_current` entry updates automatically unless a database entry with that name overrides it. Each
+ release expects a specific agent-server version and may add or change sandbox environment variables.
+ After every OpenHands Enterprise upgrade:
1. Rebuild your custom images on the release's new agent-server base version.
2. Re-export the default template (Step 3) from the refreshed ConfigMap.
- 3. Re-save `v1_current` and re-derive each custom configuration from the new template.
+ 3. Re-derive and save each API-managed custom configuration from the new template.
+ 4. If you intentionally override `v1_current`, refresh or delete that override so the new
+ installer-managed entry can take effect.
Skipping this leaves configurations pointing at the previous agent-server version, and new
conversations fail with a version mismatch error until the configurations are updated.
-### Reverting to Installer-Managed Configuration
+### Return an Entry to Installer Management
-Delete **all** configurations and the Runtime API falls back to the installer-managed default from
-the Admin Console on the next reconciler cycle:
+Delete a same-named database override to restore the installer-managed entry on the next reconciler
+cycle. For example, if `v1_current` was intentionally overridden:
```bash
-./warm-runtime-configs.sh delete php-web
./warm-runtime-configs.sh delete v1_current
-./warm-runtime-configs.sh list # should print []
+./warm-runtime-configs.sh list # v1_current now reports "source": "file"
```
-The application's image list then falls back to the single default image as well.
+Other API-managed configurations continue running. Delete them individually when you no longer want
+their pools or images in the application's selector.
### Troubleshooting
| Symptom | Cause and fix |
|---|---|
-| `HTTP 403: Admin functionality is disabled` | The runtime-api deployment has no admin password wired at all (non-standard install). On Replicated installs the secret exists; set it per Step 1. |
-| `HTTP 401` on login | Wrong password, or the challenge expired (challenges are single-use and expire after 5 minutes; the script fetches a fresh one per call). If a previously working password fails after an Admin Console deploy, the secret was reset; re-apply Step 1. |
+| `HTTP 403: Admin functionality is disabled` | The runtime-api deployment has no admin password, or the configured value is empty. On Replicated installs, set `Runtime API Admin Password` per Step 1 and deploy. |
+| `HTTP 401` on login | Wrong password, or the challenge expired. Challenges are single-use and expire after 5 minutes; the script fetches a fresh one per call. |
| `HTTP 401: ...provide a valid API key...` on list | The list endpoint authenticates with `X-API-Key`, not the admin JWT. Use the helper script. |
+| Installer-managed default is missing from the list | The release does not include overlay support, or the overlay is not enabled. Upgrade OpenHands Enterprise and confirm that the list reports `source` before saving configurations. |
| Saved a config but the dropdown does not show it | The application caches the list for 60 seconds; wait a minute and reload. Also confirm with `./warm-runtime-configs.sh list`. |
| No warm pods appear | Read the latest reconciler job log (Step 4). Look for image pull errors or scheduling failures. |
| Warm pods `Pending` | Insufficient node resources. Every warm pod reserves the full sandbox resource envelope; lower the pool `count`s or add capacity. |
@@ -538,9 +540,19 @@ GET /api/warm-runtime-configs
X-API-Key: {api-key}
```
-Returns `200` with `{"configs": [{name, image, working_dir, command, environment, count, run_as_user, run_as_group, fs_group, ...}, ...]}`.
-Only configurations saved through this API are listed; the installer-managed default (active only
-while this list is empty) does not appear.
+Returns `200` with the effective configuration set:
+
+```json
+{
+ "configs": [
+ {"name": "v1_current", "image": "...", "source": "file"},
+ {"name": "php-web", "image": "...", "source": "db"}
+ ]
+}
+```
+
+Installer-managed entries have `source: "file"`. API-managed entries have `source: "db"`; a database
+entry with the same name replaces the file entry in this effective list.
**Create or update a configuration** (admin):
@@ -561,7 +573,8 @@ DELETE /api/admin/warm-runtime-configs/{name}
Authorization: Bearer {admin-jwt}
```
-Returns `200` with a confirmation message, or `404` if no configuration has that name.
+Returns `200` with a confirmation message, or `404` if no database configuration has that name. When
+the deleted name also exists in the installer-managed file, that file entry becomes effective again.
## Reference