fix(setup,compose): bundle Redis, fix socket reconnect, and harden the setup wizard by TheodoreSpeaks · Pull Request #5964 · simstudioai/sim

TheodoreSpeaks · 2026-07-25T20:07:39Z

Summary

Self-hosting fixes found by running the wizard end-to-end across all three modes.

Compose

Bundle a redis service (prod + local) and always configure Redis in quick mode. Storage falls back to PostgreSQL without it, but the pub/sub channels (live Chat task-status, table events) have no fallback — every self-hosted stack was running without live updates.

The socket reconnect loop

getSocketUrl() was already correct; the bug was in CSP. generateRuntimeCSP gated the localhost fallback on isDev, and compose runs NODE_ENV=production, so connect-src omitted ws://localhost:3002 and the browser blocked the handshake. CSP now mirrors getSocketUrl by keying that fallback on the app URL being localhost, so proxied deployments keep their page-origin fallback.

Managed Postgres

POSTGRES_PASSWORD only applies on an empty data directory, but the volume outlives its container — so a re-run generated a fresh password the volume ignored and reported "container did not become healthy". Detect an initialized volume and ask; percent-encode the password in the DSN (@ : / # otherwise make it unparseable); and tell "Postgres rejected this password" apart from "Postgres never started".

Kubernetes

Services are ClusterIP, so a healthy install left nothing on :3000. The wizard now offers the port-forward and runs it in the foreground (realtime gets its own, or the editor socket fails).
helm --wait no longer blocks behind a static spinner: helm runs async while the cluster is polled, so it reports 3/3 pods ready · 1 starting. CronJob pods are excluded — 36 of them on a running cluster would swamp the count.

Wizard / lifecycle

Default the run-mode picker to Docker Compose; sharpen the option copy.
sim status finds compose stacks started from any directory (via docker compose ls), reports one entry per stack, and distinguishes "Docker unreachable" from "nothing installed".
Port-conflict handling survives an owner that exits between the lsof snapshot and the kill (ESRCH is the outcome we wanted, not a failure), and stops flagging this stack's own containers as blockers.
Pass SIM_AGENT_API_URL through for Sim devs pointing at a non-prod mothership, and warn when only SIM_CLI_AUTH_ORIGIN is set — that mismatch mints a key one environment and validates it against another.

Type of Change

Bug fix

Testing

Tested manually against live compose and kind clusters:

the bundled redis container comes up healthy and realtime logs Socket.IO Redis adapter connected
:3000/socket.io returns 308 while :3002 returns 200 — the reconnect cause
the k8s dual port-forward returns 200 on both :3000/api/health and :3002/health, and both children are killed on exit; with no forward :3000 returns nothing
the progress query reads 3/3 pods ready and excludes 36 CronJob pods
sim status detects a stack in another checkout and reports Docker-down state correctly

bun run type-check, bun run lint, check:api-validation:strict, docker compose config on both files, 30 CSP tests, and a build of every setup script all pass. Added csp-socket-fallback.test.ts and confirmed it fails against the previous condition.

Two items not verified end-to-end: the Postgres volume-password prompt needs a machine with an orphaned sim-postgres-data volume, and the staging mothership URL in a code comment was derived from DNS rather than repo config.

Supersedes #5966, whose commit is included here.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

… detection Compose shipped no redis service at all — REDIS_URL was ${REDIS_URL:-} in both app and realtime, so every self-hosted stack ran without it. Storage silently falls back to PostgreSQL, but the pub/sub channels (live Chat task-status, table events) have no fallback, so live updates never arrived. - compose (prod + local): add a redis:7-alpine service with a healthcheck, default REDIS_URL to redis://redis:6379, and make app/realtime depend on it being healthy. Not published to the host — only the containers need it, and binding 6379 would collide with a local Redis. An external REDIS_URL in root .env still overrides. Deliberately not written into root .env: doctor pings REDIS_URL from the host, and a compose-internal hostname would fail that probe the same way DATABASE_URL would. - dev mode: configure Redis in quick too. Quick uses a new non-interactive ensureRedis (adopt whatever answers, else start the managed container, warn only if Docker is unavailable); custom keeps the ladder, with corrected copy — the old prompt claimed Redis was only for multi-replica. - lifecycle: detect compose stacks via 'docker compose ls' instead of probing '-f <file> ps' in the working directory. Compose derives the project name from the directory it was started in, so the old probe found a stack only when run from the checkout that launched it (a globally linked sim never could) and listed the same stack once per candidate file. compose ls reports the real project and its config file, so one stack yields one install from anywhere; non-Sim projects are filtered by compose filename. Every compose op now runs in that stack's directory. - lifecycle: distinguish 'Docker unreachable' from 'nothing installed'. With the daemon down, status reported containers as 'absent' and suggested re-running setup; it now says Docker is down and marks state unknown.

…ill ignore POSTGRES_PASSWORD only applies when initdb runs on an empty data directory. The sim-postgres-data volume outlives its container (sim down keeps it, docker rm keeps it, and the wizard's own recreate path keeps it), and inspectManagedContainer recovers the password from the *container*, not the volume — so once the container is gone the password is unrecoverable. Setup then generated a fresh password and ran against the initialized volume. Postgres kept its original password and rejected every connection with 'password authentication failed for user postgres', which surfaced as a misleading 'container did not become healthy'. Detect an already-bootstrapped volume (PG_VERSION present) before choosing a password, and ask: supply the existing password, or delete the volume and start fresh (double-confirmed, since that destroys data). Refusing both fails with the exact docker volume rm command instead of looping.

…e copy Compose was listed first but only preselected when Docker happened to be running — with Docker stopped the cursor sat on 'Local dev', steering people toward a source checkout when they wanted to run Sim. Compose mode calls ensureDocker(true), which offers to start Docker Desktop, so a stopped daemon is no reason to change the default. Also tightens the hints to say what each mode is for: run bundled Sim (fastest way to start), work on Sim itself, test a production-style k8s deploy.

The stack publishes the app on 3000 and realtime on 3002 with no reverse proxy between them, but NEXT_PUBLIC_SOCKET_URL defaulted to empty — which tells the browser client to use the page origin. :3000/socket.io answers 308 (a Next redirect), not a Socket.IO handshake, so the client failed and retried forever. Default it to http://localhost:3002; a proxied deployment overrides it (or sets it empty to use the page origin). Also give COPILOT_API_KEY and SIM_AGENT_API_URL empty defaults so every compose command stops printing 'variable is not set' warnings. The app already falls back to the prod copilot backend when SIM_AGENT_API_URL is blank.

…othership Sim devs testing against a non-prod mothership export SIM_CLI_AUTH_ORIGIN so the Chat key is minted there, but nothing carried the matching backend URL into the install — the app kept defaulting to prod copilot, which rejects a staging key with 'Invalid API key'. Persist SIM_AGENT_API_URL when it is exported, so later docker compose up / dev runs stay on that backend instead of reverting to prod once the shell is gone: SIM_CLI_AUTH_ORIGIN=https://www.staging.sim.ai \ SIM_AGENT_API_URL=https://www.staging.copilot.sim.ai \ bun run setup Setting only the auth origin is the trap, so that combination warns. Neither set is the self-hoster default and stays silent — no prompts, no flags.

vercel · 2026-07-25T20:07:45Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
docs	Ready	Preview, Comment	Jul 25, 2026 9:30pm

cursor · 2026-07-25T20:07:50Z

PR Summary

Medium Risk
CSP and Compose defaults affect every self-hosted runtime; lifecycle/reset changes target the correct Compose project but are destructive if mis-identified—mitigated by Sim content markers and pinned project names.

Overview
Self-hosting fixes across Compose, CSP, and the setup/lifecycle tooling.

Compose adds an internal Redis service (prod + local) with default REDIS_URL=redis://redis:6379 on app and realtime, so pub/sub for live Chat and table events works out of the box. Optional env vars get safer empty defaults.

Socket reconnect loop: generateRuntimeCSP now allows localhost:3002 in connect-src when the app URL is localhost (not only when isDev), matching getSocketUrl() under production Compose on localhost. A dedicated Vitest file locks this in.

Setup wizard: Managed Postgres handles existing volumes (password prompt or wipe), URL-encoded DSNs, and clearer auth vs startup failures. Quick dev always provisions Redis via ensureRedis. K8s uses async helm install with pod progress, dual port-forward (app + realtime), and shared forwardCommands. Lifecycle discovers Compose via docker compose ls, pins -p and cwd, content-marks Sim stacks, and treats unreachable Docker as unknown state. Compose port checks skip ports this stack already publishes; port-kill handles ESRCH/EPERM. mothershipOverride persists/warns on SIM_AGENT_API_URL vs SIM_CLI_AUTH_ORIGIN. Wizard defaults to Compose.

^{Reviewed by Cursor Bugbot for commit 910f367. Bugbot is set up for automated code reviews on this repo. Configure here.}

…containers Two failures from one compose re-run: - 'Kill it for me' crashed setup with 'kill() failed: ESRCH: No such process'. The owner list is an lsof snapshot, so the process can exit before the signal lands — which is the outcome we wanted, not an error. ESRCH now counts as freed, EPERM warns that it must be stopped by hand, and anything else warns; the loop re-probes either way instead of aborting a setup that had already written .env. - Compose mode demanded 3000/3002 be free even when this stack was the one holding them, so re-running setup against a running install reported its own realtime container as a blocker and offered to kill Docker's listener. 'docker compose up -d' reconciles its own containers, so skip the check when the project already has some. A foreign process is still caught, and a foreign container still surfaces as a bind error from compose.

…code DSN passwords Review round on #5964: - The socket reconnect was a CSP bug, not a URL bug. getSocketUrl() already falls back to localhost:3002 for a localhost page, but generateRuntimeCSP gated that same fallback on isDev — and compose runs NODE_ENV=production, so connect-src omitted ws://localhost:3002 and the browser blocked the handshake. Key the fallback on the app URL being localhost instead, mirroring getSocketUrl. Revert the compose NEXT_PUBLIC_SOCKET_URL default: an explicit value suppresses the page-origin fallback that reverse-proxied self-hosts depend on, and ':-' treats empty as unset so the documented escape hatch could not work either. LOCALHOST_HOSTNAMES is duplicated locally because csp.ts is loaded by next.config.ts before @/ aliases resolve. - Percent-encode the password when building the Postgres DSN. A user-supplied password containing @ : / # does not merely re-parse to the wrong host — it fails to parse as a URL at all, so a correct password surfaced as a connection failure. - Tell 'Postgres rejected this password' apart from 'Postgres never started'. On the keep-the-volume path a wrong password left a healthy server and the old generic 'container did not become healthy' error, which is the confusion this change set exists to remove. Adds a CSP regression test for the unset-socket-URL production case; verified it fails against the previous condition.

TheodoreSpeaks · 2026-07-25T20:21:54Z

@greptile

TheodoreSpeaks · 2026-07-25T20:21:54Z

@cursor review

greptile-apps · 2026-07-25T20:24:41Z

Greptile Summary

This PR improves reliability across the self-hosted setup and lifecycle flows.

Bundles Redis with both Compose variants and configures Redis during quick local-development setup.
Aligns runtime CSP socket permissions with the localhost Socket.IO fallback.
Preserves managed Postgres volumes safely by detecting initialized data and requesting the existing password or explicit deletion.
Discovers Compose installations globally and targets lifecycle operations using their recorded project metadata.
Adds Kubernetes deployment progress reporting and foreground port-forwarding for both app and realtime services.
Hardens port-conflict handling and propagates custom mothership configuration.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
apps/sim/lib/core/security/csp.ts	Aligns runtime CSP socket sources with the existing localhost fallback and adds focused regression coverage.
docker-compose.local.yml	Adds an internal Redis service and makes the app and realtime services depend on its health.
docker-compose.prod.yml	Bundles Redis for production Compose and documents page-origin versus explicit socket routing.
scripts/setup/db.ts	Encodes managed Postgres credentials and handles initialized persistent volumes explicitly.
scripts/setup/lifecycle.ts	Discovers Compose projects globally, pins operations to recorded project metadata, and improves status reporting.
scripts/setup/modes/compose.ts	Avoids treating the active stack’s own published ports as startup conflicts and persists mothership overrides.
scripts/setup/modes/dev.ts	Provisions Redis in quick mode and mirrors its URL to realtime configuration.
scripts/setup/modes/k8s.ts	Reports Helm deployment progress and offers coordinated app and realtime port-forwarding.
scripts/setup/ports.ts	Handles processes disappearing during conflict resolution without aborting setup.
scripts/setup/redis.ts	Adds noninteractive Redis reuse or provisioning for quick setup.
scripts/setup/steps.ts	Persists custom agent endpoints and warns about mismatched key-minting and validation environments.
scripts/setup/wizard.ts	Makes Docker Compose the default setup mode and clarifies mode descriptions.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  Wizard[Setup wizard] --> Mode{Selected mode}
  Mode -->|Compose| Compose[Write environment and start Compose]
  Compose --> RedisC[Bundled Redis]
  Compose --> AppC[App on port 3000]
  Compose --> RealtimeC[Realtime on port 3002]
  Mode -->|Local dev| Database[Resolve managed or external Postgres]
  Database --> RedisD[Resolve or provision Redis]
  RedisD --> Dev[Start local development services]
  Mode -->|Kubernetes| Helm[Install or upgrade Helm release]
  Helm --> Progress[Poll pod readiness]
  Progress --> Forward[Forward app and realtime services]
  AppC --> Lifecycle[Lifecycle discovery and management]
  RealtimeC --> Lifecycle

_{Reviews (6): Last reviewed commit: "fix(setup): warn on both halves of a mot..." | Re-trigger Greptile}

cursor

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit a28f49f. Configure here.}

…all progress Two things made k8s mode the least satisfying path. The services are ClusterIP, so a successful install left nothing on :3000 — 'Sim is ready' was true about the cluster and useless to the user, who had to notice and run a port-forward by hand. Compose opens a browser and dev offers to start the server; k8s now offers the forward the same way and runs it in the foreground so Ctrl-C ends it. Realtime gets its own forward (kubectl takes one resource per invocation) or the editor socket fails; it is a child in the same process group, so the terminal's Ctrl-C reaches it, and it is killed explicitly when the app forward exits. 'helm --wait' then blocked for minutes with a single static spinner, so a slow image pull looked identical to a wedged install. Run helm asynchronously and poll the cluster, so the spinner reports '3/3 pods ready · 1 starting'. CronJob-owned pods are excluded: the chart schedules a lot of them (36 on a running cluster here) and they finish as Completed, which would swamp the count and make readiness jitter for reasons unrelated to the install. Restarting pods are surfaced too — a cold cluster restarts realtime while Postgres comes up, and a silent spinner made that look like nothing was happening.

Cursor (High): composeInstalls treated any project whose config basename was docker-compose.prod.yml or docker-compose.local.yml as a Sim install. Those names are common, and sim reset runs 'compose down -v' — so a stranger's stack could have had its volumes destroyed. I introduced that reach. The previous ROOT-scoped '-f' probe was implicitly safe because it could only ever see the project in this checkout; switching to a global 'compose ls' to find stacks started elsewhere means projects must be identified by content instead. Read the config file Docker recorded and require a Sim marker (the published app image, or the app Dockerfile this repo builds), so both the prod and local variants match while an unrelated file with the same name does not. An unreadable or since-deleted file is left unmanaged rather than assumed ours. Verified against a decoy nginx compose file using our exact filename: ignored, while both real Sim compose files still match.

TheodoreSpeaks · 2026-07-25T20:51:38Z

@greptile

TheodoreSpeaks · 2026-07-25T20:51:39Z

@cursor review

…h k8s forwards Review round on #5964: - ensureComposePortsFree skipped conflict handling whenever the project had any container running, so leftover db/redis (which publish neither app port) waved through a foreign process on :3000 — it then surfaced as a raw compose bind error instead of the prompt. Read the host ports the project actually publishes and skip only those; the remaining ports still get the full check. Reading from the containers rather than the file matters because what counts is what is bound right now. - The post-install note and the skip path documented only the app forward, while offerPortForward runs two. Skipping the prompt or copying the printed command left the editor's socket dead — the exact failure this change set exists to fix. Both commands now come from one forwardCommands() helper, so what is printed and what is run cannot drift.

TheodoreSpeaks · 2026-07-25T20:58:49Z

@greptile

TheodoreSpeaks · 2026-07-25T20:58:50Z

@cursor review

…ime forward Third round on the same theme, so fix it at the root rather than at another call site. - lifecycle's k8sReachHints (used by sim start/restart) still restated an app-only forward, recreating the dead editor socket the setup path had just been fixed for. forwardCommands is now exported and consumed there, so every place that tells a user how to reach a ClusterIP release derives it from one definition. - The realtime forward was spawned with stdio ignored and never checked, so a busy :3002 or a missing service killed it silently while the app forward kept running — indistinguishable from success until the editor won't connect. Keep its stderr, warn on an exit we did not ask for, and stay quiet on the intentional kill.

TheodoreSpeaks · 2026-07-25T21:05:10Z

@greptile

TheodoreSpeaks · 2026-07-25T21:05:11Z

@cursor review

composeInstalls records the real project name from 'compose ls' and status and the destructive confirms print it, but every op ran 'compose -f <file>' with only cwd set — so Compose re-derived the project from that directory. The derived name is frequently not the recorded one: a directory is lowercased and stripped of dots (Sim.Demo_Test derives simdemo_test), and an explicit -p or COMPOSE_PROJECT_NAME at creation diverges outright. stop/down/reset could therefore act on a different project than the one named in the confirm, and reset runs 'down -v'. Route every op through composeArgs(), which pins '-p <recorded project>'. cwd stays, since the file's own relative paths still resolve against it. Verified with a stack started as -p pinned-name from a directory deriving simdemo_test: the old form found 0 of its containers, the pinned form finds them.

mothershipOverride warned only when SIM_CLI_AUTH_ORIGIN was set without SIM_AGENT_API_URL, while its own copy said to set both or neither. The reverse is the same failure mirrored: with only SIM_AGENT_API_URL set, the Chat key is still minted against the default prod auth origin and then validated against the override, which rejects it — silently, which is exactly what this helper exists to prevent. Warn on either asymmetry, and read the default origin from one constant shared with the handoff so the message can't claim an origin the code no longer uses.

TheodoreSpeaks · 2026-07-25T21:17:59Z

@greptile

TheodoreSpeaks · 2026-07-25T21:18:00Z

@cursor review

mothershipOverride ran two steps after promptCopilotKey, so a half-set override minted a key against one environment, stored it, and only then warned that the other environment would reject it. Worse on a re-run: promptCopilotKey offers to keep an existing COPILOT_API_KEY and defaults to yes, so the bad key survives. Move the override ahead of the key prompt in both compose and dev, so the warning arrives while it can still change the outcome — the user can abort and set the missing half before anything is minted. Nothing in the override depends on the key, so the order is free.

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 910f367. Configure here.}

TheodoreSpeaks added 5 commits July 25, 2026 12:36