Embedded Postgres cannot initialise on Railway: initdb fails at both documented volume paths, and the deploy still reports success

[jerelvelarde]

What is wrong

With EMBEDDED_POSTGRES=on, the container cannot initialise its cluster on Railway, and it cannot do
it at either of the two mount paths this repo suggests. The two suggestions disagree with each other,
and neither one works.

The deployment does not look broken from the platform's side. Railway reports the deploy as a
success, the container starts, and agent-computer comes up:

agent-computer listening on http://localhost:4100

But the public URL serves a persistent HTTP 502, because docker/s6/s6-rc.d/api/dependencies.d/
contains postgres and migrate, and neither of those ever starts once postgres-init has failed.
The only place the real failure is visible is the container log.

This should reproduce on any platform whose persistent volume is an ext4 mount owned by root, which
is the common case, not a Railway peculiarity. docs/deployment.md names Railway specifically as one
of the recommended platforms:

Railway, Render, Fly.io. All run this image directly and all provision PostgreSQL in a click,
which makes them the shortest path from nothing to a running deployment.

Environment

• Platform: Railway, service deployed from this image, with a Railway volume attached
• EMBEDDED_POSTGRES=on
• Upstream main at commit 7fda66a

Steps to reproduce

1. Deploy this image to Railway.
2. Set EMBEDDED_POSTGRES=on.
3. Attach a Railway volume, at either /var/lib/postgresql/data or /var/lib/postgresql.
4. Deploy, then open the generated public URL.

The deploy succeeds; the URL returns 502. Read the container log for what actually happened.

Failure mode 1 — volume mounted at /var/lib/postgresql/data

This is what docs/deployment.md line 27 instructs:

Give it a volume at /var/lib/postgresql/data. Without one, a redeploy takes the audit trail with
it, and the audit trail is the product.

Verbatim from the container log:

initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
initdb: detail: It contains a lost+found directory, perhaps due to it being a mount point.
initdb: hint: Using a mount point directly as the data directory is not recommended.
Create a subdirectory under the mount point.
s6-rc: warning: unable to start service postgres-init: command exited 1

The volume is an ext4 filesystem, so its root contains lost+found, and initdb refuses to use a
non-empty directory.

Failure mode 2 — volume mounted at /var/lib/postgresql

This is what the Dockerfile comment implies, at line 126:

a database inside a container lives and dies with that container unless /var/lib/postgresql is a
mounted volume

Verbatim from the container log:

initdb: error: could not create directory "/var/lib/postgresql/data": Permission denied
s6-rc: warning: unable to start service postgres-init: command exited 1

The mount root is owned by root, and initdb runs as the unprivileged postgres user, so it cannot
create data underneath it.

Root cause

docker/s6/scripts/postgres-init.sh hardcodes the data directory and then runs initdb as
postgres without ever creating that directory or taking ownership of it:

DATA=/var/lib/postgresql/data
if [ ! -s "$DATA/PG_VERSION" ]; then
  s6-setuidgid postgres /usr/lib/postgresql/16/bin/initdb -D "$DATA" -A trust -U openbot >/dev/null

(docker/s6/s6-rc.d/postgres/run line 11 hardcodes the same path for the server itself.)

The only thing that gives postgres ownership of the path is a build-time step in the Dockerfile,
lines 131-132:

  && mkdir -p /var/lib/postgresql/data /var/run/postgresql \
  && chown -R postgres:postgres /var/lib/postgresql /var/run/postgresql

A volume mounted over that path at runtime masks the build-time chown entirely, and nothing in the
image repeats it at runtime. There is no fix-attrs.d anywhere under docker/s6/ — the fix-attrs
service visible in the s6 log is s6-overlay's built-in, which is inert without configuration — so no
runtime step chowns the mount.

postgres-init is a oneshot and its up runs as root, so it is in a position to fix this itself.

Proposed fix

Two halves, and they solve different failure modes. Both are needed.

1. Create and chown the data directory as root, before dropping to postgres. In
docker/s6/scripts/postgres-init.sh, before the initdb call:

mkdir -p "$DATA"
chown postgres:postgres "$DATA"

This fixes failure mode 2 — the volume mounted at /var/lib/postgresql. It also makes the script
correct in the plain docker run -v openbot-data:/var/lib/postgresql case, which today depends
entirely on a build-time chown that a mount hides.

It does not fix failure mode 1. If the volume is mounted directly at $DATA, the directory
already exists and already contains lost+found, and initdb still refuses it. No amount of
chowning changes that.

2. Correct the documented mount path. For failure mode 1 the fix is documentation: the
recommended mount should be the parent, /var/lib/postgresql, so that $DATA is a subdirectory of
the mount rather than the mount itself — which is exactly what initdb's own hint asks for, and what
the Dockerfile comment already says. docs/deployment.md should be corrected to agree with it, in
both places it currently says /var/lib/postgresql/data: the docker run example on line 14 and the
prose on line 27.

I have not tested this fix. It is inferred from the log output and the scripts, not verified on a
deployment.

How to check it is fixed

Deploy to Railway with EMBEDDED_POSTGRES=on and a volume at the documented path, and confirm that
postgres-init completes, that migrate and api start, and that the public URL serves the app
rather than a 502.

Worth considering separately: postgres-init failing takes the whole deployment down while the
platform still reports success. Whatever the mount path ends up being, a first-start failure that is
only visible in container logs is a bad way to find out.