[WIP] Switch database backend to postgresql

[jpodivin]

This is obviously a substantial change. There is no easy way to migrate the trace data we have collected so far, but since we can't properly query it anyway, I don't see that as a much of an issue. That being said, we could export them, if we really want to.

• arize-phoenix is now installed with asyncpg
• Database credentials are set to defaults, it's going to run inside cluster with now outside facing API anyway
• New 150 GB volume for the database
• phoenix-data is now 2 GB, that should be enough for the runtime, maybe even an overkill
• Shrinking volumes can be tricky, so we may have to delete the original and create a new one
• Uses quay.io/sclorg/postgresql-16-c9s container image

RELEASE NOTES BEGIN

Replace Phoenix SQLite backend with Postgresql.

RELEASE NOTES END

[gemini-code-assist]

Code Review

This pull request migrates the Phoenix observability backend from SQLite to PostgreSQL, updating the Docker Compose configuration, OpenShift deployment manifests, and documentation to support the new database service. The review feedback highlights a critical issue where using the official PostgreSQL image on OpenShift will fail due to permission restrictions under non-root execution. To resolve this, it is recommended to switch to the OpenShift-compatible sclorg PostgreSQL image, which requires corresponding updates to environment variables, volume mount paths, and secret keys across both OpenShift and local Compose environments. Additionally, adding a database healthcheck in Docker Compose is suggested to prevent startup race conditions.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[TomasTomecek]

Overall this looks really good, thanks for updating also the documents.

I'm wondering if we need to do some cleanup of the existing data once we deploy it.

[jpodivin]

Overall this looks really good, thanks for updating also the documents.

I'm wondering if we need to do some cleanup of the existing data once we deploy it.

That's the "best" part. There is no way to migrate the data to the new database. So the cleanup is kind of... given.

[mfocko]

LGTM

[mfocko]

Did it actually utilize most of the 150Gi?

[jpodivin]

A valid question, I'm not sure about that.

[opohorel]

oc exec deploy/phoenix -- df -h /mnt/data
Filesystem                                        Size  Used Avail Use% Mounted on
[redacted, so noone from prodsec comes after me]  150G  6.9G  144G   5% /mnt/data

[opohorel]

I would suggest going with 20Gi, so we have some headroom. we'll still save a huge portion of the allocation freed, and we hopefully don't have to worry about it too soon
edit: I meant this comment for db-data not the phoenix-data

[jpodivin]

I'll revise the PR then.