feat: Object stores and database-aware backups - #199
Merged
Conversation
…tion Object stores are now a first-class resource. A self-hosted, S3-compatible store (MinIO) can be deployed straight from the Object Stores page and connected automatically: FlatRun issues the store's credentials, creates its bucket, and registers it as a backup target. An already-running deployment can instead be connected as a store by supplying its S3 details. Self-hosted stores run as the host user so their data stays manageable without root, and they join a shared storage network (the database network by default, or a dedicated one) so the agent and apps can reach them by address. Deploying an app from the marketplace now hands off to the standard deployment flow rather than a separate one-field install.
A connected store's full contents can now be listed and managed directly: every object in the store is shown (not just backup archives), and individual objects can be uploaded, downloaded, and deleted.
A store can now be attached to another deployment: its endpoint, bucket, region, credentials, and path-style flag are written into that deployment's environment under a chosen prefix, and the service is wired to load them. A self-hosted store is addressed by its in-cluster name and the app is joined to its network; an external store uses its public URL. The deployment is restarted to pick it up.
A store's contents can now be copied to another store, for an offsite copy of a self-hosted store or a local cache of a remote one. Replication is incremental: objects already present at the same size in the target are skipped, so re-running only moves what changed.
A store's server can host many buckets, so a store is no longer limited to its one configured bucket: its buckets can be listed and created, and objects can be browsed and uploaded within any of them. Deleting objects in the store's backup bucket is refused so browsing cannot destroy backups; other buckets stay fully editable.
The bucket list now reports each bucket's object count and total size (bounded so large buckets stay responsive), and objects are listed a page at a time so a bucket of any size can be browsed. An empty, non-backup bucket can be deleted.
A deployment whose service is a database server (including the shared infrastructure database) is now detected and backed up with a full logical dump of every database it holds, using its root credentials, instead of only copying its live data directory. This makes database backups consistent and complete with no per-deployment configuration; a shared database captures all app data in one dump, and a standalone or sidecar database captures its own.
An app that uses the shared database now also dumps just its own database into its own archive, resolved automatically from the app's recorded database usage. This keeps an app's (often more frequent) backups self-contained rather than depending on the shared database's own, possibly less frequent, full backup.
When a deployment's database is dumped, its live data directory is no longer also copied into the archive: that copy duplicates the dump and, taken from a running database, can be inconsistent. Other files such as app data and config are still included. Backups also gain honoured exclude patterns, which were previously declared but ignored.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds self-hosted object stores and reworks how backups capture databases.
Object stores. Deploy a self-hosted S3 store (MinIO) from a template and FlatRun connects it automatically (reads its issued credentials, creates the bucket, registers it as a backup target), or connect an already-running deployment as an external store. A connected store supports:
Two design points a reviewer should know:
Backups. A backup used to copy a deployment's data directory, which is unsafe for a running database: the live files can be read mid-write and restored inconsistent. Backups now detect database servers from the compose and dump them logically instead:
pg_dumpall/mysqldump --all-databases) with its root credentials.Dashboard side: flatrun/ui#99.
Follow-up: the MySQL/MariaDB dump path mirrors Postgres and is not yet run against a live MySQL server.