[HWORKS-2973] Superset is not captured by backups and cannot be restored - #640
Open
jimdowling wants to merge 1 commit into
Open
[HWORKS-2973] Superset is not captured by backups and cannot be restored#640jimdowling wants to merge 1 commit into
jimdowling wants to merge 1 commit into
Conversation
https://hopsworks.atlassian.net/browse/HWORKS-2973 Superset keeps all of its state in a bundled MySQL database that no backup path captured: the native database backup covers RonDB only, the Kubernetes-object backup takes no volume data, and Superset's own Secrets were not labelled for it. An uninstall, a lost volume or a disaster recovery therefore destroyed every dashboard, chart, saved query and database connection, and because Superset encrypts stored connection passwords with its secret key, a database recovered without that key left every connection present but unusable. The chart change adds a scheduled backup, brings the three Secrets into the platform backup, and adds a restore behind a declarative writer barrier. Document the operator side of that on the disaster-recovery page. Superset joins the list of services a consistent backup covers, with a backup section describing what the dump contains, where it lands, and why the secret key has to travel with the database. The restore procedure is two steps, and the page says so rather than implying one: setting the flag holds every Superset process at zero replicas and a Job reloads and migrates the database, then clearing the flag and upgrading again brings Superset back. It also covers finding a backup id, the fresh-cluster variant and why that one needs the Velero Restore name, re-running a restore, and reading the audit trail. Two ConfigMaps are described because the split is load-bearing. Execution state is cluster-local, since a restored completion marker would otherwise let a real restore report success over a database that never received it, and only the append-only audit trail is captured by the backup. Phase progress moves forward only, which is what makes the documented claim true that a retry does not repeat destructive work. The limitations are stated rather than left to be discovered: no cross-service point-in-time guarantee, credential rotation unsupported for clusters intended for in-place restore, and a Trino connection password that can go stale if Superset and RonDB are restored to points far apart, with the check and the one-step repair. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
jimdowling
force-pushed
the
HWORKS-2973
branch
from
August 27, 2026 15:23
17345eb to
41eacfd
Compare
jimdowling
marked this pull request as ready for review
August 27, 2026 17:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://hopsworks.atlassian.net/browse/HWORKS-2973
Documents Superset backup and restore on the disaster-recovery page, and narrows the multi-service backup claim to what the independent schedules actually provide.
Pairs with logicalclocks/hopsworks-helm#2245, which implements it. Merge that first: this page describes behaviour that does not exist until the chart ships.
What is documented
Superset joins the backed-up services. The intro list said a consistent backup covers RonDB, HopsFS, OpenSearch and Kubernetes objects. Superset was absent, which was accurate before this ticket and is now the gap the chart closes. It gains an entry naming what lives in the Superset metadata database and stating plainly that it is a separate MySQL from RonDB.
A backup section covering what the dump contains, where it lands (
superset_backup/<backup-id>/, the same object storage as the other backups), the two objects each run writes, and the metadata index. It explains why the secret key must be restored alongside the database, since connection passwords are encrypted with it, and gives the command to list the backups that were taken. Operators who supply the Superset Secrets themselves are told to add the Velero selector label, because the chart only labels the Secrets it creates.A restore section covering the two-step procedure that the declarative design requires: set the restore flag with a backup id and upgrade, wait for the terminal phase, then clear the flag and upgrade again so the workloads resume. It covers finding a backup id, the fresh-cluster variant and why it needs the Velero Restore name, re-running a restore, reading the audit trail, and clearing the maintenance fence without destroying that trail.
ArgoCD. Both the Superset Secrets and the Trino auth Secrets need
ignoreDifferencesplusRespectIgnoreDifferences=truein the Application sync options. Without the sync option,ignoreDifferencesonly changes the diff ArgoCD displays and a sync still reapplies the rendered values over the restored ones.Limitations, stated rather than implied. No cross-service point-in-time guarantee, because the backups run on independent schedules. Credential rotation is unsupported for clusters intended for in-place restore. Restoring Superset and RonDB to points far apart can leave a user's copied Trino password stale, which does not self-heal, so the check and the one-step repair (delete the connection and let the backend recreate it) are given.
Notes
The upstream merge in this branch resolves against the HWORKS-2971 Trino documentation, which landed while this sat on a fork. It added a
### Trino authenticationsection and extended the Kubernetes-objects bullet with Trino. Both are kept alongside the Superset content.markdownlint-cli2reports 0 issues on the page andmkdocs build -spasses.Also corrects the setup command in this repo's own
.claudenotes, which still pointed at arequirements-docs.txtthat no longer exists (dependencies moved topyproject.toml, so the command isuv sync).Signed-off-by: Jim Dowling jim@logicalclocks.com
Co-Authored-By: Claude Opus 5 (1M context) noreply@anthropic.com