Stream restores instead of buffering them in RAM - #79
Conversation
There was a problem hiding this comment.
Pull request overview
This PR updates the backup-restore flow to avoid buffering the entire backup archive in memory by reading required metadata entries first and then streaming the remaining tar entries directly into the restored Docker volume.
Changes:
- Replace restore parsing that buffered volume contents in RAM with
openBackup(...)+ a streamingtar.Readerpipeline. - Update restore volume population to stream tar entries via
io.Pipe()into Docker’sCopyToContainer. - Adjust tests to validate the new “metadata-first then streamed entries” restore behavior.
Tip
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| internal/docker/namespace.go | Switch restore path to use the new streaming backup opener instead of buffering parsed volume data. |
| internal/docker/application_backup.go | Introduce openBackup and streaming tar copy helpers; update restore to stream into the volume. |
| internal/docker/application_backup_test.go | Update round-trip test to validate streamed entries after metadata parsing. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
8f8a9fb to
3ca23fc
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.
Suppressed comments (1)
internal/docker/application_backup.go:375
- The metadata-first archive contract is not enforced here: this loop accepts and discards arbitrary volume entries until both settings files appear. A backup with metadata at the end therefore decompresses the full data set during this pass and again after the rewind, contrary to the PR's stated design. Reject non-metadata entries until both settings files have been read, and update the test fixture to place metadata first plus cover rejection of misplaced metadata.
for appData == nil || volData == nil {
When restoring a backup, we had been reading the entire archive into memory before extracting the metadata and populating the volume. For a large backup this would be problematic.
Instead, let's stream the contents in, file at a time.