Cartographer: fix OOM when backing up servers with many attachments - #283
Open
RWolfyo wants to merge 2 commits into
Open
Cartographer: fix OOM when backing up servers with many attachments#283RWolfyo wants to merge 2 commits into
RWolfyo wants to merge 2 commits into
Conversation
… memory Every message attachment was downloaded, base64-encoded and kept on the model until the whole guild had been serialized, so peak memory scaled with the total size of a server's media - roughly 4x it, before the JSON document was built on top. The v2.2.0 guard only rejects individual files over the guild's upload limit; it puts no ceiling on the total. A server with a busy media channel produced a 1.9GB backup document and a ~16GB RSS peak, ended by the host's OOM killer. Restoring had the same problem in reverse, since the whole file was read into a string before being parsed. Backups are now zip archives: backup.json plus an attachments/ folder. Attachments stream into the archive as they download and are read back out one at a time, so memory stays flat regardless of how much media a server has. Nothing is skipped - a large backup just takes longer. The archives are also smaller, since base64 inflated every file by a third and the JSON is now compressed. Backups written before this change still load and restore: load_backup picks the format by extension and FileBackup keeps the base64 field. Measured on a server with 52MB of attachments: 288MB -> 444MB RSS across a full backup and restore, where the previous code climbed until killed.
channel.history(limit=...) yields newest-first, and restore_channel_messages replays the stored list in order, so a restored channel was rebuilt backwards. Store messages oldest-first instead. The same most-recent N messages are still the ones backed up; only the order they are written in changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
FileBackup.serializedownloads each attachment, base64-encodes it, and holds it on the model until the entire guild has been serialized. Peak memory therefore scales with a server's total media — about 4x it, since the raw bytes, the base64 bytes and the decodedstrare all live at once, beforemodel_dump_jsonbuilds the document on top.The v2.2.0 check is per-file:
That rejects individual oversized files but places no ceiling on the total, so any number of small attachments passes straight through.
On my bot this produced a 1.9 GB backup document and a ~16 GB RSS peak against a 16 GB
MemoryMax, and the OOM killer ended the process roughly every 10 minutes —last_backupnever persisted, so every restart retried the same backup and died again. Restoring such a backup would have failed the same way, sincerestorelatestdoesmodel_validate_json(path.read_text())on the whole file.The change
Backups are written as
.zip—backup.jsonplus anattachments/folder. Attachments stream into the archive as they download; on restore they are read back one at a time. Memory stays flat regardless of media volume, and nothing is dropped: a large backup simply takes longer.A backup is still a single file on disk, so the rotation, wipe, listing and size-reporting paths are untouched.
Backward compatible.
load_backuppicks the format by extension andFileBackupkeepsfilebytesfor pre-2.3.0 backups. Existing backups load and restore unchanged; only new ones are zips.The guild upload limit check is unchanged — same
filesize_limitfallback, same comparison.Second commit
Unrelated bug found while testing:
channel.history(limit=...)returns newest-first andrestore_channel_messagesreplays the list in order, so restored channels came back backwards. Messages are now stored oldest-first; the same most-recent N are still selected. Kept as a separate commit so it can be dropped independently.Testing
Run against a live bot (Red 3.5, Python 3.11):
.jsonbackup restored successfully via the base64 fallback.Version
2.2.0 → 2.3.0 — minor rather than patch, since the on-disk backup format changes.