Add opt-in Synology Hyper Backup WebDAV mode#608
Conversation
79ea69f to
3011c72
Compare
3011c72 to
a86375f
Compare
|
This pull request has been inactive for 30 days. Is it still in progress? If so, please leave a comment or make an update to keep it open. Otherwise, it will be automatically closed in 15 days. |
|
I assume there's no interest. Just FYI, without these changes (which I have been using for more than a month) I'd have requested a refund, Hyper Backup performance was simply not acceptable. |
Hey @taglia, First, I want to apologize for the long silence on this PR. It definitely wasn't because we didn't care or weren't interested. We have to balance work on many different priorities, and unfortunately we can't always respond to every issue and PR as quickly as we'd like. That said, we do review everything that comes in, and your contribution certainly didn't go unnoticed. The reason review stalled is that, in parallel, we have been landing a series of changes on main that overlap heavily with what this PR addresses:
Because several of those commits touch the exact same files your PR modifies (docker/entrypoint.sh, GET.handler.ts, PROPFIND.handler.ts), the branch now shows conflicts against main. That's the practical reason it's stuck, not a rejection of the idea. Since a good part of the caching/locking/keep-alive work is now already in main, unconditionally, without needing the WEBDAV_HYPER_BACKUP_MODE flag, could you rebase on top of current main and see what's actually left to add? One more thing worth discussing before we merge anything further: part of the remaining diff (the lock_keep_alive.@writer_version_* filtering under /Control/lock/, and the zero-byte .bucket/.index "placeholder" detection under /Pool/...) works by pattern-matching Synology's internal file and path naming conventions on the server side. I'd rather we not take that on permanently, as WebDAV is meant to stay a client-agnostic protocol, and baking in assumptions about one vendor's internal implementation means we silently break if Synology changes those conventions in a future DSM release, with no protocol-level way to detect it. It's also a real correctness risk for a backup tool specifically: if that heuristic ever misclassifies a genuine data write as a "placeholder", that's silent data loss that only surfaces at restore time. On top of that, since we now support real WebDAV LOCK/UNLOCK natively, Hyper Backup's own lock-file workaround may not even be necessary anymore, so its worth testing before keeping that logic at all. So, from your remaining delta, I'd suggest we keep the generic, protocol-agnostic pieces (fast-path caching, placeholder handling that isn't tied to Synology-specific paths, diagnostic timing logs, Docker startup tweaks) and drop or heavily reconsider the parts that pattern-match Synology's internal file naming, those are better handled on Synology's side, or at minimum kept far more narrowly scoped and only if we confirm native LOCK/UNLOCK doesn't already cover it. If you can rebase and split the PR along those lines, I'll get it reviewed promptly. And if your before/after numbers still show a meaningful improvement from the generic caching work alone, that's genuinely useful data for deciding whether it should become the default. Sorry again for how this played out, and thank you so much for sticking with it! |
Summary
This PR adds an opt-in WebDAV compatibility/performance mode for Synology Hyper Backup, enabled with
WEBDAV_HYPER_BACKUP_MODE=true.The default WebDAV behavior remains unchanged unless this flag is enabled.
Motivation
Synology Hyper Backup performs many repeated WebDAV metadata and listing operations while creating tasks, uploading backup chunks, and restoring data. Against the current WebDAV implementation this can result in long idle periods between uploads and very slow overall backup throughput.
In the tested environment, this reduced the estimated time to upload 100 GB with Hyper Backup from about 72 hours to about 11.5 hours.
Changes
WEBDAV_HYPER_BACKUP_MODEconfiguration support.GET, andHEADlookups.Validation
Tested with Synology Hyper Backup using the Docker WebDAV container.
Performance observed in the same environment:
WEBDAV_HYPER_BACKUP_MODE=trueAdditional validation:
main, completed a 4.1 GB backup/restore smoke test.Notes
This is intentionally gated behind
WEBDAV_HYPER_BACKUP_MODE=truebecause some behavior is tailored to Synology Hyper Backup's WebDAV access pattern, especially zero-byte placeholder handling and stale lock keepalive cleanup.