Skip to content

Add opt-in Synology Hyper Backup WebDAV mode#608

Open
taglia wants to merge 1 commit into
internxt:mainfrom
taglia:hyper-backup-webdav-optimizations
Open

Add opt-in Synology Hyper Backup WebDAV mode#608
taglia wants to merge 1 commit into
internxt:mainfrom
taglia:hyper-backup-webdav-optimizations

Conversation

@taglia

@taglia taglia commented Jun 17, 2026

Copy link
Copy Markdown

Summary

This PR adds an opt-in WebDAV compatibility/performance mode for Synology Hyper Backup, enabled with WEBDAV_HYPER_BACKUP_MODE=true.

The default WebDAV behavior remains unchanged unless this flag is enabled.

Motivation

Synology Hyper Backup performs many repeated WebDAV metadata and listing operations while creating tasks, uploading backup chunks, and restoring data. Against the current WebDAV implementation this can result in long idle periods between uploads and very slow overall backup throughput.

In the tested environment, this reduced the estimated time to upload 100 GB with Hyper Backup from about 72 hours to about 11.5 hours.

Changes

  • Add WEBDAV_HYPER_BACKUP_MODE configuration support.
  • Add an in-memory WebDAV fast-path cache for repeated path, item, folder listing, GET, and HEAD lookups.
  • Optimize Hyper Backup zero-byte placeholder handling for bucket/index files.
  • Use a faster file replacement path for existing files in Hyper Backup mode.
  • Hide and asynchronously clean up stale Synology lock keepalive files that can block task creation.
  • Add targeted timing logs for WebDAV request diagnostics.
  • Improve Docker WebDAV startup and healthcheck behavior.

Validation

Tested with Synology Hyper Backup using the Docker WebDAV container.

Performance observed in the same environment:

Mode Observed upload rate Estimated time for 100 GB
Existing WebDAV behavior 100 GB in ~72 hours ~72 hours
WEBDAV_HYPER_BACKUP_MODE=true 2.3 TB in ~11 days ~11.5 hours

Additional validation:

  • Completed the full 2.3 TB backup in about 11 days.
  • Completed an integrity check successfully.
  • After rebasing onto current main, completed a 4.1 GB backup/restore smoke test.
  • Restored about 2.96 GB and verified checksums for about 20 files; all matched.
  • Docker image builds successfully.

Notes

This is intentionally gated behind WEBDAV_HYPER_BACKUP_MODE=true because some behavior is tailored to Synology Hyper Backup's WebDAV access pattern, especially zero-byte placeholder handling and stale lock keepalive cleanup.

@taglia
taglia requested a review from larryrider as a code owner June 17, 2026 10:14
@taglia
taglia force-pushed the hyper-backup-webdav-optimizations branch from 79ea69f to 3011c72 Compare June 17, 2026 11:33
@taglia
taglia force-pushed the hyper-backup-webdav-optimizations branch from 3011c72 to a86375f Compare June 17, 2026 11:36
@github-actions

Copy link
Copy Markdown

This pull request has been inactive for 30 days. Is it still in progress? If so, please leave a comment or make an update to keep it open. Otherwise, it will be automatically closed in 15 days.

@github-actions github-actions Bot added the stalled This change is inactive or awaiting input label Jul 18, 2026
@taglia

taglia commented Jul 21, 2026

Copy link
Copy Markdown
Author

I assume there's no interest. Just FYI, without these changes (which I have been using for more than a month) I'd have requested a refund, Hyper Backup performance was simply not acceptable.

@github-actions github-actions Bot removed the stalled This change is inactive or awaiting input label Jul 21, 2026
@larryrider

Copy link
Copy Markdown
Contributor

I assume there's no interest. Just FYI, without these changes (which I have been using for more than a month) I'd have requested a refund, Hyper Backup performance was simply not acceptable.

Hey @taglia,

First, I want to apologize for the long silence on this PR. It definitely wasn't because we didn't care or weren't interested. We have to balance work on many different priorities, and unfortunately we can't always respond to every issue and PR as quickly as we'd like. That said, we do review everything that comes in, and your contribution certainly didn't go unnoticed.

The reason review stalled is that, in parallel, we have been landing a series of changes on main that overlap heavily with what this PR addresses:

  • WebDAV config caching (ConfigService)
  • Caching + improved XML generation in the PROPFIND handler
  • Native LOCK/UNLOCK handler support (which Hyper Backup relies on for its locking behavior)
  • Accept-Ranges/Content-Type header fixes and a file-size NaN fix in the WebDAV handlers
  • Adding a keep-alive tuning in the Docker entrypoint, which we believe was one of the main causes of the session drops that were hurting your backup throughput

Because several of those commits touch the exact same files your PR modifies (docker/entrypoint.sh, GET.handler.ts, PROPFIND.handler.ts), the branch now shows conflicts against main. That's the practical reason it's stuck, not a rejection of the idea.

Since a good part of the caching/locking/keep-alive work is now already in main, unconditionally, without needing the WEBDAV_HYPER_BACKUP_MODE flag, could you rebase on top of current main and see what's actually left to add?

One more thing worth discussing before we merge anything further: part of the remaining diff (the lock_keep_alive.@writer_version_* filtering under /Control/lock/, and the zero-byte .bucket/.index "placeholder" detection under /Pool/...) works by pattern-matching Synology's internal file and path naming conventions on the server side. I'd rather we not take that on permanently, as WebDAV is meant to stay a client-agnostic protocol, and baking in assumptions about one vendor's internal implementation means we silently break if Synology changes those conventions in a future DSM release, with no protocol-level way to detect it. It's also a real correctness risk for a backup tool specifically: if that heuristic ever misclassifies a genuine data write as a "placeholder", that's silent data loss that only surfaces at restore time. On top of that, since we now support real WebDAV LOCK/UNLOCK natively, Hyper Backup's own lock-file workaround may not even be necessary anymore, so its worth testing before keeping that logic at all.

So, from your remaining delta, I'd suggest we keep the generic, protocol-agnostic pieces (fast-path caching, placeholder handling that isn't tied to Synology-specific paths, diagnostic timing logs, Docker startup tweaks) and drop or heavily reconsider the parts that pattern-match Synology's internal file naming, those are better handled on Synology's side, or at minimum kept far more narrowly scoped and only if we confirm native LOCK/UNLOCK doesn't already cover it.

If you can rebase and split the PR along those lines, I'll get it reviewed promptly. And if your before/after numbers still show a meaningful improvement from the generic caching work alone, that's genuinely useful data for deciding whether it should become the default.

Sorry again for how this played out, and thank you so much for sticking with it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants