Non-standard behavior of WebDAV server; status code 425 is misused

[cannorin]

Describe the bug

The WebDAV implementation of OpenCloud returns 425 if the file is uploaded but the metadata is not yet processed, but it is a misuse of HTTP status code. 425 is meant to be used to indicate that "the server was unwilling to risk processing a request that might be replayed to avoid potential replay attacks" (https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/425, emphasis by me), not that the requested resource is not yet ready.

Steps to reproduce

Upload a file and access it before it is processed.

Expected behavior

It should return 404, or 503 with a Retry-After header.

Actual behavior

It returns 425.

Additional context

• It is breaking WebDAB integration of several third-party applications e.g. Zotero and an unofficial sync plugin for Obsidian.
• Related: Download fails with HTTP 425 (Too Early) after upload; no retry android#92

[butonic]

I agree that 425, which is defined in https://datatracker.ietf.org/doc/html/rfc8470 is very narrowly defined for TCP Handshakes ... but in effect the 425 status tells clients to retry the request. Yes, we absolutely highjacked that status code and yes, I am not really happy with the situation as well.

I tried to dig up the initial spec, but I seems to be missing the reasoning for 425. There are some comments mentioning 423 or 403, but they are more related to our own sync client. The issue was synced from a jira we used back then ... note to self: NEVER do that again, as we can see discussion seems to have been lost ... 😞

IIRC opencloud clients will treat a 503 as the whole server being in maintenance, so that would halt the whole sync process. A 404 would make sync clients delete the file and 423 was not cleanly handled by the desktop back in the day.

AFAICT a 409 Conflict + Retry-After header is the closest thing to a webdavy like response telling the client that the state on the server is not ready ..., but before looking into that we would need to know how other webdav (sync) clients would deal with this error. I honestly doubt they would retry, because in WebDAV 409 is already used for semantic errors. Maybe build a PoC server based on the go stdlib webdav handler that sends a 409 + retry-after header for all files for the first 10 sec ... might be a middleware. then test how zotero, davfs2 and rclone handle it.

For what it's worth, RFC 4918 §9.7.1 explicitly mandates 409 for a PUT without an existing parent collection, and other WebDAV server implementations enforce this (see e.g. golang/go#67150 and sabre-io/dav#1304). That's more evidence that clients likely treat 409 as "don't retry, something's structurally wrong" rather than "retry me", which matches my skepticism above.

I also looked at 423 again since it's the "native" WebDAV code for "temporarily can't touch this". Problem is, a real WebDAV lock (RFC 4918) is a write protection mechanism, GET isn't affected by it per spec. So returning 423 for a plain GET is arguably an even bigger stretch than what we do with 425. And if we did implement it as an actual lock (the only way to use it "correctly"), it would also block a second upload to the same path without the lock token, which is explicitly not what we want, we don't want to prevent a retry/overwrite upload from starting while postprocessing of the previous one is still running. So 423 is out, for two independent reasons.

That being said the real problem is that our clients expect the file to be immediately present in the parent folders PROPFIND response when the Upload was finished. A correct implementation for async postprocessing would make clients use a dedicated endpoint for uploads and check the status of the upload, not the parent folders PROPFIND. However, there are two problems I vaguely remember ... they might not be true anymore:

1. the web ui makes a propfind when it has finished uploading a file. It might be confusing if end users do not see their file show up in the filelist when the upload finished but is in processing. Leads tu support requests.
2. the desktop client has to not only compare the local directoy with the servers directory, but also take into account the uploads he has started, which would touch the sync algorithm that used to be rock solid. Touching it is literally scary for most devs.

I was pondering this in owncloud/ocis#7246 already

Looking at how other vendors handle this, I see three different strategies:

• OneDrive/SharePoint (Graph API) avoids the whole class of problem: with createUploadSession (basically their tus equivalent), the item simply doesn't exist in the drive until the session is committed. From their docs: "once the last byte of the file has been uploaded the upload session is completed and the final file is shown in the destination folder." No half-visible state, so no GET-before-ready case can happen, so no status code debate needed.
• oenCloud 10 / Nextloud's files_antivirus goes synchronous instead: the file is scanned before it's written to storage, blocking the whole PUT if scanning fails or can't run. Slower uploads, but again no visible in-between state. We wanted to get rid of this sync blocking because we saw timeouts that increase support requests.
• Google Drive / YouTube video processing is the closest to what we do: item shows up immediately, but comes with an explicit machine-readable status field that purpose-built clients check before attempting playback. That's an application-level check before the request, not a status code on a failed request.

Given that, I think the actual fix is architectural, not "which status code": stop making the node visible in PROPFIND before it's actually ready (i.e. do what OneDrive does). That removes 425/409/423 as a question entirely, because a GET on something that isn't listed yet is a non-issue.

A solution might be to keep PROPFIND as plain, boring WebDAV (no early visibility, no special status codes), and solve "web UI shows that something's being uploaded here" as a completely separate concern via SSE. We can announce an upload presence: "someone is currently uploading into this folder" as a start/end signal per folder. Cheap, degrades gracefully (TTL/heartbeat based, so a crashed client doesn't leave a stale indicator hanging forever), and completely decoupled from whether PROPFIND shows the file early. We can do "hide until ready" for WebDAV and still give the web UI a "someone is uploading here" hint through a different channel entirely.

However, showing the file in the PROPFIND after the upload processing has finished is a problem for the web ui IIRC. It does a PROPFIND after it has has finished uploading the bytes. The desktop an mobile clients might do the same. In any case they all need to handle async uploads more deeply. Which again would leave standard webdav clients out of scope... Unless we do sync processing for them as a fallback. which we could du with a Prefer header ... that would allow our clients to send a Prefer: respond-async header to indicate they can handle 202 status code and will check the tus upload endpoint when making PUT requests... actually our clients already make a POST, but PUT with sync processing might work for other clients.

Given that, I'd treat this issue (which status code to use) as blocked on input from @kulmann @dragotin and someone that can explain what the mobile clients do, not something we can fix in isolation by picking a "nicer" code. Whatever replaces 425 has to keep answering the same question 425 was hacked in to answer (don't let the clients think the file is gone while it's still processing).

To be clear, this is a much bigger change than swapping a status code, it touches upload semantics, the web UI, and every client that syncs against us, with a real sequencing dependency in the middle. This won't happen in one PR, and I don't think anyone should try to make it one. Help figuring out the details, client-side sync state machine especially, since I can't judge the desktop/mobile code well enough, is very welcome.

[cannorin]

Oh, I didn't expect the issue to be this fundamental. I daily drive my own OpenCloud instance and the Zotero WebDAV integration, so I am willing to help testing it if you want (e.g. capturing the logs or trying a development build).

[cannorin]

An idea that came to my mind is to expose another WebDAV endpoint that works in a boring "compatibility mode". It would incur an additional maintenance cost, but it would not affect the related projects such as the web UI and the PC/mobile clients.

[butonic]

Currently, admins can turn off async uploads with OC_ASYNC_UPLOADS=false. That will make all postprocessing synchronous. That should fix old clients.

We could also default to sync uploads when simple PUT requests are made and use async processing when TUS uploads are made (unless the admin has explicitly set OC_ASYNC_UPLOADS=false). Another way would be to let clients decide how the upload should be processed with the Prefer: respond-async header. If it is present we can return a 202 Accepted Status with a Locaton header to the tus upload url. That endpoint allows clients to check if the upload is still in processing using the TUS HEAD /data/{uploadid} call. So they can get comfortable with TUS which has even become a ver active internet draft https://datatracker.ietf.org/doc/draft-ietf-httpbis-resumable-upload/ that apple uses to let apps hand off uploads to the os to so it can finish in the background.