fix: retry UploadPart on transient backend errors

[ServerSideHannes]

Problem

RGW can accept a part, commit to a 200, stream it, and only then put the failure in the body:

InternalError: The server did not respond in time.
	status code: 200, request id: , host id:

Three retry layers all miss that shape, because each decides from the HTTP status line — which already read 200:

layer	why it misses
rclone LowLevelRetries = 20	gates on status 429/500/503 (backend/s3/s3.go:1393)
rclone string fallback	matches only transport phrases ("use of closed network connection", "unexpected EOF reading trailer"), not InternalError
botocore max_attempts: 3	sees the same 200

s3proxy is the only layer that parses the body and can see the real error.

#133 added this retry to UploadPartCopy and #138 to CompleteMultipartUpload, but plain UploadPart was left bare — and it's the only one of the three a Scylla backup uses (267 PUT, 0 UploadPartCopy measured against the backup bucket). At 50MB rclone chunks a 6.2GB SSTable is ~762 internal PUTs, so one unretried transient failure loses the whole multi-GB file.

2026-07-28 prod: 10 of 13 racks failed, ~2.4TiB never uploaded, every failure on main.companies (47 of 48 large files).

Fix

_upload_part_with_retry, mirroring _complete_multipart_upload_with_retry from #138, wired into both call sites in upload_part.py:

• _stream_and_upload_framed
• _upload_internal_part_with_semaphore

Reuses the existing base.SOURCE_READ_ATTEMPTS / SOURCE_READ_BACKOFF_SEC / is_retryable_source_error machinery from #133, so there are no new tunables and InternalError is already in the retryable set.

Why a retry is safe here: at both sites the full ciphertext is still in memory when upload_part is called — the del ciphertext happens after. So a retry re-PUTs identical bytes to the same internal part number, no body re-read needed. Same idempotency argument #133 used for the copy path.

Verification

check	result
new tests	8 passed
full unit suite	713 passed, 0 failed
ruff check / format --check	clean
reverted-fix control	6 of 8 fail

That last row matters — the retry loop was neutered to confirm the tests actually catch the bug rather than passing vacuously.

Scope

This makes an individual PUT survive a transient upstream failure. It does not protect a PUT in flight when its pod is terminated — that trigger is KEDA scale-in cliffs (30→21→10 in two minutes on 07-28), addressed separately by capping the HPA scaleDown rate. With this retry a cut part is re-attempted and lands on a surviving pod, so the file completes instead of dying.