feat(multipart): buffered (encrypted) multipart uploads + list/get fidelity fixes#52
Open
pszafarczyk wants to merge 9 commits into
Open
feat(multipart): buffered (encrypted) multipart uploads + list/get fidelity fixes#52pszafarczyk wants to merge 9 commits into
pszafarczyk wants to merge 9 commits into
Conversation
added 9 commits
June 24, 2026 15:53
Pull the envelope-crypto + upstream-PutObject sequence (DEK encrypt, KEK-version metadata tagging, client.PutObject, buffer release) out of object.put into a reusable encryptAndUpload(ctx, log) method. put keeps the detached-context setup, error-code mapping, response headers, and 200 write. No behaviour change. Prepares a single shared crypto path for buffered-multipart CompleteMultipartUpload.
Manager that buffers each part to local disk and assembles them into a single plaintext buffer for the existing encrypt→PutObject path. Not yet wired into the router or main. - Create/WritePart/Assemble/Abort over a baseDir + in-memory session table - per-part MD5 hex ETags; running object-size cap enforced during the streamed copy (LimitReader, never trusts a client-supplied length) - Cleanup goroutine: TTL eviction + orphan-dir sweep (reclaims dirs left by a restart, since the session table is not persisted) - nil-safe Metrics interface; package does not import monitoring to keep the dependency direction clean (concrete impl lands in Phase 5) Single-instance / session-affinity component by design. Tests cover create, out-of-order parts, re-upload overwrite, cap enforcement, TTL eviction, orphan sweep, concurrency, and metrics.
Expose three S3PROXY_MULTIPART_* settings for the upcoming disk-buffer
multipart mode; inert until the handlers/main wiring read them.
- MultipartBufferDir <- S3PROXY_MULTIPART_BUFFER_DIR ("" = disabled)
- MultipartMaxSize <- S3PROXY_MULTIPART_MAX_SIZE (default/clamp 5 GiB)
- MultipartTTL <- S3PROXY_MULTIPART_TTL (default 24h)
Includes matching GetX shims and config_test.go for defaults/clamping.
Replace the multipart 501 stubs with buffer-mode handlers when a *multipart.Manager is wired. Create captures object metadata and opens a disk-backed session; UploadPart de-frames aws-chunked bodies and streams each part to disk; Complete assembles the parts, reuses encryptAndUpload to store one AES-256-GCM-SIV PutObject upstream, then drops the buffer; Abort releases the session. getMultipartHandler now follows forward -> buffer -> block precedence. Serve exempts UploadPart from the 256 MiB PutObject body cap while keeping the 5 GiB ContentLength ceiling. New aws-chunked de-framer and S3 multipart XML structs back the handlers. Complete responds 200 only after upstream success and keeps the session on failure so the client can retry, matching the single-shot ack order.
Wire the disk-buffer multipart Manager into main: when S3PROXY_MULTIPART_BUFFER_DIR is set, build the Manager, start its Cleanup goroutine, and pass it to router.New (was nil). Error if --allow-multipart is also set (mutually exclusive); log the single-instance / session-affinity constraint at startup. Add multipart Prometheus collectors (uploads_active, parts_total, buffer_bytes, completed_total, aborted_total, assemble_duration) and make *monitoring.Metrics satisfy the multipart.Metrics interface so the Manager updates them. Increment completed/aborted from the Complete/Abort handlers via nil-safe router helpers.
…e 6) Operational completeness for buffered multipart uploads. - dashboards: add "Multipart (buffer mode)" row (active uploads, buffer disk bytes, parts/completed/aborted throughput, assemble p95) to both the monitoring/ source and the deployed chart copy - alerts: S3ProxyMultipartBufferHigh (disk exhaustion) and S3ProxyMultipartUploadsStuck (orphaned/affinity-misrouted sessions), tunable via values.yaml - README: new env vars, metrics, alerts, and a Multipart uploads section (forward vs buffer mode, single-instance/affinity, size/memory cap, durability ordering, ETag semantics, ListParts unsupported) - CHANGELOG: Unreleased entry - e2e: Create -> 3x UploadPart -> Complete through the proxy against MinIO; asserts single ciphertext object + DEK tag, byte-equal GET, buffer-dir reclamation - fix: existing proxy_e2e_test.go used the pre-Phase-5 router.New signature (5 args); add the missing nil manager arg so the e2e build compiles
Intercepted GetObject responses streamed the decrypted body without a Content-Length header, so Go's server fell back to chunked transfer encoding. s3cmd 2.4.0 reads content-length unconditionally and downloaded an empty file as a result. The full plaintext is already buffered before any bytes are written, so set Content-Length from len(plaintext) before WriteHeader. The upstream length describes the ciphertext (+28 bytes GCM overhead, or arbitrary for legacy objects) and can't be reused, mirroring the existing omission of x-amz-checksum-* headers.
Intercepted GETs decrypt the body but returned the upstream ETag, which S3 computes over the ciphertext at rest. Clients/SDKs that validate the body against the ETag, or cache by it, saw a mismatch. Override the response ETag with md5(plaintext) — the ETag S3 would have produced for the unencrypted object — computed on the fly from the buffer already in memory, so no stored metadata or migration is needed. Pass-through objects (no DEK tag) keep the upstream ETag. PUT-response and HEAD ETags still reflect the ciphertext and remain a known gap.
ListObjects/ListObjectsV2 previously reported the at-rest ciphertext size (plaintext + 28-byte AES-GCM-SIV envelope). Intercept bucket-level list requests, subtract the fixed encryption overhead from every <Size>, and clamp at 0. Bucket sub-resource GETs (acl, versioning, multipart listings, ?versions, …) are still forwarded unchanged. - cryptoutil: export EncryptionOverhead constant + invariant test - router: bucketOnlyPattern + isListObjects, wired into getHandler - handler: extract forwardUpstream; add handleListObjects (buffer, rewrite on 2xx, fix Content-Length) - listsize: byte-preserving <Size> regex rewrite + unit tests - e2e: assert v1 and v2 listings report plaintext size Known limitation: objects not written through the proxy (legacy plaintext, server-side copies, multipart) are reported 28 bytes short / 0 after clamp, since list responses carry no per-object encryption metadata.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in buffered multipart upload mode that keeps data at rest encrypted (the existing --allow-multipart
forwards plaintext upstream), plus a set of fixes that make intercepted reads and listings report the plaintext view
clients expect.
What changed
Buffered multipart uploads (opt-in via S3PROXY_MULTIPART_BUFFER_DIR)
CompleteMultipartUpload concatenates the parts, encrypts the assembled object through the existing AES-256-GCM
envelope path, and stores it upstream as a single ciphertext PutObject.
S3PROXY_MULTIPART_TTL (default 24h) and reclaims orphaned directories after a restart.
default/hard cap 5 GiB), S3PROXY_MULTIPART_TTL. Mutually exclusive with --allow-multipart.
_assemble_duration_seconds), a Grafana dashboard row, and two PrometheusRule alerts (S3ProxyMultipartBufferHigh,
S3ProxyMultipartUploadsStuck).
ListObjects/ListObjectsV2 report decrypted plaintext size
each , clamped at 0. Bucket sub-resource GETs (acl, versioning, multipart listings, ?versions, …) are still
forwarded unchanged.
Fixes
without a length, triggering chunked encoding).
clients match the delivered bytes. Pass-through objects keep the upstream ETag.
Constraints / known limitations
fit in RAM (~2× peak); ListParts is unsupported; Complete acks only after the upstream store succeeds.
server-side copies, multipart), since list responses carry no per-object encryption metadata.
Testing