Skip to content

feat(storage): support S3-compatible endpoints (R2, MinIO, B2) for file storage#4865

Merged
waleedlatif1 merged 4 commits into
stagingfrom
waleedlatif1/s3-storage-setup-docs
Jun 3, 2026
Merged

feat(storage): support S3-compatible endpoints (R2, MinIO, B2) for file storage#4865
waleedlatif1 merged 4 commits into
stagingfrom
waleedlatif1/s3-storage-setup-docs

Conversation

@waleedlatif1
Copy link
Copy Markdown
Collaborator

Summary

  • Add S3_ENDPOINT and S3_FORCE_PATH_STYLE env vars so self-hosters can use Cloudflare R2, MinIO, Backblaze B2, or any S3-compatible store for file storage — previously only real AWS S3 worked
  • Wire both into the shared upload S3Client (getS3Client); every storage op (upload, download, presign, multipart) routes through it, so presigned URLs honor the custom endpoint automatically
  • Endpoint is treated as trusted operator config — passed through verbatim, no SSRF/HTTPS gate (so on-prem MinIO works); forcePathStyle defaults to false (AWS S3, R2) and is set true for MinIO/Ceph
  • Make the multipart Location fallback endpoint-aware (the only hardcoded AWS hostname in the upload path)
  • Document the new vars in Helm values.yaml + values-aws.yaml, .env.example, and the English self-hosting docs (new object-storage.mdx page), including the browser-reachability + CORS requirement for direct presigned uploads

Type of Change

  • New feature

Testing

  • Extended providers/s3/client.test.ts to assert the client is constructed with endpoint/forcePathStyle when set, and absent/false for plain AWS
  • bun run scripts/check-api-validation-contracts.ts passes (no route changes); biome clean
  • Manual R2/MinIO end-to-end verification still to be run in a full environment

Notes

  • Docs added in English only; other locales (es/fr/de/ja/zh) to be handled by the translation pipeline
  • Scoped to the self-hosted storage backend; the user-facing S3 connector block/tool is unchanged (separate follow-up)

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…le storage

Add S3_ENDPOINT and S3_FORCE_PATH_STYLE env vars, wired into the shared upload
S3 client so Cloudflare R2, MinIO, Backblaze B2, and other S3-compatible stores
work for self-hosted file storage. The endpoint is trusted operator config (no
SSRF/HTTPS gate). Makes the multipart Location fallback endpoint-aware, extends
the S3 client unit tests, and documents the new vars in Helm values, .env.example,
and the English self-hosting docs (incl. browser-reachability + CORS guidance).
@vercel
Copy link
Copy Markdown

vercel Bot commented Jun 3, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs Ready Ready Preview, Comment Jun 3, 2026 6:30pm

Request Review

@cursor
Copy link
Copy Markdown

cursor Bot commented Jun 3, 2026

PR Summary

Medium Risk
Changes how production uploads are routed (custom endpoints, path-style, presigned browser PUTs); misconfiguration can break uploads without affecting core auth or app logic.

Overview
Self-hosters can point file storage at S3-compatible backends (Cloudflare R2, MinIO, Backblaze B2, etc.), not only AWS S3, via new S3_ENDPOINT and S3_FORCE_PATH_STYLE settings.

The shared upload S3Client now passes those options through for uploads, downloads, presigning, and multipart flows. Multipart completion uses an endpoint-aware fallback URL when the SDK omits Location (replacing a hardcoded AWS hostname). S3_FORCE_PATH_STYLE is parsed with envBoolean and defaults to false for AWS/R2.

Documentation adds an English object-storage self-hosting page (backend selection, buckets/IAM, Docker/Helm, R2/MinIO/RustFS, browser CORS and endpoint reachability for direct presigned uploads), a File Storage section on the env-vars page, and updates to .env.example and Helm values.yaml / values-aws.yaml. Tests cover client construction with custom endpoint/path-style; the shared test env mock exports envBoolean.

Reviewed by Cursor Bugbot for commit f3c7b14. Configure here.

Comment thread apps/sim/lib/uploads/core/setup.server.ts
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Jun 3, 2026

Greptile Summary

This PR adds S3_ENDPOINT and S3_FORCE_PATH_STYLE environment variables so self-hosters can point the S3 client at any S3-compatible store (Cloudflare R2, MinIO, Backblaze B2) instead of only AWS S3.

  • Core change: getS3Client now passes endpoint and forcePathStyle from S3_CONFIG; buildObjectFallbackUrl replaces the hardcoded AWS hostname with addressing-mode-aware logic (virtual-hosted for R2, path-style for MinIO/Ceph).
  • Config/env: S3_ENDPOINT and S3_FORCE_PATH_STYLE are declared as z.string().optional() and coerced at the consumption site via envBoolean, consistent with every other boolean env var in the codebase.
  • Docs + Helm: A new object-storage.mdx doc page covers all three backends; Helm values.yaml and values-aws.yaml are updated with commented examples for each compatible provider.

Confidence Score: 5/5

Safe to merge — all S3 operations route through the updated client, and the addressing-mode branching is correct for each provider.

The implementation correctly threads endpoint and forcePathStyle from env through config to the SDK client, with proper empty-string handling in Helm. The buildObjectFallbackUrl branching is correct, the envBoolean mock exactly mirrors the production helper, and existing AWS S3 behavior is unchanged. The only finding is a test that pairs an R2 URL with forcePathStyle: true — an invalid provider combination — but this does not affect production code paths.

apps/sim/lib/uploads/providers/s3/client.test.ts — the new endpoint test uses an R2 URL with forcePathStyle: true, which is an invalid combination and reduces the test's value as a correctness check.

Important Files Changed

Filename Overview
apps/sim/lib/uploads/providers/s3/client.ts Adds endpoint and forcePathStyle to getS3Client and introduces buildObjectFallbackUrl with correct virtual-hosted vs path-style branching; logic is sound.
apps/sim/lib/uploads/providers/s3/client.test.ts New test for custom endpoint passes an R2 URL with forcePathStyle: true, which is an invalid combination (R2 requires virtual-hosted style); should use a MinIO URL for the path-style case.
apps/sim/lib/uploads/config.ts Correctly adds endpoint (empty-string-to-undefined via
apps/sim/lib/core/config/env.ts Declares S3_ENDPOINT and S3_FORCE_PATH_STYLE as z.string().optional(), consistent with every other env var; envBoolean handles coercion at consumption site.
packages/testing/src/mocks/env.mock.ts Adds envBoolean to the mock with logic identical to the production implementation — no drift introduced.
apps/sim/lib/uploads/core/setup.server.ts Adds startup log line when S3_ENDPOINT is set; correctly guards on env.S3_ENDPOINT (truthy only for non-empty strings).
apps/docs/content/docs/en/self-hosting/object-storage.mdx New doc page covering all three backends (local, S3, Azure) plus R2/MinIO/RustFS; correct addressing-mode guidance for each provider.
helm/sim/values.yaml Adds S3_ENDPOINT and S3_FORCE_PATH_STYLE as empty-string defaults; comments are accurate.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[S3_ENDPOINT set?] -->|No| B[AWS S3
virtual-hosted, region-derived host]
    A -->|Yes| C[S3_FORCE_PATH_STYLE?]
    C -->|true| D[MinIO / Ceph RGW
path-style: endpoint/bucket/key]
    C -->|false| E[R2 / other
virtual-hosted: bucket.endpoint/key]

    subgraph getS3Client
        F[new S3Client
region + endpoint + forcePathStyle]
    end

    subgraph buildObjectFallbackUrl
        A
        B
        C
        D
        E
    end

    B --> F
    D --> F
    E --> F
Loading

Reviews (2): Last reviewed commit: "fix(storage): address review feedback an..." | Re-trigger Greptile

Comment thread apps/sim/lib/uploads/providers/s3/client.ts
Comment thread apps/sim/lib/core/config/env.ts Outdated
- Add envBoolean to the shared env test mock (createEnvMock) so config.ts's
  forcePathStyle coercion resolves — fixes failing knowledge/utils.test.ts
- Declare S3_FORCE_PATH_STYLE as z.string() (every other env var's pattern);
  it's coerced via envBoolean at the consumption site, avoiding a boolean
  type that never matches the string process.env value
- Log path-style from S3_CONFIG.forcePathStyle (envBoolean) instead of a
  separate isTruthy call, so the startup log can't disagree with the client
- Make buildObjectFallbackUrl honor forcePathStyle: virtual-hosted-style URL
  (bucket as subdomain) for R2, path-style only when forcePathStyle is set
@waleedlatif1
Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1
Copy link
Copy Markdown
Collaborator Author

@cursor review

Copy link
Copy Markdown

@cursor cursor Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit f3c7b14. Configure here.

@waleedlatif1 waleedlatif1 merged commit 648a5a1 into staging Jun 3, 2026
13 checks passed
@waleedlatif1 waleedlatif1 deleted the waleedlatif1/s3-storage-setup-docs branch June 3, 2026 18:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant