Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ import { Callout } from 'fumadocs-ui/components/callout'

## File Storage

By default Sim writes uploads to local disk. For production, point it at AWS S3 or Azure Blob. See [Object Storage](/platform/self-hosting/object-storage) for the full setup, bucket layout, and IAM policy.
By default Sim writes uploads to local disk. For production, point it at AWS S3, Azure Blob, or Google Cloud Storage. See [Object Storage](/platform/self-hosting/object-storage) for the full setup, bucket layout, and IAM policy.

| Variable | Description |
|----------|-------------|
Expand All @@ -82,6 +82,9 @@ By default Sim writes uploads to local disk. For production, point it at AWS S3
| `S3_BUCKET_NAME` | General workspace files bucket — set with `AWS_REGION` to enable S3 |
| `AZURE_STORAGE_CONTAINER_NAME` | General files container — set with Azure credentials to enable Blob (takes precedence over S3) |
| `AZURE_CONNECTION_STRING` | Azure connection string, or use `AZURE_ACCOUNT_NAME` + `AZURE_ACCOUNT_KEY` |
| `GCS_BUCKET_NAME` | General workspace files bucket — enables GCS when neither Azure Blob nor S3 is configured |
| `GCS_PROJECT_ID` | GCP project ID. Omit to infer from credentials/ADC |
| `GCS_CREDENTIALS_JSON` | Inline service-account JSON. Omit to use Application Default Credentials (Workload Identity, `GOOGLE_APPLICATION_CREDENTIALS`) |

## Email Providers

Expand Down
154 changes: 147 additions & 7 deletions apps/docs/content/docs/en/platform/self-hosting/object-storage.mdx
Original file line number Diff line number Diff line change
@@ -1,23 +1,24 @@
---
title: Object Storage
description: Configure where Sim stores uploaded files — local disk, AWS S3, or Azure Blob
description: Configure where Sim stores uploaded files — local disk, AWS S3, Azure Blob, or Google Cloud Storage
---

import { Tab, Tabs } from 'fumadocs-ui/components/tabs'
import { Callout } from 'fumadocs-ui/components/callout'
import { Step, Steps } from 'fumadocs-ui/components/steps'
import { FAQ } from '@/components/ui/faq'

Sim stores every uploaded file — knowledge base documents, chat attachments, execution outputs, profile pictures, and more — in object storage. Three backends are supported:
Sim stores every uploaded file — knowledge base documents, chat attachments, execution outputs, profile pictures, and more — in object storage. Four backends are supported:

| Backend | When to use |
|---------|-------------|
| **Local disk** | Single-node Docker, local development, evaluation |
| **[AWS S3](https://aws.amazon.com/s3/)** | Production, especially when running more than one app replica |
| **[Azure Blob](https://learn.microsoft.com/azure/storage/blobs/)** | Production on Azure |
| **[Google Cloud Storage](https://cloud.google.com/storage)** | Production on GCP |

<Callout type="warning">
Local disk writes to the container's `/uploads` directory. Files are lost when the container is recreated unless that path is on a persistent volume, and they are **not** shared across replicas. For any multi-replica or production deployment, use S3 or Azure Blob.
Local disk writes to the container's `/uploads` directory. Files are lost when the container is recreated unless that path is on a persistent volume, and they are **not** shared across replicas. For any multi-replica or production deployment, use S3, Azure Blob, or Google Cloud Storage.
</Callout>

## How the backend is selected
Expand All @@ -26,9 +27,10 @@ Sim picks the backend automatically from environment variables — there is no e

1. **Azure Blob** — used if `AZURE_STORAGE_CONTAINER_NAME` is set **and** either (`AZURE_ACCOUNT_NAME` + `AZURE_ACCOUNT_KEY`) or `AZURE_CONNECTION_STRING` is set.
2. **AWS S3** — used if `S3_BUCKET_NAME` **and** `AWS_REGION` are set (and Azure is not configured).
3. **Local disk** — the fallback when neither is configured.
3. **Google Cloud Storage** — used if `GCS_BUCKET_NAME` is set (and neither Azure nor S3 is configured).
4. **Local disk** — the fallback when none is configured.

If both Azure and S3 are configured, **Azure wins**. Set only the variables for the backend you intend to use.
If more than one backend is configured, the first match in that order wins. Set only the variables for the backend you intend to use.

## Set up AWS S3

Expand Down Expand Up @@ -201,6 +203,143 @@ AZURE_STORAGE_WORKSPACE_LOGOS_CONTAINER_NAME=workspace-logos

A full Helm example lives at `helm/sim/examples/values-azure.yaml`.

## Set up Google Cloud Storage

<Steps>

<Step>

### Create the buckets

GCS uses one bucket per purpose, mirroring the S3 layout:

```bash
export PROJECT_ID=your-project-id
export LOCATION=us-central1

# Create buckets (names must be globally unique — prefix with your org)
for name in workspace-files knowledge-base execution-files chat-files \
copilot-files profile-pictures og-images workspace-logos; do
gcloud storage buckets create "gs://myorg-sim-$name" \
--project "$PROJECT_ID" \
--location "$LOCATION" \
--uniform-bucket-level-access
done
```

Keep all buckets **private** (no `allUsers` bindings). Sim serves files through short-lived V4 signed URLs, so the buckets never need public read access.

Because uploads are sent **directly from the browser** via signed `PUT` requests, each bucket needs a CORS policy that allows your Sim origin:

```bash
cat > /tmp/cors.json <<'EOF'
[
{
"origin": ["https://your-sim-domain.com"],
"method": ["GET", "PUT"],
"responseHeader": [
"Content-Type",
"ETag",
"x-goog-meta-originalname",
"x-goog-meta-uploadedat",
"x-goog-meta-purpose",
"x-goog-meta-userid",
"x-goog-meta-workspaceid",
"x-goog-meta-workflowid",
"x-goog-meta-executionid"
],
"maxAgeSeconds": 3600
}
]
EOF

for name in workspace-files knowledge-base execution-files chat-files \
copilot-files profile-pictures og-images workspace-logos; do
gcloud storage buckets update "gs://myorg-sim-$name" --cors-file=/tmp/cors.json
done
```

<Callout type="info">
Header names must be listed individually — GCS CORS matches `responseHeader` entries exactly and does not support wildcards like `x-goog-meta-*`. `ETag` is required because large-file multipart uploads read each part's `ETag` from the browser, and CORS hides the header otherwise.
</Callout>

</Step>

<Step>

### Grant access

Create a service account (or reuse the one your workload runs as) and grant it object access on the buckets:

```bash
gcloud iam service-accounts create sim-storage --project "$PROJECT_ID"

for name in workspace-files knowledge-base execution-files chat-files \
copilot-files profile-pictures og-images workspace-logos; do
gcloud storage buckets add-iam-policy-binding "gs://myorg-sim-$name" \
--member "serviceAccount:sim-storage@$PROJECT_ID.iam.gserviceaccount.com" \
--role roles/storage.objectAdmin
done
```

You then have two ways to supply credentials:

- **Application Default Credentials (recommended on GCP)** — run Sim with the service account via GKE Workload Identity (or attach it to the GCE instance) and leave `GCS_CREDENTIALS_JSON` unset. Because there is no private key in this mode, generating signed URLs uses the IAM `signBlob` API — grant the service account `roles/iam.serviceAccountTokenCreator` **on itself**:

```bash
gcloud iam service-accounts add-iam-policy-binding \
"sim-storage@$PROJECT_ID.iam.gserviceaccount.com" \
--member "serviceAccount:sim-storage@$PROJECT_ID.iam.gserviceaccount.com" \
--role roles/iam.serviceAccountTokenCreator
```

- **Inline key (for Docker Compose or non-GCP hosts)** — create a JSON key for the service account and set `GCS_CREDENTIALS_JSON` to its contents. With a private key present, signed URLs are generated locally and no extra IAM role is needed.

</Step>

<Step>

### Configure environment variables

```bash
# Credentials — omit both when using Workload Identity / ADC
GCS_PROJECT_ID=your-project-id # optional; inferred from credentials when unset
GCS_CREDENTIALS_JSON='{"type":"service_account","client_email":"...","private_key":"..."}'

# Buckets (per purpose)
GCS_BUCKET_NAME=myorg-sim-workspace-files
GCS_KB_BUCKET_NAME=myorg-sim-knowledge-base
GCS_EXECUTION_FILES_BUCKET_NAME=myorg-sim-execution-files
GCS_CHAT_BUCKET_NAME=myorg-sim-chat-files
GCS_COPILOT_BUCKET_NAME=myorg-sim-copilot-files
GCS_PROFILE_PICTURES_BUCKET_NAME=myorg-sim-profile-pictures
GCS_OG_IMAGES_BUCKET_NAME=myorg-sim-og-images
GCS_WORKSPACE_LOGOS_BUCKET_NAME=myorg-sim-workspace-logos
```

Only `GCS_BUCKET_NAME` is strictly required to switch Sim into GCS mode. Every purpose-specific bucket falls back to the general bucket when unset — add the others so each file type lands in its own bucket.

</Step>

</Steps>

### GCS bucket reference

| Variable | Stores | Required |
|----------|--------|----------|
| `GCS_BUCKET_NAME` | General workspace files | **Yes** (enables GCS) |
| `GCS_PROJECT_ID` | GCP project ID | No (inferred from credentials/ADC) |
| `GCS_CREDENTIALS_JSON` | Inline service-account JSON | No (uses Application Default Credentials if unset) |
| `GCS_KB_BUCKET_NAME` | Knowledge base documents | Recommended (falls back to `GCS_BUCKET_NAME`) |
| `GCS_EXECUTION_FILES_BUCKET_NAME` | Workflow execution files | Recommended (falls back to `GCS_BUCKET_NAME`) |
| `GCS_CHAT_BUCKET_NAME` | Deployed chat assets | Recommended (falls back to `GCS_BUCKET_NAME`) |
| `GCS_COPILOT_BUCKET_NAME` | Copilot attachments | Recommended (falls back to `GCS_BUCKET_NAME`) |
| `GCS_PROFILE_PICTURES_BUCKET_NAME` | User avatars | Recommended (falls back to `GCS_BUCKET_NAME`) |
| `GCS_OG_IMAGES_BUCKET_NAME` | OpenGraph preview images (falls back to `GCS_BUCKET_NAME`) | Optional |
| `GCS_WORKSPACE_LOGOS_BUCKET_NAME` | Workspace logos (falls back to `GCS_BUCKET_NAME`) | Optional |

A full Helm example (Workload Identity, GKE) lives at `helm/sim/examples/values-gcp.yaml`.

## Set up an S3-compatible provider (R2, MinIO, B2)

Sim works with any S3-compatible store by pointing the S3 client at a custom endpoint. Configure it exactly like AWS S3 (buckets, access key, secret), then add `S3_ENDPOINT` — and `S3_FORCE_PATH_STYLE` where the provider requires path-style addressing. Verified with [Cloudflare R2](https://developers.cloudflare.com/r2/), [MinIO](https://min.io/), [Backblaze B2](https://www.backblaze.com/cloud-storage), and [RustFS](https://rustfs.com/).
Expand Down Expand Up @@ -280,10 +419,11 @@ After restarting with the new configuration:
If uploads fail, check the app logs for credential or permission errors (see [Troubleshooting](/platform/self-hosting/troubleshooting)).

<FAQ items={[
{ question: "What happens if I do not configure any storage variables?", answer: "Sim falls back to local disk, writing files to the /uploads directory inside the app container. This is fine for evaluation but not durable across container recreation and not shared across replicas — use S3 or Azure Blob for production." },
{ question: "What happens if I do not configure any storage variables?", answer: "Sim falls back to local disk, writing files to the /uploads directory inside the app container. This is fine for evaluation but not durable across container recreation and not shared across replicas — use S3, Azure Blob, or Google Cloud Storage for production." },
{ question: "Do I have to create all eight S3 buckets?", answer: "No. Only AWS_REGION and S3_BUCKET_NAME are required to enable S3 mode. The purpose-specific buckets are recommended so each file type is isolated; og-images and workspace-logos fall back to the general bucket if their variables are unset." },
{ question: "How do I avoid storing AWS keys in plaintext?", answer: "On EC2/ECS/EKS, attach the IAM policy to the instance role, task role, or IRSA service-account role and leave AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY unset. Sim resolves credentials through the default AWS SDK provider chain automatically." },
{ question: "Can I use both S3 and Azure Blob at the same time?", answer: "No. Sim selects a single backend. If both are configured, Azure Blob takes precedence. Set only the variables for the backend you want." },
{ question: "Can I configure more than one cloud backend at the same time?", answer: "No. Sim selects a single backend, in the order Azure Blob, then S3, then Google Cloud Storage. Set only the variables for the backend you want." },
{ question: "How do I use GCS without storing a service-account key?", answer: "Run Sim with GKE Workload Identity (or an attached GCE service account) and leave GCS_CREDENTIALS_JSON unset — credentials resolve through Application Default Credentials. Grant the service account roles/iam.serviceAccountTokenCreator on itself so signed URLs can be generated via the IAM signBlob API." },
{ question: "Are the buckets exposed publicly?", answer: "No, and they should not be. Keep them private with public access blocked. Sim serves files to users through short-lived presigned URLs, so the buckets never need public read permissions." },
{ question: "Can I use MinIO or Cloudflare R2?", answer: "Yes. Configure it like AWS S3, then set S3_ENDPOINT to your provider's endpoint. For R2, set AWS_REGION=auto and leave S3_FORCE_PATH_STYLE unset. For MinIO/Ceph, set S3_FORCE_PATH_STYLE=true. See the S3-compatible provider section above." },
]} />
16 changes: 14 additions & 2 deletions apps/sim/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ API_ENCRYPTION_KEY=your_api_encryption_key # Use `openssl rand -hex 32` to gener
# CONTEXT_DEV_API_KEY_1= # Context.dev API key #1
# CONTEXT_DEV_API_KEY_2= # Context.dev API key #2

# File Storage (Optional - defaults to local disk; use S3 or Azure Blob for production)
# File Storage (Optional - defaults to local disk; use S3, Azure Blob, or Google Cloud Storage for production)
# AWS_REGION=us-east-1 # Required with S3_BUCKET_NAME to enable S3. Use "auto" for Cloudflare R2
# AWS_ACCESS_KEY_ID= # Omit to use the instance/IRSA credential chain
# AWS_SECRET_ACCESS_KEY= # Omit to use the instance/IRSA credential chain
Expand All @@ -99,7 +99,7 @@ API_ENCRYPTION_KEY=your_api_encryption_key # Use `openssl rand -hex 32` to gener
# Instagram OAuth (Optional - Instagram App ID/Secret from Meta App Dashboard > Instagram > API setup with Instagram login)
# INSTAGRAM_CLIENT_ID=
# INSTAGRAM_CLIENT_SECRET=
# Instagram publish file uploads require S3 or Azure Blob above (Meta must fetch a public HTTPS URL).
# Instagram publish file uploads require cloud storage above — S3, Azure Blob, or GCS (Meta must fetch a public HTTPS URL).
# Gmail attachments do not need this.

# TikTok OAuth (Optional - Client Key/Secret from the TikTok for Developers app)
Expand All @@ -119,6 +119,18 @@ API_ENCRYPTION_KEY=your_api_encryption_key # Use `openssl rand -hex 32` to gener
# AZURE_STORAGE_OG_IMAGES_CONTAINER_NAME= # OpenGraph preview images (falls back to AZURE_STORAGE_CONTAINER_NAME)
# AZURE_STORAGE_WORKSPACE_LOGOS_CONTAINER_NAME= # Workspace logos (falls back to AZURE_STORAGE_CONTAINER_NAME)

# Google Cloud Storage (used when neither Azure Blob nor S3 is configured)
# GCS_PROJECT_ID= # GCP project ID (optional — inferred from credentials/ADC when unset)
# GCS_CREDENTIALS_JSON= # Inline service-account JSON. Omit to use Application Default Credentials (Workload Identity, GOOGLE_APPLICATION_CREDENTIALS)
# GCS_BUCKET_NAME= # General workspace files bucket (enables GCS; all other buckets fall back to it)
# GCS_KB_BUCKET_NAME= # Knowledge base documents
# GCS_EXECUTION_FILES_BUCKET_NAME= # Workflow execution files
# GCS_CHAT_BUCKET_NAME= # Deployed chat assets
# GCS_COPILOT_BUCKET_NAME= # Copilot attachments
# GCS_PROFILE_PICTURES_BUCKET_NAME= # User profile pictures
# GCS_OG_IMAGES_BUCKET_NAME= # OpenGraph preview images
# GCS_WORKSPACE_LOGOS_BUCKET_NAME= # Workspace logos

# Admin API (Optional - for self-hosted GitOps)
# ADMIN_API_KEY= # Use `openssl rand -hex 32` to generate. Enables admin API for workflow export/import.
# Usage: curl -H "x-admin-key: your_key" https://your-instance/api/v1/admin/workspaces
Expand Down
3 changes: 1 addition & 2 deletions apps/sim/app/api/files/authorization.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,7 @@ vi.mock('@/lib/uploads', () => ({
}))

vi.mock('@/lib/uploads/config', () => ({
BLOB_CHAT_CONFIG: {},
S3_CHAT_CONFIG: {},
getStorageConfig: vi.fn(() => ({})),
}))

vi.mock('@/lib/uploads/server/metadata', () => ({
Expand Down
22 changes: 2 additions & 20 deletions apps/sim/app/api/files/authorization.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ import { and, eq, isNull } from 'drizzle-orm'
import { NextResponse } from 'next/server'
import { getFileMetadata } from '@/lib/uploads'
import type { StorageContext } from '@/lib/uploads/config'
import { BLOB_CHAT_CONFIG, S3_CHAT_CONFIG } from '@/lib/uploads/config'
import type { StorageConfig } from '@/lib/uploads/core/storage-client'
import { getFileMetadataByKey } from '@/lib/uploads/server/metadata'
import { inferContextFromKey } from '@/lib/uploads/utils/file-utils'
Expand Down Expand Up @@ -781,23 +780,6 @@ export async function assertToolFileAccess(
* Get chat storage configuration based on current storage provider
*/
async function getChatStorageConfig(): Promise<StorageConfig> {
const { USE_S3_STORAGE, USE_BLOB_STORAGE } = await import('@/lib/uploads/config')

if (USE_BLOB_STORAGE) {
return {
containerName: BLOB_CHAT_CONFIG.containerName,
accountName: BLOB_CHAT_CONFIG.accountName,
accountKey: BLOB_CHAT_CONFIG.accountKey,
connectionString: BLOB_CHAT_CONFIG.connectionString,
}
}

if (USE_S3_STORAGE) {
return {
bucket: S3_CHAT_CONFIG.bucket,
region: S3_CHAT_CONFIG.region,
}
}

return {}
const { getStorageConfig } = await import('@/lib/uploads/config')
return getStorageConfig('chat')
}
4 changes: 2 additions & 2 deletions apps/sim/app/api/files/export/[id]/route.ts
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ import { extractEmbeddedImageIds } from '@/lib/copilot/tools/server/files/embedd
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
import { captureServerEvent } from '@/lib/posthog/server'
import type { StorageContext } from '@/lib/uploads/config'
import { USE_BLOB_STORAGE } from '@/lib/uploads/config'
import { getServeStoragePrefix } from '@/lib/uploads/config'
import { downloadFile } from '@/lib/uploads/core/storage-service'
import { getFileMetadataById } from '@/lib/uploads/server/metadata'
import { verifyFileAccess } from '@/app/api/files/authorization'
Expand Down Expand Up @@ -106,7 +106,7 @@ export const GET = withRouteHandler(
}

if (!isMarkdown(record.originalName, record.contentType)) {
const storagePrefix = USE_BLOB_STORAGE ? 'blob' : 's3'
const storagePrefix = getServeStoragePrefix()
const servePath = `/api/files/serve/${storagePrefix}/${encodeURIComponent(record.key)}`
auditExport('file', 0)
return NextResponse.redirect(new URL(servePath, request.url), { status: 302 })
Expand Down
Loading
Loading