Skip to content

Commit b9e9c2a

Browse files
committed
feat(storage): native Google Cloud Storage support for self-hosting
Adds GCS as a third object-storage backend with full parity with S3 and Azure Blob: uploads, streaming downloads, deletes, head, V4 signed URLs (single + batch), and browser/server multipart uploads via the GCS XML API. Selection precedence is Azure Blob > S3 > GCS > local disk. - new provider client at lib/uploads/providers/gcs (cached singleton, ADC/Workload Identity or inline GCS_CREDENTIALS_JSON auth) - per-context GCS_*_BUCKET_NAME config wired through getStorageConfig - shared getServeStoragePrefix() replaces hardcoded blob/s3 serve paths - docs (object-storage, environment-variables), .env.example, helm values.yaml + values-gcp.yaml storage section
1 parent 9d5ed38 commit b9e9c2a

29 files changed

Lines changed: 1711 additions & 41 deletions

File tree

apps/docs/content/docs/en/platform/self-hosting/environment-variables.mdx

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ import { Callout } from 'fumadocs-ui/components/callout'
7272

7373
## File Storage
7474

75-
By default Sim writes uploads to local disk. For production, point it at AWS S3 or Azure Blob. See [Object Storage](/platform/self-hosting/object-storage) for the full setup, bucket layout, and IAM policy.
75+
By default Sim writes uploads to local disk. For production, point it at AWS S3, Azure Blob, or Google Cloud Storage. See [Object Storage](/platform/self-hosting/object-storage) for the full setup, bucket layout, and IAM policy.
7676

7777
| Variable | Description |
7878
|----------|-------------|
@@ -82,6 +82,9 @@ By default Sim writes uploads to local disk. For production, point it at AWS S3
8282
| `S3_BUCKET_NAME` | General workspace files bucket — set with `AWS_REGION` to enable S3 |
8383
| `AZURE_STORAGE_CONTAINER_NAME` | General files container — set with Azure credentials to enable Blob (takes precedence over S3) |
8484
| `AZURE_CONNECTION_STRING` | Azure connection string, or use `AZURE_ACCOUNT_NAME` + `AZURE_ACCOUNT_KEY` |
85+
| `GCS_BUCKET_NAME` | General workspace files bucket — enables GCS when neither Azure Blob nor S3 is configured |
86+
| `GCS_PROJECT_ID` | GCP project ID. Omit to infer from credentials/ADC |
87+
| `GCS_CREDENTIALS_JSON` | Inline service-account JSON. Omit to use Application Default Credentials (Workload Identity, `GOOGLE_APPLICATION_CREDENTIALS`) |
8588

8689
## Email Providers
8790

apps/docs/content/docs/en/platform/self-hosting/object-storage.mdx

Lines changed: 137 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,24 @@
11
---
22
title: Object Storage
3-
description: Configure where Sim stores uploaded files — local disk, AWS S3, or Azure Blob
3+
description: Configure where Sim stores uploaded files — local disk, AWS S3, Azure Blob, or Google Cloud Storage
44
---
55

66
import { Tab, Tabs } from 'fumadocs-ui/components/tabs'
77
import { Callout } from 'fumadocs-ui/components/callout'
88
import { Step, Steps } from 'fumadocs-ui/components/steps'
99
import { FAQ } from '@/components/ui/faq'
1010

11-
Sim stores every uploaded file — knowledge base documents, chat attachments, execution outputs, profile pictures, and more — in object storage. Three backends are supported:
11+
Sim stores every uploaded file — knowledge base documents, chat attachments, execution outputs, profile pictures, and more — in object storage. Four backends are supported:
1212

1313
| Backend | When to use |
1414
|---------|-------------|
1515
| **Local disk** | Single-node Docker, local development, evaluation |
1616
| **[AWS S3](https://aws.amazon.com/s3/)** | Production, especially when running more than one app replica |
1717
| **[Azure Blob](https://learn.microsoft.com/azure/storage/blobs/)** | Production on Azure |
18+
| **[Google Cloud Storage](https://cloud.google.com/storage)** | Production on GCP |
1819

1920
<Callout type="warning">
20-
Local disk writes to the container's `/uploads` directory. Files are lost when the container is recreated unless that path is on a persistent volume, and they are **not** shared across replicas. For any multi-replica or production deployment, use S3 or Azure Blob.
21+
Local disk writes to the container's `/uploads` directory. Files are lost when the container is recreated unless that path is on a persistent volume, and they are **not** shared across replicas. For any multi-replica or production deployment, use S3, Azure Blob, or Google Cloud Storage.
2122
</Callout>
2223

2324
## How the backend is selected
@@ -26,9 +27,10 @@ Sim picks the backend automatically from environment variables — there is no e
2627

2728
1. **Azure Blob** — used if `AZURE_STORAGE_CONTAINER_NAME` is set **and** either (`AZURE_ACCOUNT_NAME` + `AZURE_ACCOUNT_KEY`) or `AZURE_CONNECTION_STRING` is set.
2829
2. **AWS S3** — used if `S3_BUCKET_NAME` **and** `AWS_REGION` are set (and Azure is not configured).
29-
3. **Local disk** — the fallback when neither is configured.
30+
3. **Google Cloud Storage** — used if `GCS_BUCKET_NAME` is set (and neither Azure nor S3 is configured).
31+
4. **Local disk** — the fallback when none is configured.
3032

31-
If both Azure and S3 are configured, **Azure wins**. Set only the variables for the backend you intend to use.
33+
If more than one backend is configured, the first match in that order wins. Set only the variables for the backend you intend to use.
3234

3335
## Set up AWS S3
3436

@@ -201,6 +203,133 @@ AZURE_STORAGE_WORKSPACE_LOGOS_CONTAINER_NAME=workspace-logos
201203

202204
A full Helm example lives at `helm/sim/examples/values-azure.yaml`.
203205

206+
## Set up Google Cloud Storage
207+
208+
<Steps>
209+
210+
<Step>
211+
212+
### Create the buckets
213+
214+
GCS uses one bucket per purpose, mirroring the S3 layout:
215+
216+
```bash
217+
export PROJECT_ID=your-project-id
218+
export LOCATION=us-central1
219+
220+
# Create buckets (names must be globally unique — prefix with your org)
221+
for name in workspace-files knowledge-base execution-files chat-files \
222+
copilot-files profile-pictures og-images workspace-logos; do
223+
gcloud storage buckets create "gs://myorg-sim-$name" \
224+
--project "$PROJECT_ID" \
225+
--location "$LOCATION" \
226+
--uniform-bucket-level-access
227+
done
228+
```
229+
230+
Keep all buckets **private** (no `allUsers` bindings). Sim serves files through short-lived V4 signed URLs, so the buckets never need public read access.
231+
232+
Because uploads are sent **directly from the browser** via signed `PUT` requests, each bucket needs a CORS policy that allows your Sim origin:
233+
234+
```bash
235+
cat > /tmp/cors.json <<'EOF'
236+
[
237+
{
238+
"origin": ["https://your-sim-domain.com"],
239+
"method": ["GET", "PUT"],
240+
"responseHeader": ["Content-Type", "ETag", "x-goog-meta-*"],
241+
"maxAgeSeconds": 3600
242+
}
243+
]
244+
EOF
245+
246+
for name in workspace-files knowledge-base execution-files chat-files \
247+
copilot-files profile-pictures og-images workspace-logos; do
248+
gcloud storage buckets update "gs://myorg-sim-$name" --cors-file=/tmp/cors.json
249+
done
250+
```
251+
252+
<Callout type="info">
253+
`ETag` must be in `responseHeader` — large-file multipart uploads read each part's `ETag` from the browser, and CORS hides the header otherwise.
254+
</Callout>
255+
256+
</Step>
257+
258+
<Step>
259+
260+
### Grant access
261+
262+
Create a service account (or reuse the one your workload runs as) and grant it object access on the buckets:
263+
264+
```bash
265+
gcloud iam service-accounts create sim-storage --project "$PROJECT_ID"
266+
267+
for name in workspace-files knowledge-base execution-files chat-files \
268+
copilot-files profile-pictures og-images workspace-logos; do
269+
gcloud storage buckets add-iam-policy-binding "gs://myorg-sim-$name" \
270+
--member "serviceAccount:sim-storage@$PROJECT_ID.iam.gserviceaccount.com" \
271+
--role roles/storage.objectAdmin
272+
done
273+
```
274+
275+
You then have two ways to supply credentials:
276+
277+
- **Application Default Credentials (recommended on GCP)** — run Sim with the service account via GKE Workload Identity (or attach it to the GCE instance) and leave `GCS_CREDENTIALS_JSON` unset. Because there is no private key in this mode, generating signed URLs uses the IAM `signBlob` API — grant the service account `roles/iam.serviceAccountTokenCreator` **on itself**:
278+
279+
```bash
280+
gcloud iam service-accounts add-iam-policy-binding \
281+
"sim-storage@$PROJECT_ID.iam.gserviceaccount.com" \
282+
--member "serviceAccount:sim-storage@$PROJECT_ID.iam.gserviceaccount.com" \
283+
--role roles/iam.serviceAccountTokenCreator
284+
```
285+
286+
- **Inline key (for Docker Compose or non-GCP hosts)** — create a JSON key for the service account and set `GCS_CREDENTIALS_JSON` to its contents. With a private key present, signed URLs are generated locally and no extra IAM role is needed.
287+
288+
</Step>
289+
290+
<Step>
291+
292+
### Configure environment variables
293+
294+
```bash
295+
# Credentials — omit both when using Workload Identity / ADC
296+
GCS_PROJECT_ID=your-project-id # optional; inferred from credentials when unset
297+
GCS_CREDENTIALS_JSON='{"type":"service_account","client_email":"...","private_key":"..."}'
298+
299+
# Buckets (per purpose)
300+
GCS_BUCKET_NAME=myorg-sim-workspace-files
301+
GCS_KB_BUCKET_NAME=myorg-sim-knowledge-base
302+
GCS_EXECUTION_FILES_BUCKET_NAME=myorg-sim-execution-files
303+
GCS_CHAT_BUCKET_NAME=myorg-sim-chat-files
304+
GCS_COPILOT_BUCKET_NAME=myorg-sim-copilot-files
305+
GCS_PROFILE_PICTURES_BUCKET_NAME=myorg-sim-profile-pictures
306+
GCS_OG_IMAGES_BUCKET_NAME=myorg-sim-og-images
307+
GCS_WORKSPACE_LOGOS_BUCKET_NAME=myorg-sim-workspace-logos
308+
```
309+
310+
Only `GCS_BUCKET_NAME` is strictly required to switch Sim into GCS mode. Add the others so each file type lands in its own bucket.
311+
312+
</Step>
313+
314+
</Steps>
315+
316+
### GCS bucket reference
317+
318+
| Variable | Stores | Required |
319+
|----------|--------|----------|
320+
| `GCS_BUCKET_NAME` | General workspace files | **Yes** (enables GCS) |
321+
| `GCS_PROJECT_ID` | GCP project ID | No (inferred from credentials/ADC) |
322+
| `GCS_CREDENTIALS_JSON` | Inline service-account JSON | No (uses Application Default Credentials if unset) |
323+
| `GCS_KB_BUCKET_NAME` | Knowledge base documents | Recommended |
324+
| `GCS_EXECUTION_FILES_BUCKET_NAME` | Workflow execution files (default: `sim-execution-files`) | Recommended |
325+
| `GCS_CHAT_BUCKET_NAME` | Deployed chat assets | Recommended |
326+
| `GCS_COPILOT_BUCKET_NAME` | Copilot attachments | Recommended |
327+
| `GCS_PROFILE_PICTURES_BUCKET_NAME` | User avatars | Recommended |
328+
| `GCS_OG_IMAGES_BUCKET_NAME` | OpenGraph preview images (falls back to `GCS_BUCKET_NAME`) | Optional |
329+
| `GCS_WORKSPACE_LOGOS_BUCKET_NAME` | Workspace logos (falls back to `GCS_BUCKET_NAME`) | Optional |
330+
331+
A full Helm example (Workload Identity, GKE) lives at `helm/sim/examples/values-gcp.yaml`.
332+
204333
## Set up an S3-compatible provider (R2, MinIO, B2)
205334

206335
Sim works with any S3-compatible store by pointing the S3 client at a custom endpoint. Configure it exactly like AWS S3 (buckets, access key, secret), then add `S3_ENDPOINT` — and `S3_FORCE_PATH_STYLE` where the provider requires path-style addressing. Verified with [Cloudflare R2](https://developers.cloudflare.com/r2/), [MinIO](https://min.io/), [Backblaze B2](https://www.backblaze.com/cloud-storage), and [RustFS](https://rustfs.com/).
@@ -280,10 +409,11 @@ After restarting with the new configuration:
280409
If uploads fail, check the app logs for credential or permission errors (see [Troubleshooting](/platform/self-hosting/troubleshooting)).
281410

282411
<FAQ items={[
283-
{ question: "What happens if I do not configure any storage variables?", answer: "Sim falls back to local disk, writing files to the /uploads directory inside the app container. This is fine for evaluation but not durable across container recreation and not shared across replicas — use S3 or Azure Blob for production." },
412+
{ question: "What happens if I do not configure any storage variables?", answer: "Sim falls back to local disk, writing files to the /uploads directory inside the app container. This is fine for evaluation but not durable across container recreation and not shared across replicas — use S3, Azure Blob, or Google Cloud Storage for production." },
284413
{ question: "Do I have to create all eight S3 buckets?", answer: "No. Only AWS_REGION and S3_BUCKET_NAME are required to enable S3 mode. The purpose-specific buckets are recommended so each file type is isolated; og-images and workspace-logos fall back to the general bucket if their variables are unset." },
285414
{ question: "How do I avoid storing AWS keys in plaintext?", answer: "On EC2/ECS/EKS, attach the IAM policy to the instance role, task role, or IRSA service-account role and leave AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY unset. Sim resolves credentials through the default AWS SDK provider chain automatically." },
286-
{ question: "Can I use both S3 and Azure Blob at the same time?", answer: "No. Sim selects a single backend. If both are configured, Azure Blob takes precedence. Set only the variables for the backend you want." },
415+
{ question: "Can I configure more than one cloud backend at the same time?", answer: "No. Sim selects a single backend, in the order Azure Blob, then S3, then Google Cloud Storage. Set only the variables for the backend you want." },
416+
{ question: "How do I use GCS without storing a service-account key?", answer: "Run Sim with GKE Workload Identity (or an attached GCE service account) and leave GCS_CREDENTIALS_JSON unset — credentials resolve through Application Default Credentials. Grant the service account roles/iam.serviceAccountTokenCreator on itself so signed URLs can be generated via the IAM signBlob API." },
287417
{ question: "Are the buckets exposed publicly?", answer: "No, and they should not be. Keep them private with public access blocked. Sim serves files to users through short-lived presigned URLs, so the buckets never need public read permissions." },
288418
{ question: "Can I use MinIO or Cloudflare R2?", answer: "Yes. Configure it like AWS S3, then set S3_ENDPOINT to your provider's endpoint. For R2, set AWS_REGION=auto and leave S3_FORCE_PATH_STYLE unset. For MinIO/Ceph, set S3_FORCE_PATH_STYLE=true. See the S3-compatible provider section above." },
289419
]} />

apps/sim/.env.example

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ API_ENCRYPTION_KEY=your_api_encryption_key # Use `openssl rand -hex 32` to gener
8181
# CONTEXT_DEV_API_KEY_1= # Context.dev API key #1
8282
# CONTEXT_DEV_API_KEY_2= # Context.dev API key #2
8383

84-
# File Storage (Optional - defaults to local disk; use S3 or Azure Blob for production)
84+
# File Storage (Optional - defaults to local disk; use S3, Azure Blob, or Google Cloud Storage for production)
8585
# AWS_REGION=us-east-1 # Required with S3_BUCKET_NAME to enable S3. Use "auto" for Cloudflare R2
8686
# AWS_ACCESS_KEY_ID= # Omit to use the instance/IRSA credential chain
8787
# AWS_SECRET_ACCESS_KEY= # Omit to use the instance/IRSA credential chain
@@ -99,7 +99,7 @@ API_ENCRYPTION_KEY=your_api_encryption_key # Use `openssl rand -hex 32` to gener
9999
# Instagram OAuth (Optional - Instagram App ID/Secret from Meta App Dashboard > Instagram > API setup with Instagram login)
100100
# INSTAGRAM_CLIENT_ID=
101101
# INSTAGRAM_CLIENT_SECRET=
102-
# Instagram publish file uploads require S3 or Azure Blob above (Meta must fetch a public HTTPS URL).
102+
# Instagram publish file uploads require cloud storage above — S3, Azure Blob, or GCS (Meta must fetch a public HTTPS URL).
103103
# Gmail attachments do not need this.
104104

105105
# TikTok OAuth (Optional - Client Key/Secret from the TikTok for Developers app)
@@ -119,6 +119,18 @@ API_ENCRYPTION_KEY=your_api_encryption_key # Use `openssl rand -hex 32` to gener
119119
# AZURE_STORAGE_OG_IMAGES_CONTAINER_NAME= # OpenGraph preview images (falls back to AZURE_STORAGE_CONTAINER_NAME)
120120
# AZURE_STORAGE_WORKSPACE_LOGOS_CONTAINER_NAME= # Workspace logos (falls back to AZURE_STORAGE_CONTAINER_NAME)
121121

122+
# Google Cloud Storage (used when neither Azure Blob nor S3 is configured)
123+
# GCS_PROJECT_ID= # GCP project ID (optional — inferred from credentials/ADC when unset)
124+
# GCS_CREDENTIALS_JSON= # Inline service-account JSON. Omit to use Application Default Credentials (Workload Identity, GOOGLE_APPLICATION_CREDENTIALS)
125+
# GCS_BUCKET_NAME= # General workspace files bucket (enables GCS)
126+
# GCS_KB_BUCKET_NAME= # Knowledge base documents
127+
# GCS_EXECUTION_FILES_BUCKET_NAME= # Workflow execution files
128+
# GCS_CHAT_BUCKET_NAME= # Deployed chat assets
129+
# GCS_COPILOT_BUCKET_NAME= # Copilot attachments
130+
# GCS_PROFILE_PICTURES_BUCKET_NAME= # User profile pictures
131+
# GCS_OG_IMAGES_BUCKET_NAME= # OpenGraph preview images (falls back to GCS_BUCKET_NAME)
132+
# GCS_WORKSPACE_LOGOS_BUCKET_NAME= # Workspace logos (falls back to GCS_BUCKET_NAME)
133+
122134
# Admin API (Optional - for self-hosted GitOps)
123135
# ADMIN_API_KEY= # Use `openssl rand -hex 32` to generate. Enables admin API for workflow export/import.
124136
# Usage: curl -H "x-admin-key: your_key" https://your-instance/api/v1/admin/workspaces

apps/sim/app/api/files/authorization.ts

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -781,7 +781,7 @@ export async function assertToolFileAccess(
781781
* Get chat storage configuration based on current storage provider
782782
*/
783783
async function getChatStorageConfig(): Promise<StorageConfig> {
784-
const { USE_S3_STORAGE, USE_BLOB_STORAGE } = await import('@/lib/uploads/config')
784+
const { USE_S3_STORAGE, USE_BLOB_STORAGE, USE_GCS_STORAGE } = await import('@/lib/uploads/config')
785785

786786
if (USE_BLOB_STORAGE) {
787787
return {
@@ -799,5 +799,12 @@ async function getChatStorageConfig(): Promise<StorageConfig> {
799799
}
800800
}
801801

802+
if (USE_GCS_STORAGE) {
803+
const { GCS_CHAT_CONFIG } = await import('@/lib/uploads/config')
804+
return {
805+
bucket: GCS_CHAT_CONFIG.bucket,
806+
}
807+
}
808+
802809
return {}
803810
}

apps/sim/app/api/files/export/[id]/route.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ import { extractEmbeddedImageIds } from '@/lib/copilot/tools/server/files/embedd
1212
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
1313
import { captureServerEvent } from '@/lib/posthog/server'
1414
import type { StorageContext } from '@/lib/uploads/config'
15-
import { USE_BLOB_STORAGE } from '@/lib/uploads/config'
15+
import { getServeStoragePrefix } from '@/lib/uploads/config'
1616
import { downloadFile } from '@/lib/uploads/core/storage-service'
1717
import { getFileMetadataById } from '@/lib/uploads/server/metadata'
1818
import { verifyFileAccess } from '@/app/api/files/authorization'
@@ -106,7 +106,7 @@ export const GET = withRouteHandler(
106106
}
107107

108108
if (!isMarkdown(record.originalName, record.contentType)) {
109-
const storagePrefix = USE_BLOB_STORAGE ? 'blob' : 's3'
109+
const storagePrefix = getServeStoragePrefix()
110110
const servePath = `/api/files/serve/${storagePrefix}/${encodeURIComponent(record.key)}`
111111
auditExport('file', 0)
112112
return NextResponse.redirect(new URL(servePath, request.url), { status: 302 })

0 commit comments

Comments
 (0)