perf(db): stop scanning every workspace file on workflow archive, cover the billing period aggregate (#6015)

* perf(files): stop loading every workspace file to archive a workflow's backing files

`cleanupWorkflowAliasBacking` runs on every workflow delete. It loaded every
`context='workspace'` row for the workspace — active and soft-deleted, all
columns — then discarded >99% of them in JS to find the handful of
`.changelogs/<workflowId>.md` and `.plans/<workflowId>/**` rows it owns.

In production this was the single worst query at 37.83% of total database
runtime: p50 6s, p95 9s, max 13.1s, 641 calls/day, 34.1M rows read. It is
index-served, so the cost is not a missing index — it is 59,061 rows and
~630MB of cold buffers read to archive a few files.

A file's `folderPath` is derived solely from its `folderId`, so matching on
folder membership is equivalent to the path comparison it replaces. The load
and JS filter become one targeted UPDATE keyed on a handful of folder ids.

Folders that are soft-deleted are still included when resolving which files a
workflow owns: path resolution ignores `deletedAt`, so a live file parented to
an archived folder previously matched and must continue to.

Also:
- Add `getWorkspaceShares`, replacing an id-list `IN` clause that grew with the
  file count (59,061 elements on the worst workspace) with one indexed lookup
  on `workspace_id`. Callers read the map by id, so a superset is equivalent.
- Drop `all` from the client-reachable file scope enum. It drops the
  `deleted_at` predicate and so cannot use the partial index serving the other
  two. No client requests it; server callers reach that scope directly.
- Set `fetch_types: false` on the app and realtime pools. postgres.js otherwise
  runs a blocking `pg_catalog.pg_type` roundtrip before each new connection's
  first query — 95,722 of them per day. It builds array parsers only; Drizzle
  already parses this schema's two `text[]` columns itself.

* perf(nav): add route loading boundaries and cover the billing period aggregate

Dynamic routes prefetch only down to the nearest loading boundary, and the
server stops prefetching at the first one. With no loading.tsx anywhere in the
workspace tree and a dynamic layout, Link prefetch was yielding almost nothing
and every navigation waited on a full server round trip with no feedback.

Add loading.tsx only where the fallback is provably what renders today, so
perceived speed improves without changing what users see:

- home, chat/[chatId]: reuse HomeFallback, already each page's own Suspense
  fallback.
- integrations, skills: reuse the tab-header chrome, byte-identical to each
  page's own Suspense fallback.

Deliberately not added to workspace root, settings, or w, whose pages are
redirect-only or already self-fallbacking — a boundary there would paint a
skeleton that does not match the destination.

Billing:

- Add usage_log_billing_period_cost_idx, trailing the remaining predicate
  columns and `cost` so the period aggregates resolve index-only. Confirmed
  against production: the aggregate currently runs as an Index Scan touching
  478,559 buffers because `cost` is absent from the existing index. That index
  is superseded but left in place; dropping it is a separate migration so a
  planner regression costs nothing to revert.
- Collapse the two aggregates behind /api/billing into one scan using
  SUM(...) FILTER. Besides halving the work on a nav-path query, it removes a
  latent inconsistency: as separate statements the two sums could observe
  different snapshots, making the copilot subset exceed the total.

Caching was considered and rejected for the usage aggregate. Its callers
include usage enforcement, threshold billing, and overage calculation, where a
stale-low read permits overspend and a stale-high read double-charges.

* test(files): cover cleanupWorkflowAliasBacking and correct a stale share mock

cleanupWorkflowAliasBacking had no test coverage, and the rewrite that replaced
its load-everything-then-filter body rests on a subtle equivalence: a file's
folderPath is derived solely from its folderId, and path resolution ignores
deletedAt. The second half is the easy part to get wrong — a live file parented
to an archived folder still resolves to a backing path and must still be
archived, so folders are filtered by deletedAt only when choosing which folders
to archive, never when deciding which files the workflow owns.

These tests pin that distinction: they fail if the archived-folder case is
dropped from file ownership, and separately assert that archived folders stay
out of the folder update and that unrelated workflows are never touched.

Also point the workspace files route test at getWorkspaceShares. Its mock still
named getSharesForResources, which the route no longer imports. The suite passed
regardless because all three cases exercise the upload path, so the GET listing
has no coverage — but the stale name would have handed the first GET test an
undefined function.

* fix(perf): drop the loading boundaries and correct the usage_log index order

Adversarial review found real regressions in both.

Route loading boundaries — all four removed:

The premise in their TSDoc was wrong. Each page's existing `<Suspense>` exists so
nuqs can prerender; `useSearchParams` only suspends during SSR, so on a client
navigation those fallbacks never painted. Hoisting them to loading.tsx did not
"show the same frame" — it made a previously invisible blank frame visible.

For chat, Next keys the Suspense boundary by cache key, so a chatId change mounts
a fresh suspended boundary and always commits its fallback. Switching chats would
have gone from "previous chat stays on screen" to a blank surface for the whole
RSC round trip, on the highest-frequency navigation in the product. Partial
prefetch only warms the fallback, so no latency was saved to offset it.

The integrations and skills fallbacks were correct for their own pages but also
became the fallback for four detail routes that render different chrome, inserting
a wrong intermediate frame. Fixing that needs per-child boundaries and new skeleton
UI that cannot be verified without a browser, for pages whose only server work is
`await params`. Not worth it.

Every remaining loading.tsx in this app paints real chrome. A blank one was against
the grain, and the measurable win was zero.

usage_log index:

Column order was wrong. The daily-refresh rollup filters entity type, id and
period_start but NOT period_end, so putting period_end fourth ended the usable
prefix at column three and left user_id and created_at as in-index filters rather
than scan boundaries — turning a ~2.2k-entry bitmap scan into a ~266k-entry scan.
Verified in production that period_start functionally determines period_end
(13,612 groups, zero with more than one end), so the slot bought no selectivity.
user_id and created_at now follow the shared prefix; period_end rides as payload.

Also corrected the claim that this supersedes usage_log_billing_entity_period_idx.
That index deduplicates to 17 MB across 1.42M entries because it has no
high-cardinality key column, which is what keeps prefix-only bitmap scans cheap.
It is retained deliberately, not pending a drop.

Harden cleanupWorkflowAliasBacking: gate the UPDATE on the ownership filter list
itself. `and()` and `or()` both drop undefined, so a clause that resolved to
nothing would have left a WHERE of workspace + context + not-deleted and archived
every file in the workspace. Tests now assert no UPDATE is issued when the
workflow owns nothing, plus the filename and context predicates.

Note fetch_types also disables array serializers, not just parsers; documented.

* docs(db): correct the fetch_types constraint note after differential testing

Ran both settings against a real Postgres with the schema's actual text[] columns.
Drizzle-typed selects and .returning() are byte-identical either way; only a raw
db.execute projecting an array column differs, yielding the wire form.

The previous note also claimed a raw JS-array bind fails because the serializers
come from the same catalog fetch. It does fail — but under both settings, because
Drizzle expands an array into a row constructor before postgres.js ever sees it.
That is unrelated to this flag, so the claim is removed rather than left implying
a constraint this change introduces.

* chore(db): generate the drizzle snapshot for the usage_log index migration

0271 was hand-written, so drizzle-kit's state never learned about the new index —
the next `generate` would have re-emitted it as a fresh migration against an
already-migrated database.

Ran `drizzle-kit generate` to produce the snapshot, then restored the
CONCURRENTLY form: drizzle emits a plain CREATE INDEX, which takes an ACCESS
EXCLUSIVE lock and would block writes on a 4M-row table for the duration of the
build. The generated column order matched the hand-written SQL exactly, which
also confirms schema.ts and the migration agree.

`generate` is now a no-op, and check:migrations still passes.

Author: waleedlatif1

apps/realtime/src/database/operations.ts

@@ -214,6 +214,8 @@ const socketDb = drizzle(
   instrumentPoolClient(
     postgres(connectionString, {
       prepare: false,
+      // See `packages/db/db.ts` — skips the per-connection pg_type roundtrip.
+      fetch_types: false,
       idle_timeout: 10,
       connect_timeout: 20,
       max: 10,

apps/sim/app/api/workspaces/[id]/files/route.test.ts

@@ -7,9 +7,9 @@ import { authMockFns, permissionsMock, permissionsMockFns, posthogServerMock } f
 import { NextRequest } from 'next/server'
 import { beforeEach, describe, expect, it, vi } from 'vitest'
 
-const { mockUploadWorkspaceFile, mockGetSharesForResources, mockRecordAudit } = vi.hoisted(() => ({
+const { mockUploadWorkspaceFile, mockGetWorkspaceShares, mockRecordAudit } = vi.hoisted(() => ({
   mockUploadWorkspaceFile: vi.fn(),
-  mockGetSharesForResources: vi.fn(),
+  mockGetWorkspaceShares: vi.fn(),
   mockRecordAudit: vi.fn(),
 }))
 
@@ -27,7 +27,7 @@ vi.mock('@/lib/uploads/shared/types', async (importOriginal) => {
 })
 
 vi.mock('@/lib/public-shares/share-manager', () => ({
-  getSharesForResources: mockGetSharesForResources,
+  getWorkspaceShares: mockGetWorkspaceShares,
 }))
 
 vi.mock('@/lib/posthog/server', () => posthogServerMock)
@@ -80,7 +80,7 @@ describe('workspace files upload route', () => {
     vi.clearAllMocks()
     authMockFns.mockGetSession.mockResolvedValue({ user: { id: 'user-1' } })
     permissionsMockFns.mockGetUserEntityPermissions.mockResolvedValue('write')
-    mockGetSharesForResources.mockResolvedValue(new Map())
+    mockGetWorkspaceShares.mockResolvedValue(new Map())
     mockUploadWorkspaceFile.mockResolvedValue({
       id: 'file-1',
       name: 'file.txt',

apps/sim/app/api/workspaces/[id]/files/route.ts

@@ -16,7 +16,7 @@ import {
 } from '@/lib/core/utils/stream-limits'
 import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
 import { captureServerEvent } from '@/lib/posthog/server'
-import { getSharesForResources } from '@/lib/public-shares/share-manager'
+import { getWorkspaceShares } from '@/lib/public-shares/share-manager'
 import {
   FileConflictError,
   listWorkspaceFiles,
@@ -74,10 +74,7 @@ export const GET = withRouteHandler(
 
       const files = await listWorkspaceFiles(workspaceId, { scope })
 
-      const shares = await getSharesForResources(
-        'file',
-        files.map((file) => file.id)
-      )
+      const shares = await getWorkspaceShares('file', workspaceId)
       const filesWithShares = files.map((file) => ({
         ...file,
         share: shares.get(file.id) ?? null,

apps/sim/hooks/queries/workspace-files.ts

@@ -27,7 +27,7 @@ import { useFileContentSource } from '@/hooks/use-file-content-source'
 
 const logger = createLogger('WorkspaceFilesQuery')
 
-type WorkspaceFileQueryScope = 'active' | 'archived' | 'all'
+type WorkspaceFileQueryScope = 'active' | 'archived'
 
 /**
  * Query key factories for workspace files

apps/sim/lib/api/contracts/workspace-files.ts

@@ -3,7 +3,13 @@ import { inlineFileRefQuerySchema } from '@/lib/api/contracts/primitives'
 import { shareRecordSchema } from '@/lib/api/contracts/public-shares'
 import { defineRouteContract } from '@/lib/api/contracts/types'
 
-export const workspaceFileScopeSchema = z.enum(['active', 'archived', 'all'])
+/**
+ * Client-reachable listing scopes. `all` is deliberately excluded: it drops the
+ * `deleted_at` predicate, so it cannot use the partial index that serves the
+ * other two and degrades to a full workspace scan. No client requests it, and
+ * server-side callers reach that scope directly rather than over the wire.
+ */
+export const workspaceFileScopeSchema = z.enum(['active', 'archived'])
 
 export const workspaceFilesParamsSchema = z.object({
   id: z.string({ error: 'Workspace ID is required' }).min(1, 'Workspace ID is required'),

apps/sim/lib/billing/core/billing.test.ts

@@ -8,13 +8,15 @@ const {
   mockComputeDailyRefreshConsumed,
   mockEnsureUserStatsExists,
   mockGetBillingPeriodUsageCost,
+  mockGetBillingPeriodUsageCostWithSourceSubset,
   mockGetHighestPriorityPersonalSubscription,
   mockGetHighestPrioritySubscription,
   mockResolveBillingInterval,
 } = vi.hoisted(() => ({
   mockComputeDailyRefreshConsumed: vi.fn(),
   mockEnsureUserStatsExists: vi.fn(),
   mockGetBillingPeriodUsageCost: vi.fn(),
+  mockGetBillingPeriodUsageCostWithSourceSubset: vi.fn(),
   mockGetHighestPriorityPersonalSubscription: vi.fn(),
   mockGetHighestPrioritySubscription: vi.fn(),
   mockResolveBillingInterval: vi.fn(),
@@ -35,6 +37,7 @@ vi.mock('@/lib/billing/core/usage', () => ({
 vi.mock('@/lib/billing/core/usage-log', () => ({
   COPILOT_USAGE_SOURCES: ['copilot'],
   getBillingPeriodUsageCost: mockGetBillingPeriodUsageCost,
+  getBillingPeriodUsageCostWithSourceSubset: mockGetBillingPeriodUsageCostWithSourceSubset,
 }))
 
 vi.mock('@/lib/billing/credits/daily-refresh', () => ({
@@ -50,7 +53,7 @@ describe('getPersonalBillingSummary', () => {
     mockEnsureUserStatsExists.mockResolvedValue(undefined)
     mockResolveBillingInterval.mockReturnValue('year')
     mockComputeDailyRefreshConsumed.mockResolvedValue(3)
-    mockGetBillingPeriodUsageCost.mockResolvedValueOnce(2).mockResolvedValueOnce(1)
+    mockGetBillingPeriodUsageCostWithSourceSubset.mockResolvedValue({ total: 2, subset: 1 })
     mockGetHighestPriorityPersonalSubscription.mockResolvedValue({
       id: 'personal-sub',
       referenceId: 'viewer-a',

apps/sim/lib/billing/core/billing.ts

@@ -7,7 +7,11 @@ import {
   resolveBillingInterval,
 } from '@/lib/billing/core/subscription'
 import { ensureUserStatsExists } from '@/lib/billing/core/usage'
-import { COPILOT_USAGE_SOURCES, getBillingPeriodUsageCost } from '@/lib/billing/core/usage-log'
+import {
+  COPILOT_USAGE_SOURCES,
+  getBillingPeriodUsageCost,
+  getBillingPeriodUsageCostWithSourceSubset,
+} from '@/lib/billing/core/usage-log'
 import {
   computeDailyRefreshConsumed,
   getOrgMemberRefreshBounds,
@@ -429,15 +433,13 @@ export async function getPersonalBillingSummary(userId: string, executor: DbClie
       personalSubscription?.periodStart && personalSubscription.periodEnd
         ? { start: personalSubscription.periodStart, end: personalSubscription.periodEnd }
         : defaultBillingPeriod()
-    const [ledgerUsage, copilotLedgerUsage] = await Promise.all([
-      getBillingPeriodUsageCost({ type: 'user', id: userId }, billingPeriod, undefined, executor),
-      getBillingPeriodUsageCost(
+    const { total: ledgerUsage, subset: copilotLedgerUsage } =
+      await getBillingPeriodUsageCostWithSourceSubset(
         { type: 'user', id: userId },
         billingPeriod,
         COPILOT_USAGE_SOURCES,
         executor
-      ),
-    ])
+      )
 
     const hasPersonalUsageSnapshot =
       Boolean(personalSubscription) && isPro(plan) && stats.proPeriodCostSnapshotAt !== null

apps/sim/lib/billing/core/usage-log.ts

@@ -207,6 +207,40 @@ export async function getBillingPeriodUsageCost(
   return Number.parseFloat(row?.cost ?? '0')
 }
 
+/**
+ * Period total plus the portion attributable to `source`, in a single scan.
+ *
+ * Two separate aggregates over the identical row set double the work and, because
+ * they are separate statements, can observe different snapshots — which makes the
+ * subset exceeding the total representable. One statement rules that out.
+ */
+export async function getBillingPeriodUsageCostWithSourceSubset(
+  billingEntity: BillingEntity,
+  billingPeriod: { start: Date; end: Date },
+  source: UsageLogSource[],
+  executor: DbClient = db
+): Promise<{ total: number; subset: number }> {
+  const [row] = await executor
+    .select({
+      total: sql<string>`COALESCE(SUM(${usageLog.cost}), 0)`,
+      subset: sql<string>`COALESCE(SUM(${usageLog.cost}) FILTER (WHERE ${inArray(usageLog.source, source)}), 0)`,
+    })
+    .from(usageLog)
+    .where(
+      and(
+        eq(usageLog.billingEntityType, billingEntity.type),
+        eq(usageLog.billingEntityId, billingEntity.id),
+        eq(usageLog.billingPeriodStart, billingPeriod.start),
+        eq(usageLog.billingPeriodEnd, billingPeriod.end)
+      )
+    )
+
+  return {
+    total: Number.parseFloat(row?.total ?? '0'),
+    subset: Number.parseFloat(row?.subset ?? '0'),
+  }
+}
+
 export async function getBillingPeriodUsageCostByUser(
   billingEntity: BillingEntity,
   billingPeriod: { start: Date; end: Date },

apps/sim/lib/copilot/vfs/workflow-alias-backing.test.ts

@@ -1,10 +1,10 @@
+import { dbChainMockFns, drizzleOrmMock } from '@sim/testing/mocks'
 import { beforeEach, describe, expect, it, vi } from 'vitest'
 
 const mocks = vi.hoisted(() => ({
   ensureWorkspaceFileFolderPath: vi.fn(),
   listWorkspaceFileFolders: vi.fn(),
   getWorkspaceFileByName: vi.fn(),
-  listWorkspaceFiles: vi.fn(),
   uploadWorkspaceFile: vi.fn(),
 }))
 
@@ -15,11 +15,10 @@ vi.mock('@/lib/uploads/contexts/workspace/workspace-file-folder-manager', () =>
 
 vi.mock('@/lib/uploads/contexts/workspace/workspace-file-manager', () => ({
   getWorkspaceFileByName: mocks.getWorkspaceFileByName,
-  listWorkspaceFiles: mocks.listWorkspaceFiles,
   uploadWorkspaceFile: mocks.uploadWorkspaceFile,
 }))
 
-import { ensureWorkflowAliasBacking } from './workflow-alias-backing'
+import { cleanupWorkflowAliasBacking, ensureWorkflowAliasBacking } from './workflow-alias-backing'
 
 describe('workflow alias backing', () => {
   beforeEach(() => {
@@ -79,4 +78,98 @@ describe('workflow alias backing', () => {
     expect(mocks.uploadWorkspaceFile).not.toHaveBeenCalled()
     expect(result.changelogFile).toMatchObject({ id: 'file-existing' })
   })
+
+  describe('cleanupWorkflowAliasBacking', () => {
+    /**
+     * Folder paths resolve independently of `deletedAt`, so a live file parented
+     * to an archived folder still resolves to a backing path. The archived
+     * `.changelogs` folder below therefore has to participate in file ownership
+     * while staying out of the set of folders that get archived.
+     */
+    const folders = [
+      { id: 'changelog-live', path: '.changelogs', deletedAt: null },
+      { id: 'changelog-archived', path: '.changelogs', deletedAt: new Date() },
+      { id: 'plans-wf1', path: '.plans/wf_1', deletedAt: null },
+      { id: 'plans-wf1-nested', path: '.plans/wf_1/nested', deletedAt: null },
+      { id: 'plans-wf1-archived', path: '.plans/wf_1/old', deletedAt: new Date() },
+      { id: 'plans-wf2', path: '.plans/wf_2', deletedAt: null },
+      { id: 'unrelated', path: 'documents', deletedAt: null },
+    ]
+
+    beforeEach(() => {
+      mocks.listWorkspaceFileFolders.mockResolvedValue(folders)
+    })
+
+    it('scopes file ownership by folder id, including archived folders', async () => {
+      await cleanupWorkflowAliasBacking({ workspaceId: 'workspace-1', workflowId: 'wf_1' })
+
+      const inArrayValues = drizzleOrmMock.inArray.mock.calls.map(([, values]) => values)
+
+      expect(inArrayValues).toContainEqual(['plans-wf1', 'plans-wf1-nested', 'plans-wf1-archived'])
+      expect(inArrayValues).toContainEqual(['changelog-live', 'changelog-archived'])
+    })
+
+    it('archives only live folders owned by the workflow', async () => {
+      await cleanupWorkflowAliasBacking({ workspaceId: 'workspace-1', workflowId: 'wf_1' })
+
+      const inArrayValues = drizzleOrmMock.inArray.mock.calls.map(([, values]) => values)
+
+      expect(inArrayValues).toContainEqual(['plans-wf1', 'plans-wf1-nested'])
+      expect(inArrayValues.flat()).not.toContain('plans-wf2')
+      expect(inArrayValues.flat()).not.toContain('unrelated')
+    })
+
+    it('matches the changelog by the workflow-scoped filename', async () => {
+      await cleanupWorkflowAliasBacking({ workspaceId: 'workspace-1', workflowId: 'wf_1' })
+
+      expect(drizzleOrmMock.eq).toHaveBeenCalledWith(expect.anything(), 'wf_1.md')
+    })
+
+    it('restricts the update to workspace-context files', async () => {
+      await cleanupWorkflowAliasBacking({ workspaceId: 'workspace-1', workflowId: 'wf_1' })
+
+      expect(drizzleOrmMock.eq).toHaveBeenCalledWith(expect.anything(), 'workspace')
+    })
+
+    it('still archives the changelog when the workflow has no plans folder', async () => {
+      mocks.listWorkspaceFileFolders.mockResolvedValue([
+        { id: 'changelog-live', path: '.changelogs', deletedAt: null },
+        { id: 'plans-wf2', path: '.plans/wf_2', deletedAt: null },
+      ])
+
+      await cleanupWorkflowAliasBacking({ workspaceId: 'workspace-1', workflowId: 'wf_1' })
+
+      const inArrayValues = drizzleOrmMock.inArray.mock.calls.map(([, values]) => values)
+      expect(inArrayValues).toContainEqual(['changelog-live'])
+      expect(inArrayValues.flat()).not.toContain('plans-wf2')
+      expect(dbChainMockFns.update).toHaveBeenCalledTimes(1)
+    })
+
+    /**
+     * The decisive guard: `and()` drops `undefined`, so an ownership clause that
+     * resolved to nothing would leave a WHERE matching every file in the
+     * workspace. No ownership must mean no UPDATE is issued at all.
+     */
+    it('issues no update at all when the workflow owns no backing folders', async () => {
+      mocks.listWorkspaceFileFolders.mockResolvedValue([
+        { id: 'unrelated', path: 'documents', deletedAt: null },
+      ])
+
+      const result = await cleanupWorkflowAliasBacking({
+        workspaceId: 'workspace-1',
+        workflowId: 'wf_missing',
+      })
+
+      expect(dbChainMockFns.update).not.toHaveBeenCalled()
+      expect(drizzleOrmMock.inArray).not.toHaveBeenCalled()
+      expect(result).toEqual({ files: 0, folders: 0 })
+    })
+
+    it('never archives files belonging to another workflow', async () => {
+      await cleanupWorkflowAliasBacking({ workspaceId: 'workspace-1', workflowId: 'wf_1' })
+
+      const inArrayValues = drizzleOrmMock.inArray.mock.calls.flatMap(([, values]) => values)
+      expect(inArrayValues).not.toContain('plans-wf2')
+    })
+  })
 })