fix(confluence): index mirrored/included page content via rendered view format (#5746)

* fix(confluence): index mirrored/included page content via rendered view format

The KB connector fetched page bodies as body-format=storage, which only
carries unexpanded macro references (Include Page / Excerpt Include). Those
'mirrored' articles were stripped to empty content by htmlToPlainText and never
synced. Switch getDocument to body-format=view (supported on the v2 single-item
page/blogpost GET) so built-in include/excerpt macros render inline and the
included text is indexed.

* fix(confluence): invalidate existing doc hashes on representation change

The version-based contentHash meant already-synced mirrored documents (with
stale empty content) classified as 'unchanged' and never re-hydrated with the
new rendered view content. Embed a body-representation marker in the hash so a
representation change invalidates every previously-synced Confluence document,
forcing a one-time re-hydration that picks up the expanded include/excerpt text.

* feat(connectors): full resync re-hydrates rendered content (transclusions)

Version-based change detection can't see when a Confluence page's rendered view
changes because an *included* page was edited (the container's version doesn't
bump). Add a 'Full resync' path so that drift can be recovered:

- ConnectorMeta.rehydrateOnFullSync flag (set for Confluence)
- on fullSync, classifyExternalDoc promotes unchanged deferred docs to update and
  the hydration guard re-indexes unconditionally, so rendered content is refreshed
- fullSync threaded through the manual sync contract (query param), route, and hook
- 'Sync now' / 'Full resync' dropdown on the connector card

Incremental syncs stay hash-gated and cheap; only the deliberate full resync pays
the re-index cost.

* refactor(connectors): decouple rehydrate from fullSync deletion semantics

The transclusion refresh only needs re-hydration, but reusing the fullSync flag
also activated its deletion-cleanup semantics — which bypass three previously
unreachable safety guards (empty-listing wipe, listingCapped, and the >50%
mass-deletion threshold). Since fullSync had no caller before this PR, the new
'Full resync' button would have exposed all three to any KB editor.

Introduce a dedicated 'rehydrate' request that ONLY forces re-hydration + re-index
of already-synced docs. Listing and deletion reconciliation are identical to a
normal sync (all safety guards stay armed). fullSync's cleanup semantics remain
dormant and untouched.

* refactor(connectors): tidy rehydrate flag + gate Full resync to supported connectors

Cleanup/simplify pass over the connector changes:
- use shared booleanQueryFlagSchema for the rehydrate query param (typed boolean
  at the boundary instead of a hand-rolled 'true'/'false' string enum)
- move rehydrateOnFullSync onto the client-safe ConnectorMeta so the UI can gate on it
- only Confluence (rehydrateOnFullSync) shows the Sync now / Full resync dropdown;
  every other connector keeps its original one-click sync button (Full resync is a
  no-op for them, and this restores the pre-change one-click UX)
- wrap the sync trigger in a span so its tooltip still shows while disabled (cooldown)

* fix(connectors): forward rehydrate flag through the Trigger.dev worker

executeConnectorSyncJob (the production async sync path) destructured only
fullSync from the payload and forwarded only fullSync to executeSync, silently
dropping rehydrate. A manual Full resync would therefore never re-hydrate on the
default Trigger.dev path. Forward rehydrate too.

* fix(connectors): rehydrate forces a full listing so containers aren't omitted

A rehydrate request set forceRehydrate but left listing incremental. For a
connector that is both incremental and rehydrateOnFullSync, an unchanged
container page that transcludes a changed page would be omitted from the
incremental listing and never re-hydrated. Force a full (non-incremental) listing
on rehydrate so every document is seen; deletion-safety guards stay armed (unlike
fullSync). No-op for Confluence, which is already non-incremental.

Author: waleedlatif1

apps/sim/app/api/knowledge/[id]/connectors/[connectorId]/sync/route.test.ts

@@ -138,6 +138,50 @@ describe('Connector Manual Sync API Route', () => {
     expect(mockDispatchSync).toHaveBeenCalledWith('conn-456', {
       billingAttribution,
       requestId: 'test-req-id',
+      rehydrate: false,
+    })
+  })
+
+  it('dispatches a full resync when rehydrate=true is set', async () => {
+    const billingAttribution = {
+      actorUserId: 'external-admin',
+      workspaceId: 'ws-1',
+      organizationId: null,
+      billedAccountUserId: 'owner-1',
+      billingEntity: { type: 'user' as const, id: 'owner-1' },
+      billingPeriod: {
+        start: '2026-07-01T00:00:00.000Z',
+        end: '2026-08-01T00:00:00.000Z',
+      },
+      payerSubscription: null,
+    }
+    hybridAuthMockFns.mockCheckSessionOrInternalAuth.mockResolvedValue({
+      success: true,
+      authType: 'session',
+      userId: 'external-admin',
+      userName: 'Test',
+      userEmail: 'test@test.com',
+    })
+    mockCheckWriteAccess.mockResolvedValue({
+      hasAccess: true,
+      knowledgeBase: { workspaceId: 'ws-1', name: 'Test KB' },
+    })
+    mockDbChain.limit.mockResolvedValueOnce([{ id: 'conn-456', status: 'active' }])
+    mockResolveBillingAttribution.mockResolvedValue(billingAttribution)
+
+    const req = createMockRequest(
+      'POST',
+      undefined,
+      {},
+      'http://localhost:3000/api/knowledge/kb-123/connectors/conn-456/sync?rehydrate=true'
+    )
+    const response = await POST(req as never, { params: mockParams })
+
+    expect(response.status).toBe(200)
+    expect(mockDispatchSync).toHaveBeenCalledWith('conn-456', {
+      billingAttribution,
+      requestId: 'test-req-id',
+      rehydrate: true,
     })
   })
 })

apps/sim/app/api/knowledge/[id]/connectors/[connectorId]/sync/route.ts

@@ -29,6 +29,7 @@ export const POST = withRouteHandler(async (request: NextRequest, context: Route
   const parsed = await parseRequest(triggerKnowledgeConnectorSyncContract, request, context)
   if (!parsed.success) return parsed.response
   const { id: knowledgeBaseId, connectorId } = parsed.data.params
+  const { rehydrate } = parsed.data.query
 
   try {
     const auth = await checkSessionOrInternalAuth(request, { requireWorkflowId: false })
@@ -81,7 +82,9 @@ export const POST = withRouteHandler(async (request: NextRequest, context: Route
             workspaceId: kbWorkspaceId,
           })
 
-    logger.info(`[${requestId}] Manual sync triggered for connector ${connectorId}`)
+    logger.info(
+      `[${requestId}] Manual sync${rehydrate ? ' (full rehydrate)' : ''} triggered for connector ${connectorId}`
+    )
 
     captureServerEvent(
       auth.userId,
@@ -109,12 +112,12 @@ export const POST = withRouteHandler(async (request: NextRequest, context: Route
         knowledgeBaseName: writeCheck.knowledgeBase.name,
         connectorType: connectorRows[0].connectorType,
         connectorStatus: connectorRows[0].status,
-        syncType: 'manual',
+        syncType: rehydrate ? 'manual-rehydrate' : 'manual',
       },
       request,
     })
 
-    dispatchSync(connectorId, { billingAttribution, requestId }).catch((error) => {
+    dispatchSync(connectorId, { billingAttribution, requestId, rehydrate }).catch((error) => {
       logger.error(
         `[${requestId}] Failed to dispatch manual sync for connector ${connectorId}`,
         error

apps/sim/app/workspace/[workspaceId]/knowledge/[id]/components/connectors-section/connectors-section.tsx

@@ -1,7 +1,19 @@
 'use client'
 
 import { useCallback, useEffect, useId, useMemo, useRef, useState } from 'react'
-import { Badge, Button, Checkbox, ChipConfirmModal, cn, Loader, Tooltip } from '@sim/emcn'
+import {
+  Badge,
+  Button,
+  Checkbox,
+  ChipConfirmModal,
+  cn,
+  DropdownMenu,
+  DropdownMenuContent,
+  DropdownMenuItem,
+  DropdownMenuTrigger,
+  Loader,
+  Tooltip,
+} from '@sim/emcn'
 import { createLogger } from '@sim/logger'
 import { format, formatDistanceToNow, isPast } from 'date-fns'
 import {
@@ -115,14 +127,14 @@ export function ConnectorsSection({
   }, [])
 
   const handleSync = useCallback(
-    (connectorId: string) => {
+    (connectorId: string, rehydrate = false) => {
       if (isSyncOnCooldown(connectorId)) return
 
       syncTriggeredAt.current[connectorId] = Date.now()
       addToSet(setSyncingIds, connectorId)
 
       triggerSync(
-        { knowledgeBaseId, connectorId },
+        { knowledgeBaseId, connectorId, rehydrate },
         {
           onSuccess: () => {
             setError(null)
@@ -214,7 +226,7 @@ export function ConnectorsSection({
               isSyncPending={syncingIds.has(connector.id)}
               isUpdating={updatingIds.has(connector.id)}
               syncCooldown={isSyncOnCooldown(connector.id)}
-              onSync={() => handleSync(connector.id)}
+              onSync={(rehydrate) => handleSync(connector.id, rehydrate)}
               onTogglePause={() => handleTogglePause(connector)}
               onEdit={() => setEditingConnector(connector)}
               onDelete={() => setDeleteTarget(connector.id)}
@@ -273,7 +285,7 @@ interface ConnectorCardProps {
   isSyncPending: boolean
   isUpdating: boolean
   syncCooldown: boolean
-  onSync: () => void
+  onSync: (rehydrate?: boolean) => void
   onEdit: () => void
   onTogglePause: () => void
   onDelete: () => void
@@ -327,6 +339,14 @@ function ConnectorCard({
   )
   const syncLogs = detail?.syncLogs ?? []
 
+  const canFullResync = Boolean(connectorDef?.rehydrateOnFullSync)
+  const syncDisabled =
+    connector.status === 'syncing' ||
+    connector.status === 'disabled' ||
+    isSyncPending ||
+    syncCooldown
+  const syncTooltip = syncCooldown ? 'Sync recently triggered' : canFullResync ? 'Sync' : 'Sync now'
+
   return (
     <div
       className={cn(
@@ -409,28 +429,60 @@ function ConnectorCard({
         <div className='flex flex-shrink-0 items-center gap-0.5'>
           {canEdit && (
             <>
-              <Tooltip.Root>
-                <Tooltip.Trigger asChild>
-                  <Button
-                    variant='ghost'
-                    className={CONNECTOR_ACTION_BUTTON_CLASSES}
-                    onClick={onSync}
-                    disabled={
-                      connector.status === 'syncing' ||
-                      connector.status === 'disabled' ||
-                      isSyncPending ||
-                      syncCooldown
-                    }
-                  >
-                    <RefreshCw
-                      className={cn('size-3.5', connector.status === 'syncing' && 'animate-spin')}
-                    />
-                  </Button>
-                </Tooltip.Trigger>
-                <Tooltip.Content>
-                  {syncCooldown ? 'Sync recently triggered' : 'Sync now'}
-                </Tooltip.Content>
-              </Tooltip.Root>
+              {canFullResync ? (
+                <DropdownMenu>
+                  <Tooltip.Root>
+                    <Tooltip.Trigger asChild>
+                      {/* span keeps the tooltip hoverable while the trigger button is disabled */}
+                      <span className='inline-flex'>
+                        <DropdownMenuTrigger asChild>
+                          <Button
+                            variant='ghost'
+                            aria-label='Sync options'
+                            className={CONNECTOR_ACTION_BUTTON_CLASSES}
+                            disabled={syncDisabled}
+                          >
+                            <RefreshCw
+                              className={cn(
+                                'size-3.5',
+                                connector.status === 'syncing' && 'animate-spin'
+                              )}
+                            />
+                          </Button>
+                        </DropdownMenuTrigger>
+                      </span>
+                    </Tooltip.Trigger>
+                    <Tooltip.Content>{syncTooltip}</Tooltip.Content>
+                  </Tooltip.Root>
+                  <DropdownMenuContent align='end'>
+                    <DropdownMenuItem onSelect={() => onSync(false)}>Sync now</DropdownMenuItem>
+                    <DropdownMenuItem onSelect={() => onSync(true)}>Full resync</DropdownMenuItem>
+                  </DropdownMenuContent>
+                </DropdownMenu>
+              ) : (
+                <Tooltip.Root>
+                  <Tooltip.Trigger asChild>
+                    {/* span keeps the tooltip hoverable while the button is disabled */}
+                    <span className='inline-flex'>
+                      <Button
+                        variant='ghost'
+                        aria-label='Sync now'
+                        className={CONNECTOR_ACTION_BUTTON_CLASSES}
+                        disabled={syncDisabled}
+                        onClick={() => onSync(false)}
+                      >
+                        <RefreshCw
+                          className={cn(
+                            'size-3.5',
+                            connector.status === 'syncing' && 'animate-spin'
+                          )}
+                        />
+                      </Button>
+                    </span>
+                  </Tooltip.Trigger>
+                  <Tooltip.Content>{syncTooltip}</Tooltip.Content>
+                </Tooltip.Root>
+              )}
 
               <Tooltip.Root>
                 <Tooltip.Trigger asChild>

apps/sim/background/knowledge-connector-sync.test.ts

@@ -72,6 +72,29 @@ describe('knowledge connector sync worker', () => {
     expect(mockExecuteSync).toHaveBeenCalledWith('connector-1', {
       billingAttribution: BILLING_ATTRIBUTION,
       fullSync: true,
+      rehydrate: undefined,
+    })
+  })
+
+  it('forwards the rehydrate flag to the sync engine (async worker path)', async () => {
+    mockAssertConnectorSyncPayload.mockReturnValue({
+      connectorId: 'connector-1',
+      requestId: 'request-1',
+      rehydrate: true,
+      billingAttribution: BILLING_ATTRIBUTION,
+    })
+
+    await executeConnectorSyncJob({
+      connectorId: 'connector-1',
+      requestId: 'request-1',
+      rehydrate: true,
+      billingAttribution: BILLING_ATTRIBUTION,
+    })
+
+    expect(mockExecuteSync).toHaveBeenCalledWith('connector-1', {
+      billingAttribution: BILLING_ATTRIBUTION,
+      fullSync: undefined,
+      rehydrate: true,
     })
   })
 })

apps/sim/background/knowledge-connector-sync.ts

@@ -9,13 +9,13 @@ import { executeSync } from '@/lib/knowledge/connectors/sync-engine'
 const logger = createLogger('TriggerKnowledgeConnectorSync')
 
 export async function executeConnectorSyncJob(payload: unknown) {
-  const { connectorId, fullSync, requestId, billingAttribution } =
+  const { connectorId, fullSync, rehydrate, requestId, billingAttribution } =
     assertConnectorSyncPayload(payload)
 
   logger.info(`[${requestId}] Starting connector sync: ${connectorId}`)
 
   try {
-    const result = await executeSync(connectorId, { billingAttribution, fullSync })
+    const result = await executeSync(connectorId, { billingAttribution, fullSync, rehydrate })
 
     logger.info(`[${requestId}] Connector sync completed`, {
       connectorId,

apps/sim/connectors/confluence/confluence.ts

@@ -91,6 +91,16 @@ async function fetchLabelsForPages(
   return labelsByPageId
 }
 
+/**
+ * Body representation marker embedded in the contentHash. Bumping this
+ * invalidates every previously-synced Confluence document so a one-time
+ * re-hydration picks up content newly reachable by the current extraction
+ * (e.g. the switch from `storage` to rendered `view`, which expands Include
+ * Page / Excerpt macros). Without it, already-indexed pages whose version is
+ * unchanged classify as `unchanged` and keep their stale (empty) content.
+ */
+const CONTENT_REPRESENTATION = 'view'
+
 /**
  * Produces a canonical metadata stub with a deterministic contentHash that
  * does not depend on which API surface (v1 CQL or v2) returned the page.
@@ -115,7 +125,7 @@ function pageToStub(
     contentDeferred: true,
     mimeType: 'text/plain',
     sourceUrl: options.sourceUrl,
-    contentHash: `confluence:${page.id}:${versionKey}`,
+    contentHash: `confluence:${CONTENT_REPRESENTATION}:${page.id}:${versionKey}`,
     metadata: {
       spaceId: options.spaceId,
       status: page.status,
@@ -233,10 +243,18 @@ export const confluenceConnector: ConnectorConfig = {
       if (syncContext) syncContext.cloudId = cloudId
     }
 
-    // Try pages first, fall back to blogposts if not found
+    /**
+     * Fetch the `view` representation rather than `storage`. Storage format only
+     * carries unexpanded macro references (e.g. Include Page / Excerpt Include),
+     * so "mirrored" content that pulls in another page's body is stripped to
+     * nothing by `htmlToPlainText`. The `view` representation is server-rendered
+     * HTML with those macros expanded inline, so included content is indexed too.
+     * The v2 single-item GET (`/pages/{id}`, `/blogposts/{id}`) supports
+     * `body-format=view`; only the bulk list endpoints are limited to storage/adf.
+     */
     let page: Record<string, unknown> | null = null
     for (const endpoint of ['pages', 'blogposts']) {
-      const url = `https://api.atlassian.com/ex/confluence/${cloudId}/wiki/api/v2/${endpoint}/${externalId}?body-format=storage`
+      const url = `https://api.atlassian.com/ex/confluence/${cloudId}/wiki/api/v2/${endpoint}/${externalId}?body-format=view`
       const response = await fetchWithRetry(url, {
         method: 'GET',
         headers: {
@@ -256,8 +274,8 @@ export const confluenceConnector: ConnectorConfig = {
 
     if (!page) return null
     const body = page.body as Record<string, unknown> | undefined
-    const storage = body?.storage as Record<string, unknown> | undefined
-    const rawContent = (storage?.value as string) || ''
+    const view = body?.view as Record<string, unknown> | undefined
+    const rawContent = (view?.value as string) || ''
     const plainText = htmlToPlainText(rawContent)
 
     const labelMap = await fetchLabelsForPages(cloudId, accessToken, [String(page.id)])

apps/sim/connectors/confluence/meta.ts

@@ -22,6 +22,15 @@ export const confluenceConnectorMeta: ConnectorMeta = {
     ],
   },
 
+  /**
+   * Confluence pages can transclude other pages (Include Page / Excerpt macros).
+   * Editing an included page changes a container page's rendered `view` without
+   * bumping the container's version, so its version-based hash can't detect the
+   * change. A full resync re-hydrates and re-indexes to pick up that drift. This
+   * lives on the meta so the client can offer "Full resync" only where it applies.
+   */
+  rehydrateOnFullSync: true,
+
   configFields: [
     {
       id: 'domain',

apps/sim/connectors/types.ts

@@ -127,6 +127,20 @@ export interface ConnectorMeta {
    */
   supportsIncrementalSync?: boolean
 
+  /**
+   * Whether this connector's extracted content can change without the source item's
+   * own change-detection hash changing — e.g. a Confluence page that transcludes
+   * another page via the Include Page / Excerpt macro: editing the included page
+   * changes the container's rendered `view` output without bumping the container's
+   * version, so its version-based `contentHash` stays identical.
+   *
+   * Incremental syncs remain hash-gated (cheap). On an explicit **full resync**
+   * (`fullSync`), the engine re-hydrates and re-indexes these connectors' documents
+   * even when their hash is unchanged, so transcluded/rendered-dependency changes are
+   * picked up. Only the deliberate full resync pays this re-index cost.
+   */
+  rehydrateOnFullSync?: boolean
+
   /**
    * Tag definitions this connector populates. Shown in the add-connector modal
    * as opt-out checkboxes. On connector creation, tag definitions are auto-created

apps/sim/hooks/queries/kb/connectors.ts

@@ -204,11 +204,18 @@ export function useDeleteConnector() {
 interface TriggerSyncParams {
   knowledgeBaseId: string
   connectorId: string
+  /** Force re-hydration + re-index of rendered content (the "Full resync" action). */
+  rehydrate?: boolean
 }
 
-async function triggerSync({ knowledgeBaseId, connectorId }: TriggerSyncParams): Promise<void> {
+async function triggerSync({
+  knowledgeBaseId,
+  connectorId,
+  rehydrate,
+}: TriggerSyncParams): Promise<void> {
   await requestJson(triggerKnowledgeConnectorSyncContract, {
     params: { id: knowledgeBaseId, connectorId },
+    query: rehydrate ? { rehydrate: true } : {},
   })
 }