fix(connectors): key removal on explicit source signals, never on absence

Follow-up to the connector purge fixes, from an independent audit.

YouTube inferred deletion from absence: a playlist entry whose id was missing
from a `videos.list` response was dropped, so a well-formed 200 that returned 49
of 50 requested ids hard-deleted the 50th. Playlist items instead carry a
documented `status.privacyStatus`, available as a free part on a call the
connector already makes, so the extra `videos.list` request is gone along with
its quota-failure and pagination-wedge risks. An item is now excluded only on an
explicit `private`; missing, empty, or unrecognized values keep it.

ServiceNow read every record through a guard requiring a string `sys_id`, but
the listing requests `sysparm_display_value=all`, under which every field —
`sys_id` included — comes back as `{display_value, value}`. The guard rejected
every record, so the retired-article filter was unreachable and the sys_id
object would have leaked into `externalId` and `title` had it not been. Records
are now read through the existing `rawValue` normalizer, which accepts both wire
shapes, and the fixtures use the shape the API actually returns.

Also: resolve the ServiceNow cap ambiguity with `X-Total-Count` so a table that
ends exactly on a page boundary is not read as truncated; stop the Google Sheets
comment claiming a purge the engine's zero-document guard prevents; and assert
the Outlook junk-mail invariant instead of comparing a constant to itself.

Document the behavior change: content archived, retired, or trashed at the
source is now removed from the knowledge base, and restoring it re-ingests it.

Author: waleedlatif1

apps/docs/content/docs/en/knowledgebase/connectors.mdx

@@ -162,6 +162,12 @@ To reverse an exclusion, switch to the **Excluded** tab and click **Restore** 
 
 On each run the connector fetches documents from the source and compares them against what's already stored. Only changed documents are reprocessed — new content is added, updated content is re-chunked and re-embedded, deleted content is removed. A connector syncing thousands of documents will only do real work when something actually changes.
 
+### Content Removed From View
+
+Content that still exists at the source but is no longer current is treated the same as deleted, and is removed from your knowledge base. This covers an archived Confluence page, a task in an archived Asana project, an archived Webflow CMS item, a retired ServiceNow knowledge article, a spreadsheet in the Google Drive trash, a message moved to Outlook's Deleted Items, and a cancelled incident.io incident.
+
+This keeps agents from citing content your team has already retired. Restoring the item at the source brings it back on the next sync as a new document.
+
 ### Connector Status
 
 | Status | Meaning |
@@ -196,7 +202,7 @@ You can add as many connectors as you need to a single knowledge base. Each mana
 
 <FAQ items={[
   { question: "How often do connectors sync?", answer: "You choose from hourly, every 6 hours, daily (default), weekly, or manual-only. Sub-hourly frequencies require a Max or Enterprise plan. Each connector has its own schedule." },
-  { question: "What happens if a source document is deleted?", answer: "On the next sync the connector detects the document is gone and removes it from your knowledge base automatically." },
+  { question: "What happens if a source document is deleted?", answer: "On the next sync the connector detects the document is gone and removes it from your knowledge base automatically. Content that is archived, retired, trashed, or otherwise removed from view at the source is treated the same way — restoring it at the source brings it back on the next sync." },
   { question: "What happens when I delete a connector?", answer: "The connector is removed and future syncs stop. You're given the option to also delete all documents that were synced by that connector. If you don't check that option, they stay in the knowledge base as-is." },
   { question: "What does the Disabled status mean?", answer: "After 10 consecutive full-sync failures, the connector is automatically disabled to stop retrying. Reconnect the OAuth account or click Resume to re-enable it." },
   { question: "Do metadata tags count against a limit?", answer: "Yes. Tag slots are shared across all documents in a knowledge base — 17 slots total. Multiple connectors draw from the same pool, so plan accordingly if several connectors each auto-populate tags." },

apps/sim/connectors/google-sheets/google-sheets.ts

@@ -268,8 +268,16 @@ export const googleSheetsConnector: ConnectorConfig = {
     ])
 
     /**
-     * A trashed spreadsheet is no longer current content, so it must drop out of
-     * the full listing for deletion reconciliation to purge its stored tabs.
+     * A trashed spreadsheet is no longer current content, so it drops out of the
+     * listing and stops being re-indexed.
+     *
+     * This does not by itself purge the stored tabs: an empty listing is
+     * indistinguishable from a provider outage, so the sync engine's
+     * zero-document guard deliberately skips reconciliation rather than risk
+     * wiping a knowledge base on a bad response. A forced full resync is the
+     * supported way to complete the cleanup. `validateConfig` reports the
+     * trashed state so the connector does not look healthy while serving tabs
+     * from a file its owner has thrown away.
      */
     if (isTrashedDriveFile(driveMetadata)) {
       logger.info('Spreadsheet is in the Drive trash; listing no documents', { spreadsheetId })

apps/sim/connectors/outlook/outlook.test.ts

@@ -14,6 +14,7 @@ import {
 const DELETED_ITEMS_ID = 'deleted-items-id'
 const DELETED_SUBFOLDER_ID = 'deleted-subfolder-id'
 const INBOX_ID = 'inbox-id'
+const JUNK_EMAIL_ID = 'junk-email-id'
 
 interface JsonResponseInit {
   status?: number
@@ -109,7 +110,7 @@ describe('isAllMailSync', () => {
 })
 
 describe('DELETED_ITEMS_FOLDER', () => {
-  it('excludes Deleted Items only, never Junk Email', () => {
+  it('pins the well-known folder name interpolated into the Graph URL', () => {
     expect(DELETED_ITEMS_FOLDER).toBe('deleteditems')
   })
 })
@@ -196,6 +197,29 @@ describe('listDocuments folder exclusion', () => {
     expect(result.documents.map((d) => d.externalId)).toEqual(['live'])
   })
 
+  it('keeps junk mail, which is a spam classification rather than a deletion', async () => {
+    routeFetch([
+      deletedItemsRoute,
+      childFoldersRoute,
+      [
+        /\/me\/messages\?/,
+        () =>
+          jsonResponse({
+            value: [
+              message({ id: 'm1', conversationId: 'live', parentFolderId: INBOX_ID }),
+              message({ id: 'm2', conversationId: 'junked', parentFolderId: JUNK_EMAIL_ID }),
+            ],
+          }),
+      ],
+    ])
+
+    const result = await outlookConnector.listDocuments('token', { folder: 'all' }, undefined, {})
+
+    expect(result.documents.map((d) => d.externalId).sort()).toEqual(['junked', 'live'])
+    const requested = fetchMock.mock.calls.map((call) => String(call[0]))
+    expect(requested.some((url) => url.includes('junkemail'))).toBe(false)
+  })
+
   it('keeps every conversation when the Deleted Items lookup fails with a non-2xx', async () => {
     routeFetch([
       [/mailFolders\/deleteditems\?/, () => jsonResponse({ error: 'forbidden' }, { status: 403 })],

apps/sim/connectors/servicenow/servicenow.test.ts

@@ -20,14 +20,27 @@ const KB_CONFIG = {
   contentType: 'kb_knowledge',
 } as const
 
-interface FakeRecord {
-  sys_id: string
-  workflow_state?: unknown
-  short_description?: string
-  text?: string
+/**
+ * Wraps a value the way `sysparm_display_value=all` does. Under `all` the Table
+ * API returns EVERY column — `sys_id` included — as `{ display_value, value }`,
+ * so fixtures must use this shape to exercise the real listing path.
+ */
+function field(value: string): { display_value: string; value: string } {
+  return { display_value: value, value }
+}
+
+/** Builds a `kb_knowledge` row in the `sysparm_display_value=all` wire shape. */
+function kbRecord(sysId: string, workflowState?: string): Record<string, unknown> {
+  return {
+    sys_id: field(sysId),
+    ...(workflowState === undefined ? {} : { workflow_state: field(workflowState) }),
+    short_description: field(`Article ${sysId.slice(0, 4)}`),
+    text: field(`Body of ${sysId.slice(0, 4)}`),
+  }
 }
 
-function kbRecord(sysId: string, workflowState?: unknown): FakeRecord {
+/** Builds the same row in the plain-string shape (`display_value` absent/true/false). */
+function kbRecordPlain(sysId: string, workflowState?: string): Record<string, unknown> {
   return {
     sys_id: sysId,
     ...(workflowState === undefined ? {} : { workflow_state: workflowState }),
@@ -136,6 +149,47 @@ describe('shouldIngestKBArticle', () => {
 })
 
 describe('servicenowConnector.listDocuments', () => {
+  it('accepts the sysparm_display_value=all object shape, where sys_id is an object', async () => {
+    mockFetch.mockResolvedValueOnce(jsonResponse({ result: [kbRecord(SYS_ID_A, 'published')] }))
+
+    const list = await servicenowConnector.listDocuments('key', { ...KB_CONFIG })
+
+    expect(lastRequestUrl().searchParams.get('sysparm_display_value')).toBe('all')
+    expect(list.documents).toHaveLength(1)
+    expect(list.documents[0].externalId).toBe(SYS_ID_A)
+    expect(list.documents[0].title).toBe(`Article ${SYS_ID_A.slice(0, 4)}`)
+    expect(list.documents[0].sourceUrl).toBe(`${INSTANCE_URL}/kb_view.do?sys_kb_id=${SYS_ID_A}`)
+    expect(list.documents[0].metadata.workflowState).toBe('published')
+  })
+
+  it('still accepts the plain-string shape', async () => {
+    mockFetch.mockResolvedValueOnce(
+      jsonResponse({
+        result: [kbRecordPlain(SYS_ID_A, 'published'), kbRecordPlain(SYS_ID_B, 'retired')],
+      })
+    )
+
+    const list = await servicenowConnector.listDocuments('key', { ...KB_CONFIG })
+
+    expect(list.documents.map((doc) => doc.externalId)).toEqual([SYS_ID_A])
+    expect(list.documents[0].title).toBe(`Article ${SYS_ID_A.slice(0, 4)}`)
+  })
+
+  it('skips records whose sys_id object carries no usable value', async () => {
+    mockFetch.mockResolvedValueOnce(
+      jsonResponse({
+        result: [
+          { ...kbRecord(SYS_ID_A, 'published'), sys_id: { display_value: null, value: '' } },
+          kbRecord(SYS_ID_B, 'published'),
+        ],
+      })
+    )
+
+    const list = await servicenowConnector.listDocuments('key', { ...KB_CONFIG })
+
+    expect(list.documents.map((doc) => doc.externalId)).toEqual([SYS_ID_B])
+  })
+
   it('drops retired KB records and keeps every other state', async () => {
     mockFetch.mockResolvedValueOnce(
       jsonResponse({
@@ -200,12 +254,17 @@ describe('servicenowConnector.listDocuments', () => {
     mockFetch.mockResolvedValueOnce(
       jsonResponse({
         result: [
-          { sys_id: SYS_ID_A, number: 'INC001', short_description: 'Down', state: '7' },
           {
-            sys_id: SYS_ID_B,
-            number: 'INC002',
-            short_description: 'Up',
-            workflow_state: 'retired',
+            sys_id: field(SYS_ID_A),
+            number: field('INC001'),
+            short_description: field('Down'),
+            state: { display_value: 'Closed', value: '7' },
+          },
+          {
+            sys_id: field(SYS_ID_B),
+            number: field('INC002'),
+            short_description: field('Up'),
+            workflow_state: field('retired'),
           },
         ],
       })
@@ -280,7 +339,17 @@ describe('servicenowConnector.getDocument', () => {
     const doc = await servicenowConnector.getDocument('key', { ...KB_CONFIG }, SYS_ID_A)
 
     expect(doc?.externalId).toBe(SYS_ID_A)
+    expect(doc?.sourceUrl).toBe(`${INSTANCE_URL}/kb_view.do?sys_kb_id=${SYS_ID_A}`)
     expect(lastRequestUrl().pathname).toBe(`/api/now/table/kb_knowledge/${SYS_ID_A}`)
+    expect(lastRequestUrl().searchParams.get('sysparm_display_value')).toBe('all')
+  })
+
+  it('hydrates an article returned in the plain-string shape', async () => {
+    mockFetch.mockResolvedValueOnce(jsonResponse({ result: kbRecordPlain(SYS_ID_A, 'published') }))
+
+    const doc = await servicenowConnector.getDocument('key', { ...KB_CONFIG }, SYS_ID_A)
+
+    expect(doc?.externalId).toBe(SYS_ID_A)
   })
 
   it('rejects a malformed sys_id without issuing a request', async () => {