fix(connectors): make selector dropdowns resolve options the drain has not reached (#6027)

* fix(connectors): stop the connector selector implying an option does not exist

The connector selector field ignored the drain state the shared selector hook
already exposes. Paginated selectors fill in the background and the combobox
filters client-side, so a not-yet-drained option is genuinely absent from the
list — and the dropdown said "No spaces found", which reads as "this space does
not exist" and sends users off to enter the value by hand. That is what happened
with a Confluence space on a site with thousands of personal spaces.

It now reports that the list is still filling, surfaces a failed drain instead of
claiming to load forever, and is honest when the drain stops at its page cap.

* feat(connectors): resolve selector options by exact key without waiting on the drain

The connector selector fills by draining pages in the background and filters
client-side, so an option is only findable once its page has arrived. On a large
Confluence site that is ~38 seconds, during which searching for a real space
returned nothing.

Resolves the typed value and the selected value directly through the selector's
`fetchById`, merging both into the option list, so an exact key is selectable
immediately regardless of drain progress. `confluence.spaces.fetchById` now uses
the documented v2 `keys` filter instead of scanning only the first page — which
never resolved a space sorting beyond it.

Adds `onSearchChange` to the shared Combobox so a consumer can observe the search
box, held in a ref so its identity stays stable for the handlers that capture it
without declaring it.

Also fixes a pre-existing hazard in useSelectorOptionDetail: a caller-supplied
`enabled` replaced the guard that checks a definition declares `fetchById`, so
`queryFn`'s non-null assertion would throw for the ~12 selectors without one.

* fix(connectors): paginate the Confluence pages selector instead of returning one page

`/api/tools/confluence/pages` issued a single request and discarded the
`_links.next` cursor Confluence returns, so the picker showed only the first
`limit` pages of however many exist — a search for a real page found nothing,
with no error and no truncation signal.

Threads the documented opaque cursor and moves the selector to `fetchPage`, so
the option list drains like `confluence.spaces` and reports real `hasMore` /
`truncated` state. The `title` server-side filter is unchanged and now paginates
too. `limit` is deliberately left at its existing default: the endpoint's maximum
is not confirmable from Atlassian's published reference, and raising it is not
needed for correctness.

Also guards `data.results`, which threw when the response omitted it.

* fix(connectors): query both space statuses explicitly on exact-key lookup

The exact-key lookup omitted `status`, assuming that matched a space whether it
was current or archived. Atlassian documents a `current,archived` default for
`/pages` but documents no default for `/spaces`, where `status` takes a single
value rather than an array — so the assumption was unverified, and resolving
only current spaces would silently miss archived ones. Archived spaces are
reachable through the paged path and sync works against them.

Queries `current` first and falls back to `archived` only when it finds nothing,
so the common case stays one request and the behaviour no longer depends on an
undocumented default.

* fix(connectors): revert pages drain, restore CloudWatch labels, tighten key lookup

Six adversarial verification passes against the provider spec and the surrounding
plumbing found three defects in the prior commits.

Reverts the Confluence pages selector to a single request. Draining it was not
strictly better: with no search term `title` is unset, so opening the dropdown
walked the entire site — up to 50 sequential requests and 2,500 options where
there had been one request and 50 — and the route forwards no abort signal
upstream, so superseded drains still bill the tenant's rate limit. The same
fan-out reached `loadAllSelectorOptions`, and the justification rested on `title`
semantics Atlassian does not publish. The list cap is a real gap, but it needs
confirmed server-side search, not brute force. Keeps the `results` guard.

Restores selector display names for CloudWatch blocks. Narrowing a caller's
`enabled` against `definition.enabled` disabled a detail query those selectors
satisfy without AWS context, so collapsed blocks rendered "-" instead of the log
group name. A caller that opts in is now narrowed only by the hard precondition
that `fetchById` exists, which is what `queryFn` actually asserts.

Queries both space statuses concurrently rather than sequentially: the key is
user-typed text, so a miss dominates while typing and paid two round-trips. Each
row now falls back to the status its own call requested.

Also drops the "enter the value directly" empty states — the combobox is not
editable, so there is no such affordance — and dedupes `onSearchChange`, which
several reset paths fire redundantly.

* fix(connectors): keep resolved option labels and survive a half-failed key lookup

Addresses three review findings.

A key lookup failed entirely when either status leg errored, discarding a match
the other leg had found. Only a total failure is fatal now.

Resolved option labels are remembered for the lifetime of the field. Both lookups
key on values that change — the search box clears on select and close, and a
multi-select field resolves no id at all — so the label for a just-picked option
vanished a debounce later and the trigger fell back to a raw id. This covers the
multi-select case, which is what the Confluence space field uses.

A failed exact-value lookup now reads differently from a failed list load, rather
than being reported as "still loading" or "none found".

* fix(connectors): drop remembered option labels when the selector context changes

Resolved ids are only meaningful within one selector context, so switching
credential, domain, or a dependency has to discard what was remembered under the
old one — the queries re-key, but the remembered labels would linger and mislabel
until the field remounted.

Keyed on the serialized context rather than its identity: the context memo also
depends on `sourceConfig`, so its identity changes on unrelated field edits and
would clear the cache far more often than intended.

* refactor(connectors): resolve selected option labels with useQueries

Replaces the remembered-label map with per-id queries over the selected values,
following the pattern the knowledge-base selector already uses. The map only ever
held ids searched for in the current session, so a multi-select field restored
from saved config still rendered raw ids — the case that actually matters. It
also needed two effects and a serialized-context key to avoid leaking labels
across a context change, all of which disappear: queries key on the context, so a
label cannot outlive it.

Gates the speculative lookup of typed text on a new `resolvesUnknownIds` flag,
set only where `fetchById` returns null for an id that does not exist. Most
implementations resolve a record by id, so every partial keystroke was a failed
upstream request, retried once, and its error made "could not check that exact
value" the normal empty state on those selectors.

Also: pass handlers straight to the combobox rather than through identity
wrappers that defeated its memo, move the latest-callback ref write into an
effect, rename the raw search setter so the deduping write path cannot be
bypassed, and correct the prop and staleTime docs.

Author: waleedlatif1

apps/sim/app/api/tools/confluence/selector-spaces/route.ts

@@ -23,6 +23,27 @@ const PAGE_LIMIT = 250
 
 type SpaceStatus = 'current' | 'archived'
 
+/** A row as Confluence returns it. `status` is not marked required in the v2 schema. */
+interface SpaceRow {
+  id: string
+  name: string
+  key: string
+  status?: SpaceStatus
+}
+
+interface SpacesResponse {
+  results?: SpaceRow[]
+  _links?: { next?: string }
+}
+
+/** A row as this selector emits it, with `status` always resolved. */
+interface SelectorSpace {
+  id: string
+  name: string
+  key: string
+  status: SpaceStatus
+}
+
 /**
  * Cursor format: `<status>:<innerCursor>`. Empty inner cursor means "first page
  * of that status". When current is exhausted we hand back `archived:` so the
@@ -45,7 +66,7 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
     const parsed = await parseRequest(confluenceSpacesSelectorContract, request, {})
     if (!parsed.success) return parsed.response
 
-    const { credential, workflowId, domain, cursor } = parsed.data.body
+    const { credential, workflowId, domain, cursor, spaceKey } = parsed.data.body
 
     if (!credential) {
       logger.error('Missing credential in request')
@@ -101,32 +122,73 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
     const baseUrl = `https://api.atlassian.com/ex/confluence/${cloudIdValidation.sanitized}/wiki/api/v2/spaces`
     const { status, inner } = parseCursor(cursor)
 
-    const params = new URLSearchParams({ limit: String(PAGE_LIMIT), status })
-    if (inner) params.set('cursor', inner)
-    const url = `${baseUrl}?${params.toString()}`
+    const requestSpaces = async (
+      search: URLSearchParams
+    ): Promise<{ ok: true; data: SpacesResponse } | { ok: false; response: NextResponse }> => {
+      const response = await fetch(`${baseUrl}?${search.toString()}`, {
+        method: 'GET',
+        headers: { Accept: 'application/json', Authorization: `Bearer ${accessToken}` },
+      })
+
+      if (!response.ok) {
+        const errorText = await response.text()
+        const message = parseAtlassianErrorMessage(response.status, response.statusText, errorText)
+        logger.error('Confluence API error response', { error: message, status: response.status })
+        return { ok: false, response: NextResponse.json({ error: message }, { status: 502 }) }
+      }
 
-    const response = await fetch(url, {
-      method: 'GET',
-      headers: { Accept: 'application/json', Authorization: `Bearer ${accessToken}` },
-    })
+      return { ok: true, data: await response.json() }
+    }
 
-    if (!response.ok) {
-      const errorText = await response.text()
-      const message = parseAtlassianErrorMessage(response.status, response.statusText, errorText)
-      logger.error('Confluence API error response', { error: message, status: response.status })
-      return NextResponse.json({ error: message }, { status: 502 })
+    const toSpaces = (rows: SpaceRow[] | undefined, queried: SpaceStatus): SelectorSpace[] =>
+      (rows ?? []).map((space) => ({
+        id: space.id,
+        name: space.name,
+        key: space.key,
+        // Trust the row's own status; fall back to the status this call asked for.
+        status: space.status ?? queried,
+      }))
+
+    if (spaceKey) {
+      /**
+       * Exact-key lookup, bypassing the paged drain the dropdown otherwise depends
+       * on. Both statuses are queried explicitly rather than omitting `status`: it
+       * takes a single value on this endpoint (unlike `/pages`, where it is an array
+       * with a documented `current,archived` default) and `/spaces` documents no
+       * default, while archived spaces are reachable in the paged path and sync works
+       * against them. Concurrent because the key is user-typed text, so a miss — which
+       * dominates while typing — would otherwise pay two round-trips.
+       */
+      const [current, archived] = await Promise.all([
+        requestSpaces(
+          new URLSearchParams({ keys: spaceKey, limit: String(PAGE_LIMIT), status: 'current' })
+        ),
+        requestSpaces(
+          new URLSearchParams({ keys: spaceKey, limit: String(PAGE_LIMIT), status: 'archived' })
+        ),
+      ])
+      // Only a total failure is fatal: one leg erroring must not discard a match
+      // the other leg found.
+      if (!current.ok && !archived.ok) return current.response
+
+      // A single resolution, never a page in a drained stream, so no cursor.
+      return NextResponse.json({
+        spaces: [
+          ...(current.ok ? toSpaces(current.data.results, 'current') : []),
+          ...(archived.ok ? toSpaces(archived.data.results, 'archived') : []),
+        ],
+        nextCursor: undefined,
+      })
     }
 
-    const data = await response.json()
-    const spaces = (data.results || []).map((space: { id: string; name: string; key: string }) => ({
-      id: space.id,
-      name: space.name,
-      key: space.key,
-      status,
-    }))
+    const params = new URLSearchParams({ limit: String(PAGE_LIMIT), status })
+    if (inner) params.set('cursor', inner)
+    const result = await requestSpaces(params)
+    if (!result.ok) return result.response
+    const data = result.data
 
     let nextInner: string | undefined
-    const nextLink = data._links?.next as string | undefined
+    const nextLink = data._links?.next
     if (nextLink) {
       try {
         nextInner = new URL(nextLink, 'https://placeholder').searchParams.get('cursor') || undefined
@@ -142,7 +204,7 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
       nextCursor = 'archived:'
     }
 
-    return NextResponse.json({ spaces, nextCursor })
+    return NextResponse.json({ spaces: toSpaces(data.results, status), nextCursor })
   } catch (error) {
     logger.error('Error listing Confluence spaces:', error)
     return NextResponse.json(

apps/sim/app/workspace/[workspaceId]/knowledge/[id]/components/connector-selector-field/connector-selector-field.tsx

@@ -1,16 +1,23 @@
 'use client'
 
-import { useMemo } from 'react'
+import { useMemo, useState } from 'react'
 import { ChipCombobox, type ComboboxOption, Loader } from '@sim/emcn'
+import { SEARCH_DEBOUNCE_MS } from '@/lib/url-state'
 import { SELECTOR_CONTEXT_FIELDS } from '@/lib/workflows/subblocks/context'
 import type {
   ConfigFieldMap,
   ConfigFieldValue,
 } from '@/app/workspace/[workspaceId]/knowledge/[id]/hooks/use-connector-config-fields'
 import { getDependsOnFields } from '@/blocks/utils'
 import type { ConnectorConfigField } from '@/connectors/types'
+import { getSelectorDefinition } from '@/hooks/selectors/registry'
 import type { SelectorContext, SelectorKey } from '@/hooks/selectors/types'
-import { useSelectorOptions } from '@/hooks/selectors/use-selector-query'
+import {
+  useSelectorOptionDetail,
+  useSelectorOptionDetails,
+  useSelectorOptions,
+} from '@/hooks/selectors/use-selector-query'
+import { useDebounce } from '@/hooks/use-debounce'
 
 interface ConnectorSelectorFieldProps {
   field: ConnectorConfigField & { selectorKey: SelectorKey }
@@ -34,6 +41,7 @@ export function ConnectorSelectorField({
   disabled,
 }: ConnectorSelectorFieldProps) {
   const isMulti = Boolean(field.multi)
+  const [searchTerm, setSearchTerm] = useState('')
 
   const context = useMemo<SelectorContext>(() => {
     const ctx: SelectorContext = {}
@@ -62,15 +70,68 @@ export function ConnectorSelectorField({
   }, [field.dependsOn, sourceConfig, configFields, canonicalModes])
 
   const isEnabled = !disabled && !!credentialId && depsResolved
-  const { data: options = [], isLoading } = useSelectorOptions(field.selectorKey, {
+  const {
+    data: options = [],
+    isLoading,
+    hasMore,
+    isFetchingMore,
+    truncated,
+    error,
+  } = useSelectorOptions(field.selectorKey, {
     context,
     enabled: isEnabled,
   })
 
-  const comboboxOptions = useMemo<ComboboxOption[]>(
-    () => options.map((opt) => ({ label: opt.label, value: opt.id })),
-    [options]
+  /**
+   * Label every selected value, including values restored from saved config that no
+   * in-session search would have resolved. Queries are keyed on `context`, so a label
+   * can never outlive the context that produced it, and they share keys with the
+   * speculative lookup below so an already-resolved id costs no extra request.
+   */
+  const singleValue = Array.isArray(value) ? value[0] : value
+  const selectedIds = useMemo(
+    () => (Array.isArray(value) ? value : value ? [value] : []).filter(Boolean),
+    [value]
   )
+  const selectedOptions = useSelectorOptionDetails(field.selectorKey, {
+    context,
+    detailIds: isEnabled ? selectedIds : [],
+  })
+
+  /**
+   * The option list fills by draining pages in the background and the combobox filters
+   * it client-side, so an option is only findable once its page has arrived. Where the
+   * selector's `fetchById` tolerates an unknown id, whatever the user typed is resolved
+   * directly so an exact key is selectable immediately. Gated on that flag because most
+   * implementations resolve a record by id, where a partial keystroke is a guaranteed
+   * failed upstream request rather than an empty result.
+   */
+  const resolvesUnknownIds = Boolean(getSelectorDefinition(field.selectorKey).resolvesUnknownIds)
+  const debouncedSearch = useDebounce(searchTerm.trim(), SEARCH_DEBOUNCE_MS)
+  const { data: searchedOption } = useSelectorOptionDetail(field.selectorKey, {
+    context,
+    detailId:
+      resolvesUnknownIds && isEnabled && debouncedSearch.length > 0 ? debouncedSearch : undefined,
+  })
+
+  const emptyMessage = getEmptyMessage(field.title.toLowerCase(), {
+    error,
+    hasMore,
+    isFetchingMore,
+    truncated,
+  })
+
+  const comboboxOptions = useMemo<ComboboxOption[]>(() => {
+    const base = options.map((opt) => ({ label: opt.label, value: opt.id }))
+    const seen = new Set(base.map((opt) => opt.value))
+    const extras: ComboboxOption[] = []
+    for (const option of searchedOption ? [...selectedOptions, searchedOption] : selectedOptions) {
+      if (seen.has(option.id)) continue
+      seen.add(option.id)
+      extras.push({ label: option.label, value: option.id })
+    }
+    return extras.length > 0 ? [...extras, ...base] : base
+  }, [options, selectedOptions, searchedOption])
 
   if (isLoading && isEnabled) {
     return (
@@ -88,8 +149,9 @@ export function ConnectorSelectorField({
         multiSelect
         options={comboboxOptions}
         multiSelectValues={multiValues}
-        onMultiSelectChange={(values) => onChange(values)}
+        onMultiSelectChange={onChange}
         searchable
+        onSearchChange={setSearchTerm}
         searchPlaceholder={`Search ${field.title.toLowerCase()}...`}
         placeholder={
           !credentialId
@@ -99,18 +161,18 @@ export function ConnectorSelectorField({
               : field.placeholder || `Select ${field.title.toLowerCase()}`
         }
         disabled={disabled || !credentialId || !depsResolved}
-        emptyMessage={`No ${field.title.toLowerCase()} found`}
+        emptyMessage={emptyMessage}
       />
     )
   }
 
-  const singleValue = Array.isArray(value) ? value[0] : value
   return (
     <ChipCombobox
       options={comboboxOptions}
       value={singleValue || undefined}
-      onChange={(next) => onChange(next)}
+      onChange={onChange}
       searchable
+      onSearchChange={setSearchTerm}
       searchPlaceholder={`Search ${field.title.toLowerCase()}...`}
       placeholder={
         !credentialId
@@ -120,11 +182,36 @@ export function ConnectorSelectorField({
             : field.placeholder || `Select ${field.title.toLowerCase()}`
       }
       disabled={disabled || !credentialId || !depsResolved}
-      emptyMessage={`No ${field.title.toLowerCase()} found`}
+      emptyMessage={emptyMessage}
     />
   )
 }
 
+/**
+ * Only visible once the first page has landed (`isLoading` renders a spinner
+ * before that), so "no match" here means no match among the options drained
+ * *so far* — a flat "none found" would wrongly read as "does not exist".
+ *
+ * `error` is checked before `hasMore`: a failed page halts the drain but leaves
+ * `hasMore` set, which would otherwise claim to be loading forever.
+ */
+function getEmptyMessage(
+  noun: string,
+  state: {
+    error: Error | null
+    hasMore: boolean
+    isFetchingMore: boolean
+    truncated: boolean
+  }
+): string {
+  if (state.error) return 'No match — the list failed to load. Try reopening'
+  if (state.hasMore || state.isFetchingMore) return 'No match yet — still loading…'
+  if (state.truncated) return 'No match — too many to list. Try a more exact term'
+  // `noun` is singular on some connectors ("Base") and plural on others ("Spaces"),
+  // so only this settled message puts it behind a quantifier.
+  return `No ${noun} found`
+}
+
 function resolveDepValue(
   depFieldId: string,
   configFields: ConnectorConfigField[],

apps/sim/hooks/selectors/providers/confluence/selectors.ts

@@ -49,11 +49,12 @@ export const confluenceSelectors = {
         nextCursor: data.nextCursor,
       }
     },
+    /** The server filters by key and returns nothing for a key that does not exist. */
+    resolvesUnknownIds: true,
     /**
-     * Resolves a single space label. Hits only the first page — the dropdown's
-     * `fetchPage` stream populates the options cache for spaces beyond page 1,
-     * and `useSelectorOptionMap` merges them in. Walking all pages here would
-     * double API load since the stream is already running in parallel.
+     * Resolves a single space by key via the server's exact-key lookup, independent
+     * of how far the page drain has run — a space sorting beyond page 1 would
+     * otherwise never resolve, which on a large site is most of them.
      */
     fetchById: async ({ context, detailId, signal }: SelectorQueryArgs) => {
       if (!detailId) return null
@@ -64,6 +65,7 @@ export const confluenceSelectors = {
           credential: credentialId,
           workflowId: context.workflowId,
           domain,
+          spaceKey: detailId,
         },
         signal,
       })
@@ -87,6 +89,16 @@ export const confluenceSelectors = {
       search ?? '',
     ],
     enabled: ({ context }) => Boolean(context.oauthCredential && context.domain),
+    /**
+     * Deliberately a single request, not a drain. `/pages` is cursor-paginated and
+     * this list is therefore capped at `limit`, which is a real gap — but draining it
+     * is worse: with no search term `title` is unset, so the drain would walk the
+     * entire site (up to `MAX_AUTO_DRAIN_PAGES` requests) every time the dropdown
+     * opens, and the route does not forward an abort signal upstream, so superseded
+     * drains still bill the tenant's rate limit. Fixing this properly needs
+     * server-side search whose `title` semantics have been confirmed against a live
+     * instance, not brute-force loading.
+     */
     fetchList: async ({ context, search, signal }: SelectorQueryArgs) => {
       const credentialId = ensureCredential(context, 'confluence.pages')
       const domain = ensureDomain(context, 'confluence.pages')

apps/sim/hooks/selectors/types.ts

@@ -136,6 +136,13 @@ export interface SelectorDefinition {
    */
   fetchPage?: (args: SelectorPageArgs) => Promise<SelectorPage>
   fetchById?: (args: SelectorQueryArgs) => Promise<SelectorOption | null>
+  /**
+   * Set when `fetchById` tolerates an id that may not exist, returning `null` rather
+   * than erroring. Only then is it safe to speculatively resolve whatever a user has
+   * typed — most implementations resolve a record by id and would turn every partial
+   * keystroke into a failed upstream request.
+   */
+  resolvesUnknownIds?: boolean
   enabled?: (args: SelectorQueryArgs) => boolean
   staleTime?: number
 }