fix(tables): sniff CSV delimiter and capture the real header row on import (#5888)

* fix(tables): sniff CSV delimiter and capture the real header row on import

Table CSV import derived the delimiter purely from the file extension, so
semicolon-delimited exports (European-locale Excel) parsed as a single column.
Header derivation also read Object.keys(rows[0]), so with relax_column_count a
ragged/sparse first data row dropped its trailing columns from schema inference.

- Add detectCsvDelimiter: trial-parse the head against ,/;/tab/| and pick the
  candidate with the most columns (tie-break on row-width consistency), quote-aware
  so separators inside quoted cells don't skew the guess; extension is now only the
  fallback
- Add sniffCsvDelimiterFromStream: memory-bounded (64KB) peek-then-replay for the
  streaming import paths so multi-GB files sniff without buffering
- Capture the true header row via the csv-parse columns callback on every path,
  fixing dropped columns when the first row is short
- Wire into the create route, append/replace route, background runner, client
  dialog preview, and parseFileRows (copilot)

* fix(tables): rank CSV delimiter by row consistency and bound the peek buffer

Addresses review findings on the delimiter sniffer:

- Rank candidates by row-width consistency first (modal column count), then column
  count. Ranking on column count alone let a comma that appears in a semicolon
  file's unquoted header win over the real separator; the wide-but-ragged split
  now loses to the uniform one.
- Move the partial-trailing-line trim into detectCsvDelimiter so every path
  (stream, client preview, parseFileRows) prepares the sniff sample identically
  and can't disagree on the delimiter.
- Cap the stream peeker's detection copy at the sniff window via Buffer.concat's
  length arg and replay the buffered chunks by reference, so an oversized source
  chunk can't break the bounded-memory contract.

* fix(tables): score CSV delimiter by width*consistency and dedupe headers

Addresses round-2 review findings:

- Score each delimiter candidate by modalWidth * consistency instead of
  consistency alone. Consistency-only let a separator that appears uniformly
  inside values (e.g. a pipe once per row) beat a real delimiter whose rows are
  legitimately ragged; the product rewards a split that is both wide and uniform,
  with width breaking ties. Genuinely symmetric files (two valid columns under
  either separator) fall back to the global-default candidate order.
- Dedupe the captured header row to match the parser's record keys. csv-parse
  collapses duplicate column names onto one key (last value wins), so reporting
  the raw duplicates made schema inference invent a phantom empty column and
  could fail mapping validation. dedupeHeaders keeps first-occurrence order,
  case-sensitively, matching the emitted keys.

* fix(tables): keep the final row when sniffing a complete CSV

detectCsvDelimiter always trimmed to the last newline, so a complete file with
no trailing newline lost its final data row from scoring — leaving the header
alone and letting a header-widening separator beat the real one. It now takes a
`complete` flag: the trim runs only for truncated prefixes, where a mid-record
cut must be dropped. The stream sniffer passes it from `exhausted`, and the
buffered callers (client preview, parseFileRows) pass it when the sample covers
the whole file.

* fix(tables): keep the double-cast annotation adjacent to its cast

Hoist the csv-parse options out of the multi-line parse() call so the
`// double-cast-allowed` annotation sits directly above the `as unknown as`
token — the strict boundary audit only matches an annotation within three lines
of the cast. Also simplify the stream sniffer's completeness check to `exhausted`
(the loop only ends early on end-of-stream, so it already means the whole file
fit the sniff window).

* fix(tables): observe EOF at the exact sniff-window boundary

The stream sniffer's read loop stopped as soon as the buffered size reached the
sniff window, so a file whose size is exactly the window never triggered the
extra read that observes end-of-stream — it was judged a truncated prefix and
dropped its final newline-less row, disagreeing with the buffered callers (which
mark that size complete). Read while size <= window so the boundary case sees EOF
and `exhausted` (hence `complete`) is set correctly. Regression test added.

* test(tables): use sleep helper instead of raw setTimeout promise

Author: waleedlatif1

apps/sim/app/api/table/[tableId]/import/route.ts

@@ -36,6 +36,7 @@ import {
   validateMapping,
   wouldExceedRowLimit,
 } from '@/lib/table'
+import { sniffCsvDelimiterFromStream } from '@/lib/table/csv-delimiter-stream'
 import { importAppendRows, importReplaceRows } from '@/lib/table/import-data'
 import { getUserSettings } from '@/lib/users/queries'
 import {
@@ -176,19 +177,26 @@ export const POST = withRouteHandler(async (request: NextRequest, { params }: Ro
       timezone = timezoneValidation.data
     }
 
-    const delimiter = extensionValidation.data === 'tsv' ? '\t' : ','
-    const parser = createCsvParser(delimiter)
+    // The extension only picks the fallback — the separator is sniffed from the file's
+    // head so semicolon/pipe exports (European-locale Excel) don't land in one column.
+    const { delimiter, stream: csvStream } = await sniffCsvDelimiterFromStream(
+      file.stream,
+      extensionValidation.data === 'tsv' ? '\t' : ','
+    )
+    let headers: string[] = []
+    const parser = createCsvParser(delimiter, (parsedHeaders) => {
+      headers = parsedHeaders
+    })
     // `.pipe` doesn't forward source errors; forward them so the iterator throws.
-    file.stream.on('error', (streamErr) => parser.destroy(streamErr))
-    file.stream.pipe(parser)
+    csvStream.on('error', (streamErr) => parser.destroy(streamErr))
+    csvStream.pipe(parser)
     const rows: Record<string, unknown>[] = []
     for await (const record of parser as AsyncIterable<Record<string, unknown>>) {
       rows.push(record)
     }
     if (rows.length === 0) {
       return NextResponse.json({ error: 'CSV file has no data rows' }, { status: 400 })
     }
-    const headers = Object.keys(rows[0])
 
     let effectiveMapping = mapping ?? buildAutoMapping(headers, table.schema)
     let prospectiveTable: TableDefinition = table

apps/sim/app/api/table/import-csv/route.ts

@@ -26,6 +26,7 @@ import {
   type TableDefinition,
   type TableSchema,
 } from '@/lib/table'
+import { sniffCsvDelimiterFromStream } from '@/lib/table/csv-delimiter-stream'
 import { getUserSettings } from '@/lib/users/queries'
 import { getUserEntityPermissions } from '@/lib/workspaces/permissions/utils'
 import {
@@ -107,12 +108,20 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
         { status: 400 }
       )
     }
-    const delimiter = extensionResult.data === 'tsv' ? '\t' : ','
+    // The extension only picks the fallback — the separator is sniffed from the file's
+    // head so semicolon/pipe exports (European-locale Excel) don't land in one column.
+    const { delimiter, stream: csvStream } = await sniffCsvDelimiterFromStream(
+      file.stream,
+      extensionResult.data === 'tsv' ? '\t' : ','
+    )
 
-    const parser = createCsvParser(delimiter)
+    let csvHeaders: string[] = []
+    const parser = createCsvParser(delimiter, (headers) => {
+      csvHeaders = headers
+    })
     // `.pipe` doesn't forward source errors; forward them so the iterator throws.
-    file.stream.on('error', (err) => parser.destroy(err))
-    file.stream.pipe(parser)
+    csvStream.on('error', (err) => parser.destroy(err))
+    csvStream.pipe(parser)
 
     interface ImportState {
       table: TableDefinition
@@ -139,7 +148,7 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
 
     /** Infer the schema from the buffered sample and create the (empty) table. */
     const buildTable = async (sampleRows: Record<string, unknown>[]): Promise<ImportState> => {
-      const inferred = inferSchemaFromCsv(Object.keys(sampleRows[0]), sampleRows)
+      const inferred = inferSchemaFromCsv(csvHeaders, sampleRows)
       const schema: TableSchema = { columns: inferred.columns.map(normalizeColumn) }
       const planLimits = await getWorkspaceTableLimits(workspaceId)
       const tableName = sanitizeName(file.filename.replace(/\.[^.]+$/, ''), 'imported_table').slice(

apps/sim/app/workspace/[workspaceId]/tables/components/import-csv-dialog/import-csv-dialog.tsx

@@ -25,7 +25,13 @@ import { createLogger } from '@sim/logger'
 import { getErrorMessage } from '@sim/utils/errors'
 import { truncate } from '@sim/utils/string'
 import { CSV_ASYNC_IMPORT_THRESHOLD_BYTES } from '@/lib/table/constants'
-import { buildAutoMapping, parseCsvBuffer } from '@/lib/table/import'
+import {
+  buildAutoMapping,
+  CSV_DELIMITER_SNIFF_BYTES,
+  type CsvDelimiter,
+  detectCsvDelimiter,
+  parseCsvBuffer,
+} from '@/lib/table/import'
 import type { TableDefinition } from '@/lib/table/types'
 import {
   type CsvImportMode,
@@ -107,15 +113,27 @@ interface ParsedCsv {
   sampleRows: Record<string, unknown>[]
 }
 
-/** Parses the head of a CSV/TSV for the mapping + sample, dropping any truncated final line. */
-async function parseCsvPreview(file: File, delimiter: ',' | '\t') {
+/**
+ * Parses the head of a CSV/TSV for the mapping + sample, dropping any truncated final line.
+ *
+ * The separator is sniffed from the same leading bytes the server sniffs, so the mapping shown
+ * here always matches the columns the import will actually produce.
+ */
+async function parseCsvPreview(file: File, fallbackDelimiter: CsvDelimiter) {
   const sliced = file.size > CSV_PREVIEW_BYTES
   const blob = sliced ? file.slice(0, CSV_PREVIEW_BYTES) : file
   let bytes = new Uint8Array(await blob.arrayBuffer())
   if (sliced) {
     const lastNewline = bytes.lastIndexOf(0x0a)
     if (lastNewline > 0) bytes = bytes.subarray(0, lastNewline + 1)
   }
+  // The sniff sample is the whole file only when nothing was sliced off and it fits the window;
+  // otherwise it's a truncated prefix whose last line may be partial.
+  const delimiter = await detectCsvDelimiter(
+    bytes.subarray(0, CSV_DELIMITER_SNIFF_BYTES),
+    fallbackDelimiter,
+    { complete: !sliced && bytes.length <= CSV_DELIMITER_SNIFF_BYTES }
+  )
   return parseCsvBuffer(bytes, delimiter)
 }
 
@@ -180,8 +198,7 @@ export function ImportCsvDialog({
     setParsing(true)
     setParseError(null)
     try {
-      const delimiter: ',' | '\t' = ext === 'tsv' ? '\t' : ','
-      const { headers, rows } = await parseCsvPreview(file, delimiter)
+      const { headers, rows } = await parseCsvPreview(file, ext === 'tsv' ? '\t' : ',')
       const autoMapping = buildAutoMapping(headers, table.schema)
       setParsed({
         file,

apps/sim/lib/table/csv-delimiter-stream.ts

@@ -0,0 +1,80 @@
+import { Readable } from 'node:stream'
+import {
+  CSV_DELIMITER_SNIFF_BYTES,
+  type CsvDelimiter,
+  detectCsvDelimiter,
+} from '@/lib/table/import'
+
+export interface SniffedCsvStream {
+  delimiter: CsvDelimiter
+  /**
+   * The full file contents, replayed from byte zero. Use this in place of the
+   * source stream — the source has already been partially consumed.
+   */
+  stream: Readable
+}
+
+/**
+ * Reads the head of a CSV/TSV stream, sniffs its field separator, then returns a
+ * stream that replays the buffered head followed by the untouched remainder.
+ *
+ * Memory is bounded: it buffers only the source chunks needed to reach the
+ * {@link CSV_DELIMITER_SNIFF_BYTES} window (one chunk past it, at worst), and the
+ * detection sample it copies is capped at exactly that window regardless of how
+ * large a single upstream chunk is. Those buffered chunks are then replayed
+ * *by reference* — never re-copied — so a multi-GB import stays O(sniff window)
+ * plus the single in-flight chunk. {@link detectCsvDelimiter} drops any partial
+ * trailing line, so no newline trimming is needed here.
+ */
+export async function sniffCsvDelimiterFromStream(
+  source: Readable,
+  fallback: CsvDelimiter = ','
+): Promise<SniffedCsvStream> {
+  const reader = source[Symbol.asyncIterator]()
+  const chunks: Buffer[] = []
+  let size = 0
+  let exhausted = false
+
+  // Read until the buffered size *exceeds* the window (not just reaches it) or the stream ends.
+  // The `<=` is deliberate: a file whose size is exactly the window must still trigger one more
+  // read so EOF is observed and `exhausted` becomes true — otherwise it would be misjudged as a
+  // truncated prefix and disagree with the buffered callers (which pass complete for that size).
+  while (size <= CSV_DELIMITER_SNIFF_BYTES) {
+    const next = await reader.next()
+    if (next.done) {
+      exhausted = true
+      break
+    }
+    const chunk = Buffer.isBuffer(next.value) ? next.value : Buffer.from(next.value as Uint8Array)
+    chunks.push(chunk)
+    size += chunk.length
+  }
+
+  // Copy at most the sniff window for detection — `Buffer.concat`'s length arg truncates,
+  // so an oversized final chunk can't inflate this allocation past CSV_DELIMITER_SNIFF_BYTES.
+  const sample = Buffer.concat(chunks, Math.min(size, CSV_DELIMITER_SNIFF_BYTES))
+  // `exhausted` (the loop stopped on end-of-stream, never on exceeding the window) means the whole
+  // file fit in the window, so its last row must count even without a trailing newline. Otherwise
+  // the sample is a truncated prefix whose final line may be partial and should be dropped.
+  const delimiter = await detectCsvDelimiter(sample, fallback, { complete: exhausted })
+
+  const stream = Readable.from(
+    (async function* replay() {
+      // Replay the already-read chunks by reference (no re-copy), then drain the rest.
+      for (const chunk of chunks) yield chunk
+      if (exhausted) return
+      while (true) {
+        const next = await reader.next()
+        if (next.done) return
+        yield next.value
+      }
+    })()
+  )
+
+  // `Readable.from` closes the generator on destroy, which returns the source
+  // iterator — but an early destroy of the wrapper before the generator is
+  // pulled would otherwise leak the source's socket.
+  stream.on('close', () => source.destroy())
+
+  return { delimiter, stream }
+}

apps/sim/lib/table/import-runner.ts

@@ -19,6 +19,7 @@ import {
 } from '@/lib/table'
 import { assertRowCapacity, notifyTableRowUsage } from '@/lib/table/billing'
 import { withGeneratedColumnIds } from '@/lib/table/column-keys'
+import { sniffCsvDelimiterFromStream } from '@/lib/table/csv-delimiter-stream'
 import { appendTableEvent } from '@/lib/table/events'
 import {
   addImportColumns,
@@ -103,6 +104,11 @@ export async function runTableImport(payload: TableImportPayload): Promise<void>
     // Stream the file rather than buffering it — a ~1M-row import must never be held in memory.
     source = await downloadFileStream({ key: fileKey, context: 'workspace' })
 
+    // The kickoff route's extension-derived delimiter is only the fallback — the separator is
+    // sniffed from the file's head so semicolon/pipe exports don't collapse into one column.
+    const sniffed = await sniffCsvDelimiterFromStream(source, delimiter)
+    const csvStream = sniffed.stream
+
     // Append must continue after the existing rows; create/replace start empty. Read once up
     // front (the import is the table's sole writer) and assign contiguous positions / threaded
     // order keys from it.
@@ -123,11 +129,14 @@ export async function runTableImport(payload: TableImportPayload): Promise<void>
       },
     })
 
-    const parser = createCsvParser(delimiter)
+    let csvHeaders: string[] = []
+    const parser = createCsvParser(sniffed.delimiter, (headers) => {
+      csvHeaders = headers
+    })
     // `.pipe` doesn't forward source errors; forward so the iterator throws.
-    source.on('error', (err) => parser.destroy(err))
+    csvStream.on('error', (err) => parser.destroy(err))
     byteCounter.on('error', (err) => parser.destroy(err))
-    source.pipe(byteCounter).pipe(parser)
+    csvStream.pipe(byteCounter).pipe(parser)
 
     let schema: TableSchema | null = null
     let headerToColumn: Map<string, string> | null = null
@@ -142,7 +151,7 @@ export async function runTableImport(payload: TableImportPayload): Promise<void>
      * map onto the existing schema, optionally auto-creating `createColumns` first.
      */
     const resolveSetup = async () => {
-      const headers = Object.keys(sample[0])
+      const headers = csvHeaders
 
       if (mode === 'create') {
         const inferred = inferSchemaFromCsv(headers, sample)