investigate: COPY FROM STDIN broken in remote-worker (multitenant) mode

[fuziontech]

TL;DR

COPY FROM STDIN from any client (Fivetran, dbt, psql \copy, etc.) is broken in the remote-worker / multitenant K8s topology and has been since that backend shipped. Every COPY fails with:

IO Error: No files found that match the pattern "/tmp/duckgres-copy-*.csv"

This is architectural, not a transient bug. The control plane writes the spool file to its own pod's filesystem and asks the worker — which is a separate pod with no shared volume — to read from that path.

This PR does not yet fix it. It documents the limitation in code and improves the error so operators stop chasing the symptom. Fix options are discussed below.

Reproduction (production, 2026-05-07 ~20:50 UTC)

From the posthog_data_import user, control plane pod duckgres-67d4574c84-bt6rw, EKS posthog-mw-prod-us:

COPY "billing_public_bil-staging-..." (...) FROM '/tmp/duckgres-copy-3578801345.csv' (FORMAT CSV, AUTO_DETECT FALSE, ...)
→ flight execute update: rpc error: code = InvalidArgument desc = failed to execute update:
   IO Error: No files found that match the pattern "/tmp/duckgres-copy-3578801345.csv"

Three different staging tables in the same Fivetran sync hit the same error within the same second. Every Fivetran sync since the migration to MTCP has been losing rows on this path silently (CP returns the COPY error to the client, but Fivetran retries and eventually moves on with the underlying tables empty).

Root cause

server/conn.go:3848 (now annotated) creates the spool tempfile via os.CreateTemp("", "duckgres-copy-*.csv") — i.e. in $TMPDIR of the control plane pod, which resolves to /tmp in the duckgres-cp container.

server/conn.go:3908 then constructs COPY <table> FROM '<tmpPath>' ... and ships it to the executor:

copySQL := BuildDuckDBCopyFromSQL(tableName, columnList, tmpPath, opts)
result, err := c.executor.Exec(copySQL)

In remote-backend mode c.executor is a flightclient.FlightExecutor that forwards the SQL to a worker pod over Arrow Flight SQL DoPutCommandStatementUpdate (duckdbservice/flight_handler.go:508). The worker calls session.Conn.ExecContext(ctx, query) against its local DuckDB, which opens the path against the worker pod's filesystem.

The two pods do not share a volume:

$ kubectl -n duckgres get deploy duckgres -o json | jq '.spec.template.spec.volumes'
[{"emptyDir": {}, "name": "certs"}]   # CP: just /app/certs

$ kubectl -n duckgres get pod duckgres-67d4574c84-worker-XXXXX -o json | jq '.spec.volumes'
[{"emptyDir": {}, "name": "data"}, ...] # worker: /data, separate emptyDir

So the tempfile is invisible to the worker and the COPY fails.

The author of flight_executor.go actually flagged this:

// server/flightclient/flight_executor.go:269
func (e *FlightExecutor) ConnContext(ctx context.Context) (sqlcore.RawConn, error) {
    return nil, fmt.Errorf("ConnContext not supported in Flight mode (use batched INSERT for COPY FROM)")
}

…but the COPY FROM STDIN path in conn.go does not detect Flight mode and does not have a batched-INSERT fallback. It just hands a local-path COPY FROM to the executor.

Why it works in standalone / process-backend

• Standalone: CP and DuckDB are the same process, same filesystem.
• Process-backend control plane: workers are local child processes on the same host; /tmp is shared.
• Remote-backend control plane (this bug): workers are pods on different nodes, with no shared mount.

Why no test caught it

server/conn_test.go has thorough coverage of COPY FROM STDIN (regex, blob fallback, multi-line quoted fields, etc.), but every test runs against an in-process DuckDB executor. There is no integration test that runs COPY FROM STDIN end-to-end through the remote Flight executor against a separate worker process. Adding one would have caught this immediately.

Fix options

In rough order of "cleanest first":

1. Arrow Flight SQL CommandStatementIngest (preferred)

Apache Arrow Flight SQL added a dedicated bulk ingest command (Arrow ≥17 / FlightSQL spec rev). The CP would parse the CSV stream into Arrow record batches and DoPut them into a transient table on the worker, then run INSERT … SELECT … (or use the ingest command's transactional semantics directly). Pros: standard, no custom RPC. Cons: requires confirming the Go bindings support CommandStatementIngest and adding the worker handler.

2. Custom DoPut for raw CSV bytes

Add a duckgres-specific FlightDescriptor that streams CSV bytes to the worker; the worker writes them to its own /tmp and runs the existing COPY FROM <local-path>. Pros: minimal change to the DuckDB-side execution (reuses the optimized parser). Cons: a custom RPC the team has to maintain.

3. Stage to S3 / HTTPFS

The CP writes the CSV to a known S3 prefix (one bucket per org or global), and rewrites COPY FROM 'stdin' to COPY FROM 's3://…'. Workers already have S3 credentials for DuckLake. Pros: zero code on the worker. Cons: S3 write+read latency for every COPY (Fivetran ships many small batches), and a cleanup story for orphaned CSVs.

4. Batched INSERT

What the existing flight_executor.go comment hints at: parse the CSV in CP, send INSERT INTO … VALUES (…), (…) chunks to the worker. Pros: works today over the existing executor. Cons: bypasses DuckDB's parallel CSV parser; ~10–100x slower for big batches, which is exactly what Fivetran sends.

Recommendation: option 1 if CommandStatementIngest is wired through in our Arrow version, otherwise option 2.

What this PR contains

• A documented comment at the spool-file creation site (server/conn.go) describing the limitation and pointing readers to the relevant remote handler.
• An error-message wrap so operators see why the COPY failed (mentions remote-worker mode and recommended workarounds), instead of a raw "No files found" pattern error.
• This RCA in the PR description.

No behavioral change to any working topology. No fix yet — that's a follow-up.

Test plan

• go build ./... and go build -tags kubernetes ./...
• go test ./server/... (existing COPY tests still pass; they exercise the in-process path which works fine)
• Reproduce in staging MTCP, confirm the new error message surfaces.
• Spec out the chosen fix option in a follow-up issue/PR.

🤖 Generated with Claude Code

[fuziontech]

Superseded by the actual fix in #NEW_PR (real implementation, not just diagnostics). Will update with link.

[fuziontech]

Superseded by the actual fix in #552 — that PR streams CSV bytes through Flight DoPut to the worker pod instead of relying on a shared filesystem.