release: chdb 3.1.0-rc.1 — raw/streaming insert on chdb-core v26.5.1-rc.1#44
Merged
Merged
Conversation
This was referenced Jun 14, 2026
Raw passthrough (insert({ values: Buffer|Uint8Array|string, format })):
- native InsertRawWorker: payload pinned via Persistent reference, prefix
assembly + execution on the libuv thread (the main thread never copies
or scans the payload), binary-safe via chdb_query_n (NUL bytes survive)
- two row ledgers: rowsSent (payload view, native non-empty-line scan for
line-delimited formats) and rowsWritten/bytesWritten (engine-reported
write progress via chdb_result_rows_written/bytes_written,
chdb-io/chdb-core#88; includes cascaded materialized-view writes - same
semantics as the HTTP X-ClickHouse-Summary)
- per-insert SETTINGS channel, columns list, abort/timeout with honest
single-shot semantics; the engine's "(at row N)" parsed into failedAtRow
Streaming input (insert({ values: Readable|AsyncIterable, format })):
- pull-based for-await loop, at most one bounded chunk in flight; cuts
only at raw newlines (line-delimited formats only - CSV/WithNames are
rejected with the single-shot workaround in the message)
- failures are typed errors that always settle the promise: source-error,
stall (opt-in stallTimeout), backpressure-overflow, write-failure with
ABSOLUTE failedAtRow, row-too-large, abort; all carry progress snapshots
- onProgress per flushed chunk; at-least-once documented (flushed chunks
are not rolled back)
Acceptance gates (CHDB_TEST_HUGE) all green locally against a libchdb
build that exposes write progress: 256MB insert with event-loop jitter
p99 < 10ms, 1GB Buffer (past the V8 string ceiling), 4GB stream with
bounded RSS, flat-RSS leak gate. 140 existing v3 tests unaffected. CI lib
pin bump to the chdb-core release carrying #88 is a follow-up (local
build used here).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pre-release test line carrying the raw/streaming insert API (the previous commit), for the Langfuse OTEL ingestion-buffer cookbook. Built on the v26.5.1-rc.1 engine, which has the written-rows accessors that API needs and that the v26.5.0 stable line lacks. - update_libchdb.sh -> v26.5.1-rc.1 (valid-semver pre-release tag; the prior v26.5.1rc1 tag was removed upstream). - version 3.1.0-rc.1 -> publishes under npm dist-tag 'next', leaving 'latest' (chdb@3.0.0) untouched; install via 'npm i chdb@next' or chdb@3.1.0-rc.1. - optionalDependencies pin the native subpackages to 26.5.1-rc.1, matching the engine version the publish workflow derives and publishes. Inherits the publish-pipeline hardening already on main (macos-15-intel runner, idempotent/conflict-tolerant publish, id-token provenance, and the test-session teardown that fixed the macos-15-intel flake). Full v3 suite green on v26.5.1-rc.1: 155 passed, 4 heavy-gated skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
ShawnChen-Sirius
force-pushed
the
feat/p0-raw-insert
branch
from
bccc8e1 to
8440f38
Compare
ShawnChen-Sirius
changed the title
The v3 suite went all-red on every x64 CI runner (ubuntu-latest and
macos-15-intel, all Node versions) with every query after the first
failure reporting "Code: 236 ... server is shutting down due to a fatal
error (ABORTED)". arm64 runners usually stayed green.
Root cause is a test-isolation race against libchdb's single-connection,
no-interrupt model, not an engine bug:
- abort/timeout settle the JS promise early, but the C ABI has no
interrupt, so the native query/insert keeps running on the libuv thread
("the write may still complete", as the timeout test name admits).
- The describe-local afterEach then calls session.close(), which destroys
the native connection while that op is still in flight. Closing a
connection out from under a running op aborts the in-process engine for
the rest of the process, and because the whole suite shares one fork and
one connection (singleFork), every subsequent test cascades to ABORTED.
- It only reproduced on x64 because the crash is timing-dependent: on x64
runners the 1ms-timeout 200k-row insert was reliably still running when
close() landed; arm runners (and the author's local box) usually won the
race. macos-14/22 failing intermittently confirms it is timing, not arch.
Fix:
- Track native ops that were settled early by abort/timeout.
- close() defers the native CloseConnection until those ops drain instead
of racing them; a new session is never created before the prior
connection is fully released.
- The v3 setup afterEach drains pending ops before closing sessions, so an
early-settled op stays local to the test that started it.
Full suite now green on linux-x86_64: v2 mocha 24 passing, v3 vitest
155 passed / 4 skipped, typecheck clean. The new async-stress lifecycle
test "closing a session while its async query is in flight does not crash
(100x)" exercises the close() path directly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
wudidapaopao
approved these changes
Jun 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
chdb@3.1.0-rc.1pre-release test line: the raw/streaminginsert()API on the v26.5.1-rc.1 engine, for the Langfuse OTEL ingestion-buffer cookbook (chdb-io/cookbook#9). Rebased onto currentmain, so it inherits everything that landed while 3.0.0 shipped.Contents (2 commits on top of main)
feat(l1): raw passthrough insert + backpressured stream input—insert({ values: Buffer | Readable, format }):JSON.parseon the main thread, no V8 ~512 MB string ceiling.maxChunkBytesbounds resident memory to O(chunk), source paused per in-flight chunk so backpressure reaches the client's TCP window,stallTimeoutfails a silent producer.failedAtRow,reasontaxonomy), atomic batches, exactUInt64ns timestamps, losslessbigint. Requires chdb-core #88's written-rows accessors (present in v26.5.1-rc.1, absent in v26.5.0 — which is why this is a separate line from 3.0.0).release: chdb 3.1.0-rc.1 on chdb-core v26.5.1-rc.1:update_libchdb.sh→v26.5.1-rc.1.version3.1.0-rc.1→ publishes under npm dist-tagnext, leavinglatest(chdb@3.0.0) untouched. Install vianpm i chdb@nextorchdb@3.1.0-rc.1.optionalDependenciespin the native subpackages to26.5.1-rc.1(matches the engine version the publish workflow derives).Test
Full v3 suite green against v26.5.1-rc.1: 155 passed / 4 heavy-gated skipped.
Inherits the publish-pipeline hardening already on main (macos-15-intel runner, idempotent/conflict-tolerant publish, id-token provenance, and the test-session teardown that fixed the macos-15-intel flake), so the
v3.1.0-rc.1tag publish runs clean.