feat(qwp): add browser and Node.js QWP client - #62
Open
glasstiger wants to merge 177 commits into
Open
Conversation
…sion prepare() checked the row count against MAX_ROWS_PER_BATCH and the column count against QWP_MAX_COLUMNS_PER_TABLE, each against its own constant and neither against the bytes received. Their product does not have to be reachable: 1,048,576 rows of 2,048 columns is 2.1 billion cells. decode() then allocates two rowCount-length arrays per column -- the null layout and the expanded values -- measured at 16 bytes per cell. QWP_MAX_ZSTD_DECOMPRESSED_SIZE bounds decompressed bytes only, and bytes are the wrong unit here. An all-NULL column costs one bit per cell before Zstd, and RLE encodes a whole bitmap run in a single byte, so a compressed body detaches the declared grid from the wire entirely. 64 MiB of all-NULL bitmaps -- inside the Zstd cap -- describes 511 columns of 1,048,576 rows, about 1.07 billion array slots. Measured against the built bundle: 140 bytes allocated 83 MB (593,763x), 1,727 bytes exhausted a 1 GB heap, and 6,655 bytes declaring 62,917,817 decompressed bytes aborted the process with exit 134. The frames are well formed -- the decoder accepted them and returned every column all-null -- so any compromised or buggy server can emit one as the first RESULT_BATCH of an ordinary query(). Nothing capped it earlier: reserveMaterializedBatch gates on batch count rather than size, credit is accounted in compressed wire bytes, and the ws socket carries no maxPayload. The zero-copy queryViews() path allocates one pooled Int32Array per column instead, which amplifies less but still reaches roughly 2 GB at 511 columns. Cap the product. 32Mi cells is about 512 MB decoded at the measured 16 bytes per cell: far above any plausible result -- the widest supported table at 16k rows, or a full 1,048,576-row batch at 32 columns -- and far below what the two independent caps permitted. The check runs in prepare(), before a column is read, because reading one is what allocates; it covers decode() and decodeView() alike, and continuation batches against their established schema. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
discardHotSpare() wraps and rethrows anything but ENOENT from the spare's unlink or the directory fsync, and it shared a try block with closeSegmentHandles(). A read-only, full or descriptor-starved volume therefore skipped the second call and stranded one FileHandle per live segment. Nothing reopens them: close() memoizes closePromise and the finally below sets `closed` regardless, so a long-lived process that opens and closes store-and-forward senders against a degraded volume leaks descriptors until it exits. load()'s failure path already separates the two for this reason, closing the segment handles and the recovery handles in a finally before releasing the lock. Give close() the same shape. The hot-spare failure is still reported -- it is still the first `failure` and close() still rejects with it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
c754bfb added ./src/qwp/index.ts, browser.ts and node.ts to typedoc.json to make the QWP entry points reachable for docs, but the npm script still passed src/index.ts positionally, and TypeDoc treats positional arguments as entry points that override the config file. `pnpm docs` therefore emitted the root entry alone, with warnings that QwpTableWriter, QwpWriterColumn and the four QwpExtraOptions members were "referenced but not included in the documentation"; one output file mentioned connectQwpNodeSender, the copy of QWP.md placed in media/. Dropping the argument emits modules/qwp.html, qwp_browser.html and qwp_node.html, and 43 files reference QwpSender. Four entry points move the README to project level, where {@link Sender} and {@link SenderOptions} no longer resolve, so both now name their module. That removes the last two warnings the switch introduced. docs/ is regenerated at release time -- its history is v4.1.0, v4.2.0 -- so it is left alone here and the next release picks up the QWP pages. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
at() and atNow() threw before startNewRow(), leaving hasTable set and position past endOfLastRow. Every later table() then raised "Table name has already been set" -- including after a successful flush(), because compact() moves bytes without touching the row flags -- so one rejected row ended the sender for good unless the caller knew to call reset(), which discards whatever was staged. Two ways in. A row whose every value is nullish cannot be encoded, and this release turned that from a per-value type error into the documented outcome of the omission contract, so it is now reachable from data rather than from a programming mistake. And the timestamp unit is only validated inside writeTimestamp, which runs after at() has written the separator: an unknown unit left a trailing space in an open row, and retrying at() with a good unit appended a second separator and corrupted the line. Roll the row back on either. at()/atNow() now leave the buffer exactly as it was before table(), which is the contract the QWP sender's cancelRow() already offers, and rows already in the buffer are untouched. A caller catches the error and starts the next row from table(). Note this narrows what a caught error leaves behind: code that caught the empty-row error and then added a column to the same open row no longer sees it. That row could never be closed through table() again, so the pattern could not survive a loop. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
reset() zeroes pendingRowCount and pendingByteCount synchronously, while a flush already in flight subtracts its own snapshot in releaseStagedRows() after its await. Both ran against the same counters, so the rows were retired twice. Executed with the interleaving forced: two staged rows leave pendingRows at -2 and pendingBytes at -32, permanently, and three further rows under autoFlushRows: 3 then produce no frame at all -- every row- and byte-triggered auto-flush stays late by that offset for the rest of the sender's life, and metrics.pendingRows reads negative to anything polling it. Both entry points are public and documented and neither claims they exclude each other; Sender.reset() delegates straight through. The ILP sender is immune because flush() snapshots and calls resetAutoFlush() before its first await. Version the staging instead. reset() bumps a generation, flushNow() captures it alongside its snapshots, and releaseStagedRows() retires nothing when they disagree -- the tables those snapshots hold are already detached from `tables`, so there is nothing left to splice either. No rows are lost or duplicated either way; this is the counters only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Opening a connection runs under two deadlines: connectTimeoutMs covers the TCP/TLS transport, and authTimeoutMs takes over for the upgrade and authentication exchange as soon as transportConnected resolves. Both defaulted to 15 seconds independently, and QWP.md documented the default for both as an em dash, so a caller who narrowed connect_timeout got no part of what the key's own description promises -- "deadline for establishing one connection". Measured against a peer that accepts TCP and never answers the upgrade, which is what a stalled proxy or load balancer looks like: ws::...;connect_timeout=200 held the first atNow() for 15,018ms. Adding auth_timeout_ms=300 brought it to 305ms, but nothing pointed at that key, and its default was undocumented. authTimeoutMs now falls back to connectTimeoutMs before the default. This only ever tightens a bound the caller set explicitly -- it changes nothing when both are set or when neither is -- and passing auth_timeout_ms still buys the slower phase an independent budget, which is the right thing when an upgrade legitimately outlasts the transport. QWP.md now states both defaults and which phase each covers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… window decompressQwpZstdFrame drove fzstd's streaming Decompress, which keeps a window buffer the size of the declared window and memmoves the whole thing down after every block. Reframing single-segment made that window the entire content, so the cost was blocks x contentSize: 862 ms for a legitimate 64 MiB batch that decodes in 6 ms, and 1.5 s for the 4 KB all-NULL frame the grid-cap test builds -- which is how "bounds the grid a RESULT_BATCH declares" came to exceed the 5 s test timeout on every CI runner while taking 3.2 s locally. The shift is an amplification vector in its own right: a few kilobytes of RLE blocks buy seconds of memcpy, and nothing about that needs the frame to be malformed. fzstd's one-shot decompress() decodes straight into the output buffer whenever the window spans it, resolving matches against the output instead of a copy, so the reframe now targets that path. What that path does not give is the byte accounting: it never reports how far it got, it zero-pads a frame that stops short, and it truncates one that runs long. So the reframe appends an eight-byte RLE block after the frame's own blocks -- clearing the last-block flag on the block that carried it -- and declares the content size that marker needs. The marker lands exactly where the frame's output ends, so finding it at the declared content size is the guarantee `written === contentSize` used to give, and the bytes it occupies are the headroom that lets an over-long frame write past the declared size instead of being silently truncated into it. Both mismatch messages are unchanged, and a short frame still reports its real size: the marker is the last thing it writes, and everything past that is the untouched zero tail of the buffer. Handing fzstd a buffer of our own would have been the obvious way to keep exact accounting, and it is a trap: it compares that argument against a sentinel with `!=`, which coerces the whole Uint8Array to a string. 840 ms for a decode that takes 8 ms. Measured on 1,048,576-row all-NULL frames, decode falls from 104 ms to 3.8 ms at 128 columns, from 1,365 ms to 11 ms at 480, and from 1,544 ms to 16 ms at 511. The test that timed out now runs in 28 ms. Frames carrying a content checksum now reach fzstd without those four bytes, since the marker has to be the last block; nothing verified them before either. A test covers that shape, because none did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…inal classifyUpgradeRejection marked every non-101 status except 421 as retryable: false. connectLoop rethrows a non-retryable error before it reaches the attempt or duration budget, and pump()'s catch then latches the connection terminal, so a single blip ended the sender for the life of the process. Measured against a ws server that accepts the first upgrade and answers the reconnect with a status for 250ms before recovering, on the documented default connect string ws::addr=host:port with no tuning: 503, 502, 500, 504 and 429 each produced exactly one reconnect attempt and never recovered. 421 made five and honoured the budget. Raising the budget to maxAttempts: 200 and maxDurationMs: 60000 still produced one attempt, because the throw precedes the exhaustion check. Afterwards every flush rejects with the original upgrade error even once the server is healthy, rows staged at that moment are never delivered, close() throws rather than draining them, and failTerminal discards the unacked frames held in the in-memory replay buffer. connect() still resolves true and metrics.connected still reads true, so a health check sees a healthy sender. A rolling restart behind nginx or an ALB is exactly this shape, and the rest of the library already says so: the browser bootstrap uses statusCode >= 500, and the ILP HTTP transport's RETRIABLE_STATUS_CODES lists 500, 503, 504 and more as "server errors and gateway timeouts that may be transient". 5xx and 429 now join 421. 401 and 403 stay terminal, and a 4xx other than 429 is a client-side mistake that byte-identical replay cannot fix, so it stays non-retryable too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
readOwnerFile collapsed every failure into undefined, ownsOwnerDirectory read undefined as "not mine", and beat() read that as a takeover and called markCompromised() -- which also stops the heartbeat, so nothing could ever clear it. One failed read permanently ended a journal nobody had touched. The read is the only step of the heartbeat that needs a file descriptor: stat() and utimes() do not. Process-wide descriptor pressure, from anywhere in the host application, therefore hits precisely this call while the rest of the beat still succeeds; EIO and NFS ESTALE land the same way. Reproduced with a real kernel fault and no mocking -- ulimit -n 120, then exhausting descriptors with openSync across one beat -- and the surviving stat() proved the mtime was unchanged, which is the evidence of ownership the code then threw away. The latch also bought nothing. Injecting the same EIO on stat() instead leaves the beat transient: lost stays false until provenAtMs goes stale at +21.9s, the fault clears, the next beat refreshes, and release() succeeds. Injecting it on the record read latched at the first beat, while provenAtMs was 5.9s old, and never recovered. Both are equally safe against the double-write the design fears; only one is recoverable. Collateral from one injected EMFILE: append rejects QwpReplayStoreLockLostError with retryable false so nothing retries it, close() throws, the .lock.owner directory leaks because release() skips removal when compromised, and a same-process reopen is refused for a full staleness window. The message -- "taken over by another process while it was open" -- was false; the record still held our own pid and token. readOwnerFile now reports absent, present or unreadable. Only a record that is genuinely gone or genuinely someone else's marks the lock lost; a failed read skips the beat and lets the next one retry, exactly as the catch below it already does for stat(), with provenAtMs staleness supplying the fail-closed guarantee. release() keeps a lock it cannot vouch for on the retry list instead of reporting a release that never happened. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ms it persistAcknowledgedThrough fsyncs the watermark only under "append" durability, while writeManifest fsyncs its record and the directory whatever the mode. A trim runs straight after the ACK that emptied the segment, so the head could reach disk while the watermark justifying it was still in the page cache. A syscall trace of the unmodified program, with only the fs import redirected to a recording shim, shows the order: write .ack-watermark with no sync, then write sf-manifest.bin, sync it, sync the directory. Under "append" the same trace syncs the watermark, and the control run recovers cleanly from the same crash -- the single missing fsync is the whole difference. Recovery rejects the resulting pair with "sequence has a gap" and quarantines the slot behind a .failed sentinel; the fresh slot starts empty and the orphan drainer skips *.unreplayable-N, so every live frame is abandoned rather than a bounded suffix. QWP.md promises "periodic" can lose the most recent checkpoint window, and this loses the journal. Worth being precise about reachability: a real SIGKILL does not produce it. The un-fsynced write reaches the page cache and survives a process crash, an OOM kill and a container stop -- verified by forking a producer, killing it, and reopening. Only a kernel panic, a power cut or a hard VM loss drops that page, which is exactly the failure "periodic" exists to bound. writeManifest now flushes a pending watermark first. The flag driving it is tracked separately from acknowledgementDirty, which only schedules the periodic checkpoint, so the ordering holds under "memory" too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
assertReady() is the only reader of slotLock.lost, and it guards the six public
mutators. Background maintenance and every teardown step ran outside it -- and
close() is reached by exactly the terminal path a lost lock triggers, so losing
the slot was what set the deletions going rather than what stopped them.
failTerminal calls closeStore(), which calls store.close(), which is itself one
of the unfenced paths.
What that destroyed, executed across real processes with control arms:
- close() alone, with no fault injection anywhere, deleted the successor's
live .symbol-dict; its restart then failed with "corrupt QWP symbol
dictionary: invalid magic".
- A trim deleted three of the current owner's segment files, six frames of
payload, and sf-manifest.bin; the restart failed with "segment chain has a
gap" and the slot was quarantined.
- The manifest is written into one of two slots chosen by generation parity,
so an ex-owner's generation 7 record physically overwrote the successor's
generation 9, and recovery then read "segment lies beyond the manifest
active boundary".
- Dropping .ack-watermark resurrected an already acknowledged frame for
re-send.
Control runs where the evicted holder stayed suspended recovered every frame,
so the deletions are the cause rather than the symptom.
Every mutating maintenance and teardown step now checks ownership first and
releases in-memory state only. discardHotSpare still closes its descriptor,
which is ours either way, but no longer unlinks a name the successor may have
re-created. drainPendingMaintenance drops its queue instead of retrying against
somebody else's files, so close() still finishes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A journal that fails recovery was quarantined on the first attempt, moved aside behind a .failed sentinel and reported as data loss. Yet the failed load's own close() runs drainPendingMaintenance(), which drops a watermark stranded by a torn checkpoint -- so the condition that rejected the journal is usually gone by the time the caller is told the data is unreadable. Executed against the stale-watermark state a power loss leaves: the first load throws "sequence has a gap [previous=1, received=18]", and a second attempt on the same directory resolves with all six frames. Copying the quarantined directory out and loading it also resolves with six frames -- the bytes were intact and replayable the whole time, and only the decision to stop after one try lost them. connectQwpNodeIngress now retries the directory once before giving up on it. Only a second recovery failure quarantines; a transport fault or an aborted connect during the retry says nothing about the journal, so it propagates and leaves the directory alone. This is independent of durability mode: any recovery failure the store repairs on close was being abandoned, not only the one the missing ACK fsync produced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
encodeUdpDatagrams searched for each datagram's last row between start+1 and
table.rowCount, so the first probe of every datagram encoded half the rows still
left. sliceRows then made it worse: values holds non-null entries only, so it
walked every row before the slice to turn a row index into a value index. Two
independent quadratics over the same batch.
Measured through Sender.fromConfig("udp::addr=host:9007;auto_flush=off;"), one
documented option, then staging rows and calling flush():
rows before after
4000 67ms 29ms
16000 535ms 45ms
32000 2110ms 88ms
64000 10913ms 182ms
128000 - 353ms
encodeUdpDatagrams is a plain synchronous loop, so this is a hard event-loop
stall: a 1ms heartbeat recorded zero ticks inside a 2291ms encode span while
firing 45 outside it. No timers, sockets or health checks run in that window,
and throughput fell as the batch grew.
The search now gallops its upper bound outward from the last accepted run, and
sliceRows skips the null scan for a column that has none -- the common case, and
an exact shortcut rather than an approximation. The split is unchanged: the
datagram boundaries are identical, and the row-encode count for 4000 rows drops
from 395,826 to 31,508, now scaling 2.0x per doubling instead of 3.85x.
The default UDP configuration was never affected, because its byte trigger
keeps a flush near one datagram. sliceRows is also what the ingress batch-cap
bisector walks with, so that path gets the same relief.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
resolveQwpConfig() built the sender options as
`{ ...options.qwp?.sender, log: options.log ?? undefined }`, and
resolveQwpNodeClientConfig() spreads that object last over its own defaults. So
a caller who passed only `qwp.sender.log` had it overwritten by an explicit
undefined, and QwpSender fell back to `() => undefined`.
The result is total silence from the sender, including the warn that completed
rows are being discarded at close and the error reporting how many were lost --
the messages a logger is configured to catch. Only adding a second, top-level
`log` produced any output, and that one then won anyway.
Sibling fields of the same documented object -- awaitDurableAck, autoFlushRows
-- always took effect, and the neighbouring webSocket merge already resolves
per field with `??`, so this was a slip rather than a precedence rule. The
top-level logger still wins; it just falls back to the QWP one instead of to
undefined.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard that rejects a Zstd frame whose output runs past its declared content size worked by appending a run of one byte and looking for that run at the declared size. A run cannot say where it starts. A frame that overshot by k bytes pushed the marker to contentSize + k and left its own k bytes in front of it, so the check still matched whenever those bytes were the marker byte -- only min(k, 8) of them had to. Overshooting by one therefore needed a single byte, 1 in 256, and 0xa5 is a legal UTF-8 continuation byte (0xc2 0xa5 is "¥"), so a VARCHAR ending in one collided without anyone trying. An RLE final block made it worse: fzstd's fill clamps, so any overshoot at all landed, and a 1,613-byte frame declaring 8 bytes with 52MB of nominal output was accepted. It is not a harmless missed rejection. The output is truncated to the declared size, and that truncation also swallows the "unexpected trailing byte(s)" the same bytes raise when declared honestly -- so the client reports a complete, successful result for a frame it is supposed to reject. The marker is now eight distinct bytes written as a raw block, since an RLE block can only repeat one. No proper prefix of the pattern equals a proper suffix, so no shift can reproduce it and testing the declared offset is now the whole test. Eight bytes of slack past the marker keep a small overshoot inside the buffer so it can still be reported with the size it really decoded; anything further cannot place the marker and fzstd rejects it first. Locating the marker for that message no longer scans back over zeros: fzstd stages a block's literals in the unwritten tail of the output buffer, so the tail is not reliably zero. It searches for the pattern instead, bounded, because a hostile frame chooses how far off its output ends. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every fluent setter returned on a nullish value before addColumn had looked at anything, so the sender's availability, the row state and the column name were never checked for a row that omitted the column. All 26 setters accepted an illegal name, a non-string name, a call made before table(), and a call on a closed sender, as long as the value happened to be null. The consequence is that a call site is accepted or rejected by which rows happen to carry a value. A typo'd column name -- "user-id", say -- is silently fine on every row where the field is absent, then throws on the first row that has one, and failRow() discards that whole row. Rows already staged reach the server missing a column the caller believed they were writing, and the field that is usually null is exactly the field a caller reaches for the nullish rule with, so the typo survives testing. The ILP senders had this bug and fixed it in validateColumnCall(), with a regression test naming this failure mode. README.md documents the nullish rule as shared by the ILP and QWP senders, so they have to agree; only the compiled writers were already safe, because they validate every schema key at compile time whatever the values. The constants that describe a column rather than a row's value -- a decimal's scale, a geohash's precision -- are now checked before the value too, matching SenderBufferV3. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both keys were parsed from the ws/wss cluster string, validated, and then applied to the egress connection factory alone. On the ingress side target degenerated to "accept any role" and the health tracker ran zone-blind, so every endpoint ranked as same-zone and configuration order alone decided where writes landed -- including on a replica that answered the upgrade. Through Sender.fromConfig it was total, because that path uses only options.ingress: a bogus target still threw "Invalid target", while a valid one did nothing whatsoever. QWP.md lists both under "Reconnect and failover", and annotates the neighbouring failover key as egress-only but not these two, and the ingress section promises endpoints are "ranked by observed health ... and then by zone affinity". Both now reach the ingress factory and its health tracker, and QWP.md says plainly that they cover both directions. Role matching also had to become fail-open for an endpoint that declares no role. Egress always learns one from SERVER_INFO, but ingress reads it from an upgrade response header that an older server may not send and a proxy may strip, and the egress rule applied unchanged rejected such an endpoint outright -- an existing failover test using target=replica against a header-less mock server went from connecting to exhausting its reconnect budget. A server that does know its role still rejects a misdirected write itself, with the 421 this client classifies as ROLE_REJECTED, so nothing is lost by trusting it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…lock QwpIngressSession.connect() receives an AbortSignal and passed it to exactly two places: the eager initial connection, and the non-reconnecting branch. The eager connection is skipped whenever a replay store or background store-and-forward is configured -- precisely the configurations that take a slot lock -- and QwpReconnectingIngressConnection.connect() had no signal parameter at all. So close() aborted nothing on the one path where a connect owns a lock. The result: close() resolved, with no error and no warning, while the abandoned connect went on holding the slot for the rest of its connect budget. Against a peer that accepts TCP and never answers the upgrade -- a stalled proxy or load balancer -- that is the full connect_timeout, and 30s with a reconnect budget; a second sender on the same directory failed with QwpReplayStoreLockedError naming its own process. The abandoned session also kept doing real work after shutdown, re-sending a journaled frame and, in one shape, renaming a slot to .unreplayable-N. QWP.md says the journal "takes an exclusive lock when it is loaded and holds it until the sender or session closes", and the signal's own documentation says a connect still negotiating "can be torn down instead of outliving the sender by up to its connect/auth deadline". Both described the intent, not the behaviour; connectAbort itself is new in this branch, so this is incompletely-wired new work rather than legacy. The signal now reaches that connect. It is checked before the store is loaded, again once the lock is held, and an abort during the connect closes the connection -- which closes the store and releases the lock. A lock can still outlive close() by an in-flight load, which is bounded by the load rather than by the connect budget. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The row-count half of "documents the auto-flush defaults the sender actually applies" staged 999 rows against a live 100ms auto-flush interval. The work is about 2ms, but it is 999 awaits, and on a contended two-core CI runner the event loop can take longer than the interval to get through them -- so the interval trigger fired mid-loop and the assertion saw a partial row count: "expected [ 773 ] to deeply equal []". The race is inherent to the test rather than new, but this branch's added suites raised the parallel load enough to lose it. Reproduced deterministically by injecting a 150ms stall into the loop, which fails identically with "expected [ 302 ] to deeply equal []" and passes with the clock frozen. Only Date is faked, which is all the interval check reads, so the flush machinery's own timers keep working. The interval half already drives a faked clock explicitly and is unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The client now ships a browser build alongside the Node.js one, so naming it after a single runtime no longer describes who can use it. The sibling clients are named for the language their consumers write, and consumers here write JavaScript or TypeScript -- both served by the same JavaScript artifact and its type declarations, so neither is excluded by the broader name. This changes the human-facing name only: the package description, the typedoc title, the package documentation header, README prose, the examples manifest, and CLAUDE.md. The published package name, every import specifier, the tsconfig path mappings, and the repository URLs are untouched, so nothing an existing consumer resolves against moves. Two comments that had settled on "TypeScript client" are folded in here too, so the release that raises the question does not leave a third name behind. References that genuinely mean the Node.js runtime keep it: the store-and-forward locking section reasons about runtimes rather than products, and the pooled Node client and the orphan drainer are Node-only APIs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CmYejbzDF32mYojAwbTqM4
The Enterprise lane is now build-and-test-e2e-javascript-client and takes javascriptClientCommit / javascriptClientPrNumber, following this repository's rename away from Node.js. The dispatch resolves the pipeline by name and sends those parameters, so both sides have to agree or the lookup fails. The job stays gated behind ENTERPRISE_E2E_ENABLED, so nothing runs until the pipeline is renamed in Azure DevOps to match. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CmYejbzDF32mYojAwbTqM4
Seven observability callbacks were invoked inside a synchronous-only try/catch. That guards a thrown error, but an async callback returns a promise: when it rejects, the rejection escapes the try/catch and Node >= 15 turns an unhandled rejection into process termination by default. A single async onEvent, onError, onSenderError, onRecoveryQuarantine or onRecoveryDataLoss -- all reachable from the plain public API -- took the whole process down. The orphan-drain path was the decisive one. QWP.md says callbacks are "placed on bounded asynchronous inboxes and never invoked inside ACK, reconnect, or orphan-recovery protocol stacks ... Callback failures are contained", and the drainer does feed reconnect events through QwpNotificationDispatcher, whose dispatchOne already contained a returned promise. But the wrapper installed as the dispatcher's handler swallowed the sync throw and returned undefined, so that check had nothing to attach to and the user's promise orphaned through the very inbox meant to hold it. The containment already existed twice -- QwpNotificationDispatcher and a private safelyInvoke in ingress-session -- so this consolidates both onto one helper, src/_qwp/_internal/safe-callback.ts. safelyInvoke() contains a synchronous throw and a rejected promise alike, routing either to a guarded onFailure that can never re-escape, and uses the portable then(undefined, ...) rather than catch() so a bare thenable is handled too. All seven sites now go through it, each keeping its own failure behaviour: swallow, log once, or fall back to the default handler. Verified as a real process rather than under Vitest, which masks the crash by handling unhandled rejections itself: on Node 20.11.0 the old sync-only pattern exits 1 and safelyInvoke exits 0 with onFailure seeing the rejection. Added a safe-callback suite and an async-rejection case to the dispatcher suite, both asserting no unhandledRejection fires. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Nothing asserted that a wss:// producer verifies the server certificate or that
its Authorization header carries the operator's credentials unchanged. Both
fail silently -- a disabled check still connects, transposed Basic credentials
still form a header -- so mutating either the TLS agent or the authorization
encoding left the whole suite green. The nearest assertion only checked that
ingress.agent exists, never its rejectUnauthorized, ca, pfx or passphrase.
This is a departure from the client's own standard rather than a general gap:
sender.transport.test.ts exercises ILP TLS for real against test/certs, so the
fixtures a QWP test needs already exist.
The new suite covers both wss construction paths. The documented `wss::`
connect string, resolved by parseQwpNodeClientConfig(), is asserted for
rejectUnauthorized, a custom ca, and a pfx trust store with its passphrase
across tls_verify on/unsafe_off and tls_roots, plus the unconfigured case that
must build no agent at all so node's verifying default applies. The programmatic
`new Sender({ protocol: "wss", ... })` object, handled by sender.ts, is asserted
by capturing the ingress options handed to createQwpNodeSender: the agent's ca
and rejectUnauthorized, the Basic header's username:password order, and the
Bearer prefix.
Each admitted mutation was re-applied and now turns the suite red: disabling
verification in createTlsAgent fails three connect-string tests, the same in the
sender.ts agent fails two, and swapping the Basic order or dropping the Bearer
prefix fails two. The unsafe_off and tls_verify=false cases stay green under the
TLS mutations, so the tests assert the intent rather than a constant.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dateColumn and fixedDecimalColumn -- the shared body of decimal64Column, decimal128Column and decimal256Column -- returned on a nullish value through a raw === null || === undefined check, bypassing omitsNullish. So on a nullish row they skipped the sender availability, row state and column name checks every other setter runs, and the decimals also skipped the scale check. A misspelled or over-long name, or a bad scale constant, then surfaced only on the rows that happened to carry a value and stayed silent on the rest -- which is how a typo reaches production. An inventory of all 26 setters confirms these four were the only ones left: the rest already route a nullish value through omitsNullish, and doubleColumn is safe because it delegates to floatColumn. This is the same class commit 266438f fixed for the setters it covered, and README.md documents the nullish rule as shared across QuestDB clients, so they must agree. dateColumn now goes through omitsNullish like its sibling timestampColumn. fixedDecimalColumn hoists the scale check above the gate -- the scale describes the column, not this row's value -- then routes the name through omitsNullish, exactly as decimalColumn already does. The regression test written for this bug covered seven setters and omitted all four; it now asserts dateColumn's name, each fixed-width decimal's scale constant, and decimal64Column's over-long name all raise on a nullish value, and that a valid nullish dateColumn/decimal64Column call is still omitted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
qwpColumnNameKey builds its result one code unit at a time, and it ran twice per cell: once when the cell is staged, then again at flush inside getOrCreateColumn -- even though buildTable iterates row.columns, a Map already keyed by exactly that value. Two changes remove the redundant work. buildTable now iterates the map entries and passes the key it already holds into getOrCreateColumn, which takes it as an optional third argument defaulting to qwpColumnNameKey(name), so every other caller is unchanged. That key is provably the one getOrCreateColumn would have computed: each entry's stored canonical name is one whose key is the map key it lives under. Separately, qwpColumnNameKey gains an all-lower-case-ASCII fast path -- a name of only lower-case-stable code units is returned unchanged, and the first upper-case ASCII or non-ASCII code unit resumes the per-code-unit mapping from the stable prefix -- so the common name skips the rebuild entirely. A local before/after run of the shipped benchmarks (benchmarks/sender.bench.ts, build and encode) measured roughly +33% on trades, +12% on wide and +8% on sparse; trades is the primary ingest path. The surface is new in this branch, so this is measurable headroom rather than a regression from an earlier release. A new identifiers suite asserts the fast path is byte-identical to the per-code-unit reference across upper-case, non-ASCII, surrogate-pair and U+0130 inputs, so two spellings of one column still collide on the same key. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…g it
README says the nullish rule applies to the compiled QWP writers and that a QWP
row whose every value is nullish is sent with no columns; QWP.md says regular
fields may all be null and a designated timestamp is required only "when
present". But for a schema without a designated timestamp,
encodeCompiledWriterRow threw QwpWriterRowError: row must contain at least one
non-null value -- a string that appears nowhere in README.md, QWP.md, any test
or any example. The exact fluent analogue, table("t").symbol("side", null)
.atNow(), is accepted and encodes a real frame, so the two APIs disagreed on
documented behaviour.
Dropping the columns.size === 0 guard makes the writer send the all-nullish row
with no columns, exactly as the fluent path does. The guard only ever bit a
timestamp-less schema: a designated timestamp is required earlier in the same
function -- it throws when nullish and stages a column when present -- so a
schema that declares one always reaches this point with at least one column. A
column-less row encodes cleanly; the encoder's non-null-row check is a
per-column count consistency check that a row with no columns has nothing to
fail.
Two tests cover it: a timestamp-less writer now stages and flushes an all-null
row as a zero-column, real-encoding frame, and a schema that declares a
designated timestamp still rejects a nullish one -- the requirement the dropped
guard sat next to.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A ws/wss connect string that carries max_datagram_size or multicast_ttl was rejected with the hint "(applies to legacy http/tcp/udp transports only)". But those two keys are UDP-only: http and tcp reject them as well, with "max_datagram_size and multicast_ttl are only supported for QWP UDP transport". So the hint sent the user to two more protocols that also refuse the key. The same string is correct for the four sibling keys it is shared with, which really do span all three transports, and options.ts already had the right wording; only these two entries were wrong. The test pinned the wrong string. The hint now reads "(applies to the legacy udp transport only)" for both keys, and the test asserts it for max_datagram_size and multicast_ttl alike. Neither key was documented anywhere a reader would look, even though the auto_flush_bytes entry cites max_datagram_size as its own default. A "UDP specific options" block in the SenderOptions reference -- the configuration reference README points to -- now documents both, with defaults (1400 and 0), the 0-255 TTL range, and that only udp accepts them; and the QWP.md UDP prose now spells the key names instead of only describing the values. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
releaseStagedRows() returns 0 on a staging-generation mismatch, which is correct for retiring pending rows -- reset() has already zeroed those counters, so subtracting again would drive pendingRows negative (the bug 0a1fccf fixed). But that one return value also fed totalRowsPublished += sentRows, so a reset() landing while a flush awaited its publication boundary lost the count for rows whose frames had already entered the ingress session -- exactly what the field documents. The counter then skewed permanently low: a five-row flush interrupted this way put all five rows on the wire and reported totalRowsPublished 0. The published count and the retired count are different questions. The flush now takes the published count from its own snapshots -- the rows it sent, regardless of a concurrent generation bump -- and releaseStagedRows() keeps doing only the pending retirement. The same retired-count value was wrong for three siblings that shared it, all now reading the published count: the flush debug log, the deferred-transaction row tally (the open transaction still holds those rows), and the > 0 guard that counts a committed transaction. The no-reset path is unchanged, since there the two counts are equal. A test holds a flush at its publication boundary, drops a reset() between "frame entered the session" and "rows retired", and asserts all five rows reached the wire with totalRowsPublished at five and pendingRows at zero rather than negative. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
scheduleMaintenance() rejected every parked appender with the maintenance failure and then, on the very next line, scheduled the retry that makes the rejection unnecessary. A background segment trim that briefly fails -- a read-only or full filesystem, a restarted maintenance worker -- therefore rejected a parked store-and-forward append with a retryable "could not trim QWP store-and-forward segment" error a few milliseconds into its append deadline, even though the retry self-heals about a second later and the identical append then succeeds. totalAppendTimeouts stays zero, so it is not the deadline error a caller watches for, and it contradicts the sf_dir wait contract QWP.md states: the journal ceiling is the one error a producer sees. The retried batch already releases parked appenders through signalCapacity() on success, and each appender keeps its own append deadline, so a permanent failure still ends in the typed append timeout rather than hanging. Dropping the reject leaves them parked for that retry. A released appender re-runs appendOnce() through enqueue(), which is serialized behind the maintenance batch, so it runs only after the batch has cleared the failure -- it never observes the stale one at assertReady(). The checkpoint sibling still rejects its waiters, correctly: that class has no retry outside durability "periodic", which is not the connect-string default. A test parks an append at capacity, fails the trim once, and asserts the append resolves when the retry frees space rather than being rejected, with totalAppendTimeouts still zero. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
markerCounts scanned the whole .sfa file for bytes in the A-Z range, so it also tallied framing bytes: the segment header ends in a microsecond wall-clock timestamp and each frame header carries a CRC32C. Whenever one of those bytes happened to equal 'A' (0x41) or 'B' (0x42) on a given run -- about 1.5% of the time per segment -- the durability assertion saw 321 instead of 320 and failed. The append path already rejects the reclaimed holder before it writes anything (assertReady throws QwpReplayStoreLockLostError ahead of the segment write), which the +1 rather than +64 discrepancy confirms, so no payload byte ever leaked; the flake was purely in the test helper. Walk the frame framing instead and count only payload bytes, which are pure marker fill by construction, making the check exact and deterministic. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add a complete QWP client surface that works in both browsers and Node.js while leaving the existing ILP transports Node-only.
QWP support ships as a preview:
QWP.mddocuments the compatibility baseline for the first QWP release, and imports from internal source paths are never supported.Entry points
@questdb/nodejs-clientSender, including QWP ingress selected withws::,wss::, orudp::@questdb/nodejs-client/qwp/browser@questdb/nodejs-client/qwp/node@questdb/nodejs-client/qwpThe package root keeps the existing Node.js transports and dependencies. The browser entry point has no Node.js imports, so supporting browsers does not require breaking the existing client.
Ingress
Senderintegration with fluent rows, batching, byte/interval auto-flush, commits, transactions, and ACK watermarkssender.writer(table, schema).row({...})) for repeated rows on one schema, with the full QWP column-type setudp::, Node-only) behind the same fluent row APIEgress
Observability
onProgress,onError, and the Java-parityonSenderErrorrejection stream for event-driven telemetryAPI and platform integration
qdb_sessionbenchmarks/) covering encoder floors, the high-level sender, egress views, store-and-forward persistence policies, and a live end-to-end laneQwpBrowserSessionAuthTest,QwpIngressUpgradeProcessorOnHeadersReadyTest, andQwpEgressMaxBatchRowsTest, plus the Enterprise REST/OIDC login suitesCompatibility
http/https/tcp/tcpssenders are unchanged, with the two exceptions below.auth: {keyId, token}supplies only the private scalar, and the JWK was completed with a hardcoded public point unrelated to it. Node.js accepted that inconsistent pair without validating it up to v24 and rejects it from v26 withERR_CRYPTO_INVALID_JWK, so TCP auth failed outright on that runtime. The point is now derived from the private key. Signing only ever used the private scalar, so signatures, credentials, and auth outcomes are unchanged on every Node.js version; callers passing a completejwkobject were never affected.nullandundefinednow omit the column, which QuestDB records as NULL; most column methods previously threw a type error. On protocol v2 this also changes the wire bytes forarrayColumn(name, null), which used to emit an explicit NULL-array marker — QuestDB rejects that encoding withARRAY_INVALID_TYPE(verified against 9.4.3), so omitting it is itself a fix. A row whose every value is nullish now fails when the row is closed rather than at the column call. Code that relied on the throw as a data-quality guard should validate before calling the sender.flock/LockFileEx; the Node.js client uses a pure-JavaScript directory lock and cannot participate in those kernel locks, so neither sees the other. The persistence format stays cross-client for sequential handoff — a directory written by one runtime can be opened by the other once the first has closed it — and two Node.js processes still exclude each other. Depending on a native addon for kernel locks was the alternative, and it left store-and-forward broken on any platform or Node.js major without a prebuilt binary.Dependencies
ws(Node WebSocket transport). There is no native dependency; store-and-forward locking is pure JavaScriptfzstdis bundled into the build output for egress decompression;THIRD_PARTY_NOTICES.mdrecords its licenseResolved issues
Null or undefined column and symbol values are omitted across the existing ILP senders and the new QWP senders.
Fixes Client should skip columns if value is null #28
The new QWP sender introduces the sender.write().row() API.
Fixes State-machine builder #60
Validation
pnpm vitest run benchmarks): 3 files / 14 tests passedpnpm typecheckpnpm typecheck:qwp-browserpnpm typecheck:benchpnpm eslintpnpm lint:benchpnpm buildDependencies and provenance