Open
Conversation
Reorder class members (fields, methods) alphabetically by kind and visibility across all java-questdb-client source and test files. This follows the project convention of alphabetical member ordering. Remove decorative section-heading comments (// ===, // ---, // ====================) that no longer serve a purpose after alphabetical reordering. Incorporate the orphaned "Fast-path API" comment block into the QwpWebSocketSender class Javadoc. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The bucket boundary constants for two's complement signed ranges were inverted — the min and max magnitudes were swapped. For an N-bit two's complement integer the valid range is [-2^(N-1), 2^(N-1) - 1], so: 7-bit: [-64, 63] not [-63, 64] 9-bit: [-256, 255] not [-255, 256] 12-bit: [-2048, 2047] not [-2047, 2048] With the old boundaries, a value like 64 would be placed in the 7-bit bucket, but 64 in 7-bit two's complement decodes as -64, silently corrupting timestamp data at bucket boundaries. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The opcode parameter in beginFrame(int) was accepted but never stored or used — the opcode only matters when writing the frame header in endFrame(int), where all callers already pass the correct value. Remove the misleading parameter to make the API honest. Also remove beginBinaryFrame() and beginTextFrame() which were just wrappers passing an unused opcode. The single caller in WebSocketClient is updated to call beginFrame() directly. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The single-host fallback in configuration string parsing used
DEFAULT_HTTP_PORT for all non-TCP protocols. This works by
coincidence since DEFAULT_HTTP_PORT and DEFAULT_WEBSOCKET_PORT
are both 9000, but is semantically incorrect. Use the proper
DEFAULT_WEBSOCKET_PORT constant when the protocol is WebSocket.
Also clean up Javadoc: remove a dangling isRetryable() reference
from Sender.java, fix grammar ("allows to use" -> "allows
using"), and tidy LineSenderException Javadoc.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
inc() called putAt0() directly without adding the key to the `list` field. This caused keys() to be incomplete and valueQuick() to return wrong results for keys inserted via inc(). Add the missing list.add() call, consistent with putAt() and putIfAbsent(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a high surrogate was detected in the UTF-8 encoding paths, the next char was consumed and used as a low surrogate without validating it was actually in the [0xDC00, 0xDFFF] range. This produced garbage 4-byte sequences and silently swallowed the following character. Add Character.isLowSurrogate(c2) checks in all 5 putUtf8/hasher encoding sites and both utf8Length methods. Invalid surrogates now emit '?' and re-process the consumed char on the next iteration, consistent with Utf8s.encodeUtf16Surrogate(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The boolean[] wrapper and anonymous WebSocketFrameHandler were allocated on every loop iteration inside waitForAck(), generating GC pressure on the data ingestion hot path. Hoist both into reusable instance fields: ackResponse (WebSocket response buffer), sawBinaryAck (plain boolean replacing the boolean[] wrapper), and ackHandler (a static nested class AckFrameHandler replacing the per-iteration anonymous class). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The close() method in async mode was only waiting for pending batches to be written to the wire (via sendQueue.close()), but did not wait for the server to acknowledge receipt. This caused data loss when close() was called without an explicit flush(), since the connection was torn down before the server finished processing. Add sendQueue.flush() and inFlightWindow.awaitEmpty() before sendQueue.close() to match the behavior of flush(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add TYPE_GEOHASH to elementSize(), allocateStorage(), and addNull() so geohash values are stored as longs (8 bytes) with -1L as the null sentinel. Also add an explicit case in QwpConstants.getFixedTypeSize() to document that GEOHASH is intentionally variable-width on the wire. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the XorShift128 Rnd PRNG with a new ChaCha20-based SecureRnd for WebSocket frame mask key generation. The previous implementation seeded Rnd with System.nanoTime() and System.currentTimeMillis(), which is predictable and does not meet RFC 6455 Section 5.3's requirement for strong entropy. SecureRnd implements ChaCha20 in counter mode (RFC 7539), seeded once from java.security.SecureRandom at construction time. After initialization there are zero heap allocations — all state lives in two pre-allocated int[16] arrays. Each ChaCha20 block yields 16 mask keys, so the amortized cost is minimal. Includes a known-answer test using the RFC 7539 Section 2.3.2 test vector to verify correctness of the ChaCha20 implementation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove the redundant bitsInBuffer % 8 outer guard. Since ensureBits() always loads whole bytes, the invariant (totalBitsRead + bitsInBuffer) % 8 == 0 always holds, making bitsInBuffer % 8 and totalBitsRead % 8 equivalent checks. The simplified version uses only totalBitsRead % 8 which more clearly expresses the intent. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
utf8Length() correctly counts lone surrogates as 1 byte ('?'
replacement), but putUtf8() let them fall through to the BMP
3-byte encoding path. This 2-byte-per-surrogate discrepancy
corrupts varint-prefixed string lengths written by putString().
Add a Character.isSurrogate() check before the 3-byte BMP branch
in all three putUtf8() implementations: NativeBufferWriter,
WebSocketSendBuffer, and OffHeapAppendMemory. Add tests verifying
lone high/low surrogates write 1 byte and that putUtf8() and
utf8Length() agree for all surrogate edge cases.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
addSymbol() and addSymbolWithGlobalId() incremented size without writing data or incrementing valueCount when value was null and the column was non-nullable. This caused a permanent misalignment between logical row count and physical data. Delegate null values to addNull(), which already handles both nullable (mark in bitmap) and non-nullable (write sentinel) cases correctly. Add a test that verifies size and valueCount stay in sync for non-nullable symbol columns with null values. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
QwpTableBuffer.addNull() had no handling for TYPE_DOUBLE_ARRAY and TYPE_LONG_ARRAY in the non-nullable branch. This caused valueCount and size to advance without writing array metadata (dims, shapes), corrupting array index tracking for subsequent rows. The fix writes an empty 1D array (dims=1, shape=0, no data elements) as the sentinel value, keeping dims/shapes/data offsets consistent. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
sendCloseFrame() and sendPing() used the main sendBuffer to build and send control frames. If the caller had an in-progress data frame in sendBuffer (obtained via getSendBuffer()), these methods would destroy it by calling sendBuffer.reset(). Switch both methods to use controlFrameBuffer, which already exists for exactly this purpose — sendCloseFrameEcho() and sendPongFrame() were already using it correctly. Add unit tests that verify the sendBuffer is preserved across sendCloseFrame() and sendPing() calls. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Numbers.ceilPow2(int) returns 2^30 for inputs between 2^30+1 and Integer.MAX_VALUE due to internal overflow handling. This caused grow() to allocate a buffer smaller than the required capacity. Fix by taking the max of ceilPow2's result and the raw required capacity before clamping to maxBufferSize. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When readFrom() parsed an error response where status != OK and the buffer was large enough to enter the outer branch (length > offset + 2), but msgLen was 0, the inner condition (msgLen > 0) failed without clearing errorMessage. On reused WebSocketResponse objects this left the error message from a previous parse visible to callers. Add an else branch to the inner if that sets errorMessage = null when msgLen is 0 or the message bytes are truncated. Add a regression test that parses an error-with-message followed by an error-with-empty-message on the same object and asserts errorMessage is null. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
QwpWebSocketSender.reset() only reset the current table buffer, leaving other tables' data intact and pendingRowCount nonzero. The Sender.reset() contract requires all pending state to be discarded. Now iterate every table buffer in the map, zero pendingRowCount and firstPendingRowTimeNanos, and clear the current-table and cached-column references. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
ensureBits() loaded bytes into the 64-bit bitBuffer without checking whether all 8 bits of the incoming byte would actually fit. When bitsInBuffer exceeded 56, the left-shift lost high bits that overflowed position 63, and bitsInBuffer could grow past 64 — silently corrupting the buffer. Add a bitsInBuffer <= 56 guard to the while loop so we never attempt to load a byte when fewer than 8 free bit positions remain. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
skip() advanced the position without calling ensureCapacity(), so a skip that exceeded the current buffer capacity would let subsequent writes corrupt native memory past the allocation. Add the missing ensureCapacity(bytes) call, matching the existing pattern in WebSocketSendBuffer.skip() and OffHeapAppendMemory.skip(). Add a regression test that skips past the initial capacity and verifies the buffer grows. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
flushPendingRows() and flushSync() reset table buffers but did not clear cachedTimestampColumn and cachedTimestampNanosColumn. If the table buffer's columns were ever recreated rather than just data- reset, the stale references would become dangling. Null both fields at the start of each flush method, consistent with what reset() and table() already do. Add QwpWebSocketSenderFlushCacheTest to verify the invariant for both micros and nanos timestamp paths. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
putBlockOfBytes() accepted a long len parameter but cast it to int before calling ensureCapacity(). When len > Integer.MAX_VALUE, the cast wraps to a negative number, so ensureCapacity() skips the buffer grow, but copyMemory() still uses the original long len, causing a buffer overflow. Validate len fits in int range before casting in both NativeBufferWriter and WebSocketSendBuffer. Use the narrowed int value consistently for ensureCapacity(), copyMemory(), and position update. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merge QwpWebSocketSenderResetTest and QwpWebSocketSenderFlushCacheTest into a single QwpWebSocketSenderStateTest class. Both tested QwpWebSocketSender internal state management with the same reflection pattern and superclass. The merged class unifies the setField/ setConnected helpers into one setField method and keeps all three test methods in alphabetical order. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
WebSocketSendQueue accepted a queueCapacity parameter in its constructor, validated it, and logged it, but never used it. The actual queue is a single volatile slot (pendingBuffer) by design — matching the double-buffering scheme where at most one sealed buffer is pending while the other is being filled. Remove the parameter from the entire chain: WebSocketSendQueue constructor, QwpWebSocketSender field and factory methods, Sender.LineSenderBuilder API, and all tests and benchmark clients that referenced it. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename elementSize() to elementSizeInBuffer() in QwpTableBuffer to make explicit that it returns the in-memory buffer stride, not the wire-format encoding size. Update javadoc on both elementSizeInBuffer() and QwpConstants.getFixedTypeSize() to document the distinction: getFixedTypeSize() returns wire sizes (0 for bit-packed BOOLEAN, -1 for variable-width GEOHASH), while elementSizeInBuffer() returns the off-heap buffer stride (1 for BOOLEAN, 8 for GEOHASH). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
encodeColumn() and encodeColumnWithGlobalSymbols() had nearly identical switch statements across 15+ type cases, differing only in the SYMBOL case. Merge them into a single encodeColumn() with a boolean useGlobalSymbols parameter. Similarly merge the duplicate encodeTable() and encodeTableWithGlobalSymbols() into one method. This removes ~100 lines of duplicated code. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove WebSocketChannel, ResponseReader, and WebSocketChannelTest. These classes are dead code — the actual sender implementation (QwpWebSocketSender) uses WebSocketClient and WebSocketSendQueue instead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…va-questdb-client into jh_experiment_new_ilp
…xperiment_new_ilp
…va-questdb-client into jh_experiment_new_ilp
- Stack-allocate iovec/WSABUF in sendToScatter JNI to avoid per-send calloc on the UDP hot path (falls back to heap for >16 segments) - Add in-progress row guard to QwpWebSocketSender.flush() so partial rows are rejected, matching QwpUdpSender behavior - Merge canUseGorilla() and calculateEncodedSize() into a single-pass calculateEncodedSizeIfSupported() to eliminate a redundant scan over timestamp data - Fix native memory leak in SegmentedNativeBufferWriter and WebSocketSendBuffer constructors when a later allocation fails - Close waiter registration race in InFlightWindow by setting the waiting thread reference before re-checking the condition - Remove unnecessary volatile on maxSentSymbolId (user-thread only) - Add 49 exhaustive QwpGorillaEncoder tests covering round-trip encode/decode, all 5 bucket boundaries, degenerate cases, and the size overflow guard - Add 57 WebSocketFrameWriter tests covering header encoding for all 3 payload length formats, mask key serialization, XOR masking with RFC 6455 vectors, and round-trip with the parser - Remove banner comments from QwpWebSocketEncoderTest - Fix stale ILP4/ILP v4 references in Javadoc - Remove TODO.md, fix redundant Math.max(16,4) in NativeSegmentList Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Contributor
Author
[PR Coverage check]😍 pass : 4224 / 5365 (78.73%) file detail
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TODO in run_tests_pipeline.yaml! Change before merging!
Change to
Summary
This PR adds a new WebSocket-based ingestion path to the Java client using QWP (QuestDB Wire Protocol), a binary protocol that replaces text-based ILP for higher-throughput data ingestion. The existing HTTP and TCP ILP senders remain unchanged.
Users select the new transport via
Sender.builder(Transport.WEBSOCKET). The builder accepts WebSocket-specific options such asasyncMode,autoFlushBytes,autoFlushIntervalMillis, andinFlightWindowSize.Architecture
The implementation follows a layered design:
Protocol layer (
cutlass/qwp/protocol/)QwpTableBufferstores rows in columnar format using off-heap memory (zero-GC on the data path).QwpSchemaHashcomputes XXHash64 over column names and types, enabling server-side schema caching. The client sends a full schema on the first batch and a hash reference on subsequent batches if the schema has not changed.QwpGorillaEncoderapplies delta-of-delta compression to timestamp columns.QwpBitWriterhandles bit-level packing for booleans and null bitmaps.QwpConstantsdefines the wire format: "QWP1" magic bytes, type codes, feature flags, status codes.Client layer (
cutlass/qwp/client/)QwpWebSocketSenderimplements theSenderinterface. It uses a double-buffering scheme: the user thread writes rows into an activeMicrobatchBuffer, which is sealed and handed to an I/O thread when an auto-flush trigger fires (row count, byte size, or time interval).QwpWebSocketEncoderserializesQwpTableBuffercontents into binary QWP frames, including delta symbol dictionaries (only new symbols since the last acknowledged batch).InFlightWindowimplements a lock-free sliding window protocol that tracks batches awaiting server ACKs, providing backpressure from the server to the user thread.WebSocketSendQueueruns the dedicated I/O thread, managing frame transmission and ACK/NACK response parsing.GlobalSymbolDictionaryassigns sequential integer IDs to symbol strings and supports delta encoding across batches.WebSocket transport (
cutlass/http/client/,cutlass/qwp/websocket/)WebSocketClientis a zero-GC WebSocket implementation with platform-specific subclasses for Linux (epoll), macOS (kqueue), and Windows (select).WebSocketFrameParserandWebSocketFrameWriterhandle RFC 6455 frame serialization, including fragmentation, close-frame echo, and ping/pong.WebSocketSendBufferbuilds masked WebSocket frames directly in native memory.Bug fixes and robustness improvements
The PR fixes a number of issues found during development and testing:
WebSocketClientconstructor and on allocation failure.sendQueueleak on close when flush fails.WebSocketClient,WebSocketSendBuffer,QwpTableBuffer), array dimension products, andputBlockOfBytes().receiveFrame()throwing instead of returning false, which masked I/O errors as timeouts.cancelRow()truncation.SecureRnd(ChaCha20-based CSPRNG) for WebSocket masking keys instead ofjava.util.Random.Code cleanup
The PR removes ~11,000 lines of dead code:
ConcurrentHashMap(3,791 lines),ConcurrentIntHashMap(3,612 lines),GenericLexer,Base64Helper,LongObjHashMap,FilesFacade, and others.Numbers,Chars,Utf8s,Rnd, andColumnType.ParanoiaState,GeoHashes,BorrowedArray,HttpCookie.CI changes
ClientIntegrationTestsCI stage that starts a QuestDB server and runs the client's integration tests against it (both default and authenticated configurations).sedportability for macOS CI runners.Test plan
QwpSenderTest(8,346 lines) exercises the fullSenderAPI surface for all column types, null handling, cancelRow, schema changes, and error pathsQwpWebSocketSenderTesttests WebSocket-specific sender behavior including async modeQwpWebSocketEncoderTestvalidates binary encoding for all column types and encoding modesLineSenderBuilderWebSocketTestcovers builder validation and configuration for the WebSocket transportassertMemoryLeakwrappers added to client tests to detect native memory leaks