T4: Wire central forward sink to incremental readRowsSince watermark#156
Merged
philcunliffe merged 1 commit intoJun 26, 2026
Merged
Conversation
Switch `forwardPartition` from a full-partition `storage.readRows(tablePath)`
scan to the cursor-aware `storage.readRowsSince(tablePath, { since })`, driven
by a per-(sink instance, partition) watermark loaded from the sink's stateDir
watermark store (T3). Each acked chunk advances the watermark to that chunk's
last `after` token (ship first, advance second), so a crash re-sends at most one
chunk and the server idempotency ledger now backstops only a bounded in-flight
suffix instead of the whole partition. A tick with no new rows yields zero rows,
sends zero chunks, and writes zero bytes.
Chunking (MAX_CHUNK_ROWS/MAX_CHUNK_BYTES), the Retry-After backpressure loop,
and `batchIdForChunk` derivation are unchanged. A missing/unreadable watermark
or underivable key falls back to a full scan (at-least-once + server dedup),
never a silent skip.
Implements task T4 of incremental-sink-reads (LLP 0040 §3-4, 0042).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Task-Id: T4
philcunliffe
merged commit a3f0808
into
integration/incremental-sink-reads
6 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements task T4 of incremental-sink-reads.
Wires the central
forwardsink (forwardPartition) from a full-partitionreadRowsscan to the cursor-awarereadRowsSince({ since }), driven by the per-(sink instance, partition) watermark store (T3) under the sinkstateDir. The watermark advances per acked chunk to that chunk's lastafter(ship first, advance second); chunking, the Retry-After backpressure loop, andbatchIdForChunkare unchanged; the server idempotency ledger now backstops only a bounded in-flight suffix. No-new-rows ticks send 0 chunks / 0 bytes; a missing/unreadable watermark falls back to a full scan (at-least-once + server dedup).Gates:
npm test(1423 pass),npm run typecheck, andnpm run lintall green. Thecentral_forward_outboxhermetic smoke is pre-existing red on the base integration branch (verified by stashing T4: identicalbytesWritten=0failure) due to a fixturediscoverPartitionsvssource=unknownmismatch unrelated to this change.Task-Id: T4