Skip to content

T6: exactly-once acceptance tests + smoke for incremental sink reads#158

Merged
philcunliffe merged 1 commit into
integration/incremental-sink-readsfrom
task/incremental-sink-reads/T6
Jun 26, 2026
Merged

T6: exactly-once acceptance tests + smoke for incremental sink reads#158
philcunliffe merged 1 commit into
integration/incremental-sink-readsfrom
task/incremental-sink-reads/T6

Conversation

@philcunliffe

Copy link
Copy Markdown
Contributor

Implements task T6 of the incremental-sink-reads change set (LLP 0042).

The exactly-once proof. T4/T5 ship comprehensive unit suites, but those stub
storage and so cannot prove the design's load-bearing claim: that a row-resident
_hyp_ingest_seq watermark survives the two cache rewrites that motivate the
whole design (LLP 0039) — a retention front-prune and a compaction generation
swap.

What this adds

  • test/core/sink-incremental-acceptance.test.js — a deterministic acceptance
    suite driving the real kernel cache (createQueryStorageService), the
    real retention enforcer, real maintainCache compaction, both
    sinks (central forward + core local-fs blob via their own
    activate/create), and the real sink driver's outbox respool. Eight tests:
    • forward + blob: ~0 bytes on a no-new-rows tick, ~N on an N-new tick;
    • forward + blob: exactly-once across a retention front-prune;
    • forward + blob: exactly-once across a compaction generation swap
      (asserts the generation directory actually swapped);
    • forward: watermark composes with the driver-outbox respool — the replay
      reads only the un-acked suffix with a stable X-Hyp-Batch-Id;
    • blob: a lost watermark after a durable PUT re-PUTs the same object key
      (idempotent overwrite, no duplicate blob).
  • hypaware-core/smoke/flows/incremental_sink_compaction.js — a hermetic
    acceptance smoke proving the blob sink (parquet + local-fs) reads straight
    through a compaction generation swap via the real driver across three ticks.
  • Adds an explicit {#exactly-once-argument} anchor to LLP 0040 §5 so the new
    @refs resolve.

Checks

npm test (1446 pass / 1 pre-existing skip), npm run typecheck, npm run lint
all green; npm run smoke -- incremental_sink_compaction passes.

Task-Id: T6

🤖 Generated with Claude Code

Add the LLP 0040 exactly-once proof the T4/T5 unit suites can't give
(they stub storage): a deterministic acceptance test driving the REAL
kernel cache, retention enforcer, maintainCache compaction, BOTH sinks
(central forward + core local-fs blob), and the driver outbox respool.

Covers: ~0 bytes on a no-new-rows tick, ~N on an N-new tick, exactly-once
across a retention front-prune and a compaction generation swap for both
sinks, and watermark vs. driver-outbox respool composition (suffix-only
replay + idempotent batch-id / re-PUT). Adds a hermetic smoke proving the
blob sink reads straight through a compaction generation swap via the real
driver. Anchors LLP 0040 section 5 so the @ref resolves.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Task-Id: T6
@philcunliffe philcunliffe merged commit 35d77bd into integration/incremental-sink-reads Jun 26, 2026
6 checks passed
@philcunliffe philcunliffe deleted the task/incremental-sink-reads/T6 branch June 26, 2026 00:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant