ETL watermarks - #190
Open
davidabram wants to merge 4 commits into
Open
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
davidabram
force-pushed
the
etl-watermarks
branch
from
August 8, 2026 14:01
830a52b to
940325a
Compare
davidabram
force-pushed
the
etl-watermarks
branch
from
August 8, 2026 21:10
940325a to
bda496e
Compare
Expose a typed transaction-scoped execute and query surface over the shared connection. Successful closures commit atomically, while closure and commit failures trigger best-effort rollback without operation retries. Add focused coverage for commit visibility, rollback, database error context, and Agent Trace DWH compatibility. Plan: incremental-agent-trace-etl-transactional-watermarks Task: T01 Co-authored-by: SCE <sce@crocoder.dev>
Extract bounded, ordered agent trace batches from consistent read snapshots without reserving the source database write lock. Add contention-only retry with rollback between attempts so concurrent hook writers can continue while transient database locks recover safely. Document the ETL boundary and shared read-transaction seam, with coverage for batching, snapshot visibility, concurrent writes, and retry classification. Plan: incremental-agent-trace-etl-transactional-watermarks (T02) Co-authored-by: SCE <sce@crocoder.dev>
Add deterministic exact-JSON SHA-256 transformation and idempotent destination loading for agent traces. The batch transaction creates lineage dimensions, verifies replay hashes, rejects integrity conflicts, and advances the source watermark only after all rows succeed, preserving atomic rollback across facts, dimensions, and progress.\n\nPlan: incremental-agent-trace-etl-transactional-watermarks, T03 Co-authored-by: SCE <sce@crocoder.dev>
Add a bounded multi-batch bridge that validates repository metadata, extracts source snapshots, and atomically loads Agent Trace facts with lineage watermarks through the lock-owned DWH replica. Preserve idempotent replay, source-lineage handling, contention behavior, and exact JSON hashing while keeping transport synchronization and credentials outside the ETL boundary. Update the durable architecture context and completion evidence for the production API and focused coverage.\n\nPlan: incremental-agent-trace-etl-transactional-watermarks (T04) Co-authored-by: SCE <sce@crocoder.dev>
davidabram
force-pushed
the
etl-watermarks
branch
from
August 9, 2026 10:03
bda496e to
73be0a4
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.