ETL conversation - #191
Open
davidabram wants to merge 6 commits into
Open
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
davidabram
force-pushed
the
etl-conversation
branch
from
August 8, 2026 21:58
e2b0e20 to
fb2976a
Compare
Extract bounded, ordered agent trace batches from consistent read snapshots without reserving the source database write lock. Add contention-only retry with rollback between attempts so concurrent hook writers can continue while transient database locks recover safely. Document the ETL boundary and shared read-transaction seam, with coverage for batching, snapshot visibility, concurrent writes, and retry classification. Plan: incremental-agent-trace-etl-transactional-watermarks (T02) Co-authored-by: SCE <sce@crocoder.dev>
Centralize source-contention retries, batch-size validation, watermark access, and batch accounting so Agent Trace ETL can share the small mechanics needed by the planned conversation table pipelines. Preserve the existing table-specific extraction and loading behavior while keeping destination watermark updates transactional.\n\nPlan: incremental-conversation-etl, T01 Co-authored-by: SCE <sce@crocoder.dev>
Add bounded, contention-safe source extraction with validated roles and an independent lineage watermark. Load message facts and lineage dimensions atomically, treating matching logical replays as already present and rejecting conflicting role/timestamp data.\n\nPlan: incremental-conversation-etl, T02 Co-authored-by: SCE <sce@crocoder.dev>
Add bounded, short-snapshot extraction and atomic DWH loading for message parts. Validate supported types, preserve exact text with lowercase SHA-256, and detect replay conflicts within source lineage. Track independent parts watermarks so parts can load before parent messages while concurrent source writers remain unblocked. Plan: incremental-conversation-etl (T03) Co-authored-by: SCE <sce@crocoder.dev>
Add a conversation-level API that verifies source metadata once, runs the existing message and part ETLs through the replica, and reports table-level stats with independently configurable batches and progress. Cover initial, incremental, no-op, out-of-order, equal-timestamp, and source-contention behavior while documenting the composition boundary. Plan: incremental-conversation-etl (T04) Co-authored-by: SCE <sce@crocoder.dev>
Document the completed conversation ETL contract for independent message and part watermarks, source-lineage identities, exact content handling, and append-only source semantics. Keep the shared runner and focused test cleanup aligned with the documented behavior and validation evidence.\n\nPlan: incremental-conversation-etl (T05) Co-authored-by: SCE <sce@crocoder.dev>
davidabram
force-pushed
the
etl-conversation
branch
from
August 9, 2026 10:07
fb2976a to
e0945bd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.