ETL code change - #192
Open
davidabram wants to merge 7 commits into
Open
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
davidabram
force-pushed
the
etl-code-change
branch
from
August 8, 2026 22:19
bc78d63 to
82938d4
Compare
Extract bounded, ordered agent trace batches from consistent read snapshots without reserving the source database write lock. Add contention-only retry with rollback between attempts so concurrent hook writers can continue while transient database locks recover safely. Document the ETL boundary and shared read-transaction seam, with coverage for batching, snapshot visibility, concurrent writes, and retry classification. Plan: incremental-agent-trace-etl-transactional-watermarks (T02) Co-authored-by: SCE <sce@crocoder.dev>
Add a bounded multi-batch bridge that validates repository metadata, extracts source snapshots, and atomically loads Agent Trace facts with lineage watermarks through the lock-owned DWH replica. Preserve idempotent replay, source-lineage handling, contention behavior, and exact JSON hashing while keeping transport synchronization and credentials outside the ETL boundary. Update the durable architecture context and completion evidence for the production API and focused coverage.\n\nPlan: incremental-agent-trace-etl-transactional-watermarks (T04) Co-authored-by: SCE <sce@crocoder.dev>
Expose one strict normalization boundary for unified and structured diff-trace payloads so ETL callers receive explicit errors for malformed, unsupported, and invalid inputs while recent readers retain best-effort skip accounting. Add focused normalization coverage and document the shared parser contract.\n\nPlan: incremental-code-changes-etl (T01) Co-authored-by: SCE <sce@crocoder.dev>
Code-change ETL lacked a bounded source-read and destination-independent transformation boundary. Add ordered integer-ID extraction with shared contention retry, strict patch and structured payload normalization, source metadata preservation, checked code-change metrics, and lowercase SHA-256 hashes of original payloads. Keep destination loading, identity replay, watermark mutation, and runner orchestration out of this foundation.\n\nPlan: incremental-code-changes-etl, T02 Co-authored-by: SCE <sce@crocoder.dev>
Ensure code-change facts are inserted or replay-validated by repository and source lineage, with conflicts rejected instead of overwritten. Commit dimensions, facts, and the diff_traces watermark in one transaction so failed batches leave no partial progress. Plan: incremental-code-changes-etl, T03 Co-authored-by: SCE <sce@crocoder.dev>
Add the bounded CodeChangesEtl runner and replica-owned entrypoint for loading diff_traces into the DWH with independent source-lineage watermarks, atomic replay/conflict handling, and session-only conversation joins. Extend coverage for growth, batching, invalid-row replay, lineage isolation, contention, and incremental behavior, and document the public contract.\n\nPlan: incremental-code-changes-etl, T04 Co-authored-by: SCE <sce@crocoder.dev>
Replay validation duplicated the destination row shape as an untyped tuple, making the identity check harder to audit alongside the strict ETL contract. Represent stored code-change fields explicitly and centralize the lookup so identical replays remain idempotent while field mismatches still fail without overwrite. Document the exact source projection, pre-transaction transformation, required tool metadata, source-lineage watermark, atomic loading, and session-only relationship so the implementation and its operational boundary remain discoverable. Plan: incremental-code-changes-etl (T01, T02, T03, T04, T05) Co-authored-by: SCE <sce@crocoder.dev>
davidabram
force-pushed
the
etl-code-change
branch
from
August 9, 2026 10:11
82938d4 to
0d67ab3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.