fix(etl): don't delete rows from an ETL migration - #433
Merged
Conversation
This was referenced Aug 4, 2026
0035 added the users current-row arbiter and, because violations already existed, deleted the duplicates first. These migrations run automatically at indexer start, so that made a `go get` on pkg/etl something that silently removes rows from a consumer's production database — a category of change no dependency bump should be able to make. The index stays here: it is the ETL's invariant, and it belongs beside the four 0030 added. The repair moves to the owning application, as api ddl 0237_users_one_current_row_backfill. Ordering is guaranteed rather than hoped for. api's deployment runs `bridge migrate` as a pre-roll Job that every serving Deployment DependsOn, with runMigrations=false on the serving pods, so the ddl migrations complete before an indexer carrying this one can start — and a failed migration blocks the rollout instead of half-applying. If it does run against a database that still has duplicates it now fails with a unique violation instead of repairing them. That is the intended signal: it means the owning application has not run its backfill. Databases built solely by this indexer cannot contain duplicates, so they are unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rickyrombo
force-pushed
the
fix/etl-0035-index-only
branch
from
August 4, 2026 23:50
0996046 to
9aac510
Compare
This was referenced Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #425. Pairs with AudiusProject/api#1010.
Problem
0035added theuserscurrent-row arbiter and, because violations already existed, deleted the duplicates first. ETL migrations run automatically at indexer start (SkipMigrationsis leftfalseby api), so that madego get github.com/OpenAudio/go-openaudio/pkg/etl@…a thing that silently removes rows from a consumer's production database.Whatever the merits of the repair, a dependency bump should not be able to do that.
Split
0030added.0237_users_one_current_row_backfill.Ordering is guaranteed, not hoped for
The index cannot be created while duplicates exist, so the backfill must run first. api's deployment guarantees it:
bridge migrateruns as a pre-roll Job that every serving DeploymentDependsOn, withrunMigrations=falseon the serving pods. The ddl migrations therefore complete before an indexer carrying this migration can start — and a failed migration blocks the rollout instead of half-applying.Both orders verified against a fixture:
Failing loudly in the wrong order is the intended behaviour: it means the owning application has not run its backfill. Databases built solely by this indexer cannot contain duplicates, so they are unaffected.
Note on v1.6.3
pkg/etl/v1.6.3shipped the delete-and-index version of0035. Nothing has applied it —:stableis stillv1.8.2and api has not merged its bump — so no database has run it. api#1009 (which pins v1.6.3) is safe either way: with api#1010 merged, the ddl backfill runs first in the pre-roll Job and0035's delete finds nothing to do.🤖 Generated with Claude Code