Skip to content

fix(etl): don't delete rows from an ETL migration - #433

Merged
rickyrombo merged 1 commit into
mainfrom
fix/etl-0035-index-only
Aug 5, 2026
Merged

fix(etl): don't delete rows from an ETL migration#433
rickyrombo merged 1 commit into
mainfrom
fix/etl-0035-index-only

Conversation

@rickyrombo

Copy link
Copy Markdown
Contributor

Follow-up to #425. Pairs with AudiusProject/api#1010.

Problem

0035 added the users current-row arbiter and, because violations already existed, deleted the duplicates first. ETL migrations run automatically at indexer start (SkipMigrations is left false by api), so that made go get github.com/OpenAudio/go-openaudio/pkg/etl@… a thing that silently removes rows from a consumer's production database.

Whatever the merits of the repair, a dependency bump should not be able to do that.

Split

  • index stays here — it's the ETL's invariant, and it belongs beside the four 0030 added.
  • repair moves to the owning application — api ddl 0237_users_one_current_row_backfill.

Ordering is guaranteed, not hoped for

The index cannot be created while duplicates exist, so the backfill must run first. api's deployment guarantees it: bridge migrate runs as a pre-roll Job that every serving Deployment DependsOn, with runMigrations=false on the serving pods. The ddl migrations therefore complete before an indexer carrying this migration can start — and a failed migration blocks the rollout instead of half-applying.

Both orders verified against a fixture:

index first          -> ERROR: could not create unique index "users_current_uniq_idx"
                        DETAIL: Key (user_id)=(98311147) is duplicated.
backfill then index  -> both apply; violations 0; indisvalid = t
re-running either    -> no-op

Failing loudly in the wrong order is the intended behaviour: it means the owning application has not run its backfill. Databases built solely by this indexer cannot contain duplicates, so they are unaffected.

Note on v1.6.3

pkg/etl/v1.6.3 shipped the delete-and-index version of 0035. Nothing has applied it — :stable is still v1.8.2 and api has not merged its bump — so no database has run it. api#1009 (which pins v1.6.3) is safe either way: with api#1010 merged, the ddl backfill runs first in the pre-roll Job and 0035's delete finds nothing to do.

🤖 Generated with Claude Code

0035 added the users current-row arbiter and, because violations already
existed, deleted the duplicates first. These migrations run automatically at
indexer start, so that made a `go get` on pkg/etl something that silently
removes rows from a consumer's production database — a category of change no
dependency bump should be able to make.

The index stays here: it is the ETL's invariant, and it belongs beside the
four 0030 added. The repair moves to the owning application, as api ddl
0237_users_one_current_row_backfill.

Ordering is guaranteed rather than hoped for. api's deployment runs
`bridge migrate` as a pre-roll Job that every serving Deployment DependsOn,
with runMigrations=false on the serving pods, so the ddl migrations complete
before an indexer carrying this one can start — and a failed migration blocks
the rollout instead of half-applying.

If it does run against a database that still has duplicates it now fails with
a unique violation instead of repairing them. That is the intended signal: it
means the owning application has not run its backfill. Databases built solely
by this indexer cannot contain duplicates, so they are unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@rickyrombo
rickyrombo force-pushed the fix/etl-0035-index-only branch from 0996046 to 9aac510 Compare August 4, 2026 23:50
@rickyrombo
rickyrombo merged commit 30b1c1b into main Aug 5, 2026
5 checks passed
@rickyrombo
rickyrombo deleted the fix/etl-0035-index-only branch August 5, 2026 00:12
This was referenced Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant