fix(etl): index genesis-migration entities without data loss

[rickyrombo]

Problem

The ETL rejects almost everything genesis-writer replays. Indexing a real migration (54.3M txs from a 2026-06-30 prod snapshot) produced 41,809 rejections and zero rows in users, tracks, playlists, follows, saves, reposts:

entity manager migration validation rejected  entity_type=User action=Create
entity_id=7415  reason="user id 7415 below offset 3000000"

indexer.go converted ManageEntityLegacyMigration into a plain ManageEntityLegacy and dispatched it through the production handlers, so validations meant for newly submitted entities were applied to replayed historical state:

check	impact if unfixed
legacy ID offsets (user 3M / track 2M / playlist 400k)	65% of users (2,054,760), 54% of tracks (788,794), 45% of playlists (121,120) — ids are below the offsets by definition
content limits + reserved handles	~310 users, 48 tracks/playlists that predate the rules
wallet_signature / dashboard ecrecover proofs	all 28,927 associated wallets, all 298 dashboard wallet users

This contradicts cmd/genesis-writer/README.md, which states the distinct migration proto type exists so indexers verify only the migration authority.

Approach: don't teach the handlers about migration

Dispatcher.Clone() derives the migration handler set from the production one; RegisterMigrationOverrides replaces only the handlers whose validation policy differs. Every other entity keeps production behavior automatically, so the two registration lists cannot drift.

The migration handlers reuse the production insert functions verbatim (insertUser, insertTrackAndRoute, insertPlaylistAndRoute, insertAssociatedWallet, insertFollow, …) — one implementation of how an entity is written, so route generation, stems, aggregates and triggers stay shared.

Production validators are unchanged and contain zero migration conditionals. All policy lives in entity_manager/migration.go, which documents what each override drops and why. Production-side changes are extract-function refactors only.

Also fixes silent state loss

• insertUser hardcoded is_verified/is_deactivated/is_available — 3,181 verified users were being silently unverified. State now travels in metadata, and only the migration handler may set it, so a new account still cannot self-assign verification (TestProductionInsertUser_IgnoresStateMetadata).
• Orphaned content — the writer filtered out deactivated (61,528) and unavailable (1,416) users while still migrating their content, orphaning 65,634 tracks and 9,774 playlists that the ETL then correctly rejected as "user does not exist". All current rows now migrate with their state, including soft-deleted tracks/playlists/social rows, so a parity check can distinguish an intentional omission from real data loss.
• Associated wallets are signed as the owning user, not the wallet being linked. The indexer takes the wallet from metadata and uses the signer only to authorize against the user, so signer authority is still enforced rather than bypassed — the only dropped check is the unreplayable ownership proof.

Subtleties worth knowing

• No omitempty on state flags. It drops false, and the indexer cannot distinguish absent from false (is_available defaults to true), so an unavailable row would silently become available.
• Flags must live in the inner data envelope. NewParams unwraps metadata["data"]; fields outside it are invisible to MetadataBoolOr.

Unrelated genesis-writer robustness (found running this end to end)

• maintenance_work_mem → 2GB for the post-load index rebuild. At 1GB the btree sort fails with invalid memory alloc request size on a 54M-row core_transactions; 2GB succeeds. Previously the run died after all data was written.
• synchronous_commit = off on the dst pool, so an external --dst-dsn isn't slowed by a per-block fsync.

Testing

pkg/etl builds and its full suite passes. New migration_test.go covers: legacy rows accepted; production still rejects the same rows; account state round-trips into the INSERT (verified/deactivated/unavailable/defaults); production insert ignores state metadata; social is_delete piped through; validation failure blocks the write; overrides replace only the intended handlers without mutating the production set.

Verified end to end against the real snapshot: users went from 0 indexed / 41,809 rejections to 3,090,600 indexed at exactly 10k/block with 0 rejections (the 4 shortfall are duplicate is_current rows in the source, correctly deduped).

make lint fails on this repo before this change (pre-existing findings in pkg/mediorum, pkg/core/console); none of the findings are in files this PR touches.

Not included

Soft-deleted muted_users still needs the same treatment, and pkg/etl/parity's em_block boundary logic was written for the Python→Go cutover and should be reviewed before its match rate is trusted on an all-genesis dataset.

🤖 Generated with Claude Code

[rickyrombo]

Added a second commit extending the parity tool to cover plays, which the 13-table comparison didn't touch — so a migration could lose or corrupt play history without parity noticing.

Plays can't use the existing strategy (load every ETL row, one source lookup per row) at ~40M rows, so it uses three layers instead:

1. Total row counts — catches wholesale loss.
2. Full merge-walk of aggregate_plays (~1.7M rows, read once per side, no per-row queries). Because it holds the per-track play count, this covers all 40M plays in aggregate — a dropped or misattributed subset surfaces here even though no individual play was inspected.
3. Bounded random sample (default 2000, --plays-sample) compared field by field, to catch corruption that preserves counts. TABLESAMPLE keeps it cheap where ORDER BY random() would not.

Two data properties the comparison has to respect, both of which would otherwise produce false results:

• user_id is NULL for anonymous plays (~47% of rows), so sample lookups match with IS NOT DISTINCT FROM.
• (user_id, play_item_id, created_at) is not unique — the source has ~1.8M duplicate triples — so the lookup compares occurrence counts rather than assuming one row.

Only the fields genesis-writer actually carries are compared (user_id, play_item_id, created_at, city, region, country); source, slot and signature aren't in the migration payload and would report as false differences.

Off by default (--plays).

A full end-to-end validation run is in progress against the 2026-06-30 snapshot: regenerating the genesis with these writer fixes, then booting a node from that output and indexing it with the fixed ETL, then running parity. I'll post the results here.