fix: F-2026-18797 | [Dual Defense] Universal Client Chain Registry Lifecycle: Stale-Remove Deadlock and Unclosed DB Handles

[Aman035]

F-2026-18797 Chain registry lifecycle: stale-remove deadlock and unclosed DB handles

Two independent defects on the same path, both confirmed.

Variant A, deadlock

The stale sweep held chainsMu.RLock() and called removeChain, which takes chainsMu.Lock(). sync.RWMutex is not reentrant, so the refresh goroutine parks forever while still holding the read lock, and every later reader of the registry blocks behind it.

Fixed by collecting the stale ids under the read lock, releasing, then removing. Extracted as removeStaleChains so the behaviour is directly testable rather than reachable only through a network fetch.

Swept the rest of the file: no other read-locked section calls a method that takes the write lock.

Variant B, leaked handles

getChainDB opens a fresh pool on every call with no cache. Three paths in addChain return after opening without closing (unsupported VM type, NewClient failure, Start failure), and removeChain / StopAll dropped map entries without closing either.

A chain that cannot start is retried every refresh tick, so a persistent misconfiguration grows one pool per tick, each holding up to MaxOpenConns(10) descriptors.

Fixed by tracking the handle per chain and closing it on removal and shutdown, plus a deferred close that releases the handle on every path where ownership does not pass to a live client.

A fourth path the finding does not list

ensurePushChain opens its own database and writes the client straight into the registry, bypassing addChain. Same defect on two failure returns, and on success the handle was never recorded, so StopAll left the push chain database open.

Found by checking what else opens a database rather than stopping at the paths the finding named. Fixed the same way.

Tests

• The stale sweep completes rather than hanging, and the registry is usable afterwards, asserted by acquiring the write lock and by querying the delisted chain.
• Five consecutive failed adds with the same broken config leave zero handles open, checked by capturing each handle the opener returned and confirming it no longer answers.
• Removal and shutdown both close the handles they held; the push chain path is covered for failure and success.
• Mutation checked on every guard: restoring the remove-inside-RLock hangs the deadlock test with its own message, and dropping any of the four closes fails its own test.
• Package is clean under -race.

The database opener is now a field so tests can observe handle lifecycle. The first version of the leak test asserted on a map count and passed under mutation, because a failed add never reaches the map, so the count was unchanged whether or not the handle leaked.