oracle: fix shutdown race in signing orchestrator

[JohnnyLawDGB]

Fixes #441.

Problem

OracleSigningOrchestrator registers itself as a validation-interface subscriber (signing_orchestrator.cpp:195), and Shutdown() destroys it immediately after Stop():

g_signing_orchestrator->Stop();   // UnregisterValidationInterface(this) — non-blocking
g_signing_orchestrator.reset();   // destroys the object a queued callback is about to use

validationinterface.h:27 marks that unregister path DEPRECATED. This is not safe to use when the RPC server or main message handler thread is running, and :32-35 spells out why: "unregistration is nonblocking and can return before the last notification is processed." Both conditions hold at the callsite — init.cpp:280 runs before StopRPC() (:285), before node.scheduler->stop() (:303), and before FlushBackgroundCallbacks() (:334).

So a queued BlockConnected runs on the destroyed object: OnBlockConnected() → CleanupOldSessions() → std::lock_guard<std::mutex> lock(m_sessions_mutex) on freed memory.

Locking a destroyed std::mutex is UB, and the runtimes differ, which is why one bug produced two CI symptoms:

• glibc — the scheduler thread parks in a futex and never wakes, so Shutdown() blocks behind it forever in CScheduler::stop() → the node hangs.
• libc++ — the same operation aborts → exit code -6.

-stopatheight makes the collision near-certain because StartShutdown() fires from KernelNotifications::blockTip() during block connection (kernel_notifications.cpp:63-66), so the emission for that block is in flight exactly as shutdown proceeds.

How far this reaches beyond that option is narrower than the mechanism first suggests, and #441 now carries the measurement: a notification dispatched after Stop() no longer reaches the unregistered orchestrator, so only an already-in-flight emission is dangerous. 300 ordinary digibyte-cli stop trials against a node continuously connecting blocks, unpatched, produced no hangs, no aborts, and no callback-after-teardown. Treat this as a latent use-after-free on an API the header marks unsafe, not a demonstrated operator-facing hazard.

Fix

One call, honouring the documented contract — let the in-flight callback finish before destroying what it touches:

g_signing_orchestrator->Stop();
SyncWithValidationInterfaceQueue();
g_signing_orchestrator.reset();

#441 lists two alternatives I'd be glad to switch to if preferred: RegisterSharedValidationInterface() (the header's own suggestion for race-free cleanup), or moving the destruction after node.scheduler->stop() — the pattern node.peerman already uses, unregistered at init.cpp:296 but destroyed at :309.

Regression test

The use-after-free is a microsecond-wide race and can't be asserted on directly, so src/test/oracle_shutdown_tests.cpp pins the contract that prevents it: Shutdown() must not return while a queued notification is still pending. It occupies the queue with a callback that outlives a non-draining Shutdown(), then checks the callback finished first.

Reverting just the SyncWithValidationInterfaceQueue() line makes it fail, and the failure is not only the assertion — the freed orchestrator gets touched:

test/oracle_shutdown_tests.cpp(53): error: check callback_finished.load() has failed
unknown location(0): fatal error: signal: SIGSEGV, si_code: 128
                     (memory access violation at address: 0x0)

With the fix restored: *** No errors detected.

Testing

Unit: full suite passes on this branch — 3408 cases with the new test, no errors.

Functional: the race needs load to lose — solo, rpc_blockchain.py almost always passes. Harness is 6 concurrent copies on an 8-core box, each with a distinct --portseed so instances can't collide on bind ports:

for i in $(seq 1 6); do test/functional/rpc_blockchain.py --portseed=$i & done; wait

Build	_test_stopatheight failures
unpatched develop (16159311b3)	reproduced on round 1, on two separate attempts
this branch	0 in 60 runs (10 rounds × 6 concurrent)

One of the unpatched reproductions was caught mid-hang, with the scheduler and shutoff threads deadlocked against each other — thread states are in #441.

---

Investigation and patch assisted by AI tooling; the root cause was reproduced and the fix verified locally against develop (16159311b3).