proxy-io: Reference-count Connection objects - #336
Draft
ryanofsky wants to merge 11 commits into
Draft
Conversation
Split connection teardown out of ~Connection into an idempotent disconnect() method, with the destructor delegating to it. This is a behavior-neutral refactor: the same steps run in the same order on destruction. Having a separate disconnect() method allows severing a connection while keeping the Connection object alive, which the next commits use to let shutdown code wait for in-flight server call bodies to finish after a disconnect (bitcoin/bitcoin#35845). Two details are new: - disconnect() cancels the m_on_disconnect handlers before severing the connection. Previously they were implicitly canceled when the TaskSet member was destroyed. When disconnect() is called separately from destruction, this is required for correctness: severing the stream completes m_network.onDisconnect(), and the registered handlers (_Serve, ConnectStream) destroy the Connection object out from under the caller. - disconnect() explicitly releases m_thread_pool and m_thread_map so worker thread teardown happens at disconnect time whether or not the object is destroyed right away. Previously this happened implicitly during member destruction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a per-connection ServerObjectTracker counting live ProxyServer objects, incremented in the ProxyServerBase constructor and decremented in its destructor, with Connection::waitDrained() blocking until the count reaches zero and Connection::pendingServerObjects() exposing it for logging. Disconnecting a connection cancels the KJ promise of an in-flight call, but a C++ server method body already dispatched to a worker thread runs to completion. Counting live server objects turns Cap'n Proto's object lifetime rules into a usable quiescence signal: a ProxyServer object is not destroyed until its outstanding calls finish (the target capability is kept alive for the duration of a call and pinned by post()/PassField via thisCap()), so after disconnect() the count drains to zero exactly when no server call body is still executing. Waiting for that lets shutdown code avoid freeing application state that a still-running call body dereferences (bitcoin/bitcoin#35845). The tracker is held via shared_ptr by the Connection and by every ProxyServer object because objects kept alive by in-flight calls can outlive the Connection on some teardown paths (see ~ProxyServerBase), and their destructors must decrement state that is still valid. It must be declared before m_rpc_system, whose construction creates the bootstrap server object that registers itself with the tracker. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a deterministic mptest regression test for bitcoin/bitcoin#35845: hold a server method body in flight on a worker thread, call Connection::disconnect(), and assert that Connection::waitDrained() blocks until the body finishes and its server object is destroyed. Also covers destroying an already-disconnected connection (~Connection noticing disconnect() has run). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. ReviewsSee the guideline and AI policy for information on the review process. ConflictsNo conflicts as of last run. LLM Linter (✨ experimental)Possible places where named args for integral literals may be used (e.g.
2026-08-14 16:34:03 |
… the m_incoming_connections list. Currently the list holds Connection by value so the view yields Connection&. When keepconn+notrack later changes the list to list<shared_ptr<Connection>>, the accessor will be updated to return a transform view, so Bitcoin Core code that iterates via this accessor compiles unchanged across that type change. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fix a race between a thread exiting after making IPC calls and a
connection being destroyed by its onDisconnect handler on the event loop
thread. The race was between ~ThreadContext destroying the thread-local
request_threads/callback_threads maps with no locking, and the SetThread
cleanup function (run by Connection::disconnect) erasing entries from
those maps on the event loop thread. When the two ran concurrently, both
could destroy the same ProxyClient<Thread> object: the SetThread cleanup
reset m_disconnect_cb just before ~ProxyClient<Thread> checked it
unsynchronized, so the exiting thread proceeded to destroy the object
while the event loop's map erase destroyed it too. The doubled
destruction consumed m_context.cleanup_fns on one thread, so the other
never unregistered the ProxyClientBase disconnect callback, and
Connection::disconnect then invoked that callback on the freed map node
(heap-use-after-free reading m_client, followed by a double free of the
node reported by glibc as "double free or corruption").
Fix by making map entry removal the synchronization point deciding which
side destroys each ProxyClient<Thread>:
- Add an explicit ~ThreadContext that removes map entries one at a time
under Waiter::m_mutex and destroys each removed node after releasing
the mutex (so ~ProxyClient<Thread> can lock EventLoop::m_mutex without
violating lock order), instead of destroying the maps unlocked.
- Change the SetThread cleanup function to look its entry up by
connection key under Waiter::m_mutex instead of dereferencing the
captured map iterator, extract it, and destroy the node outside the
lock, following the same pattern PassField already uses for mp.Context
arguments. If the entry is gone, the owning thread extracted it first
and is responsible for destroying it.
- Guard the removeSyncCleanup call in ~ProxyClient<Thread> with a
m_context.connection check, because when the entry was extracted by
~ThreadContext first, a concurrent disconnect still runs both the
SetThread cleanup (a no-op now) and the ProxyClientBase disconnect
callback, leaving m_disconnect_cb set but pointing at a spliced-out
list iterator that must not be passed to removeSyncCleanup. The
disconnect callback nulls m_context.connection, and posted functions
cannot interleave with Connection::disconnect on the event loop
thread, so a null connection reliably indicates this case.
The race is long-standing and reachable on master via connections
created by ConnectStream, whose onDisconnect handler deletes the client
Connection on the event loop thread when the peer disconnects while an
exiting thread may be running ~ThreadContext. It was exposed by the
"Waiting for in-flight server call to finish after disconnect" test
because commit bb47369f202b62b8b64f5a52984ff2c40d64ecdd ("Fix error
handling when creating clients") extended the delete-on-disconnect
handler to every ProxyClient created with destroy_connection=true,
including the test setup's directly-created client connection: the
server-side disconnect in the test then deleted the client Connection on
the event loop thread exactly while the test's call thread was exiting.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Make Connection objects shared_ptr-owned (created via a new Connection::make() factory whose custom deleter destroys the object on the event loop thread), and have every proxy object share ownership of its connection (ProxyContext::connection becomes shared_ptr<Connection>, populated via enable_shared_from_this). A Connection now always outlives its proxy objects and survives disconnect() as an inert husk until the last reference is dropped, which removes the long-standing rule that ~ProxyServerBase must not dereference m_context.connection. Because server-side proxy objects now hold references back to their connection, constructing the bootstrap server object during the Connection constructor would call shared_from_this() before any shared_ptr owner exists. Server-side setup is therefore split in two: make() constructs the connection, and a new Connection::serve(make_client) method starts the RPC system afterwards. (A side effect is that the member-initialization-order constraint on m_server_objects is gone, since make_client no longer runs during construction.) A consequence of the reference cycle m_rpc_system exports -> ProxyServer -> ProxyContext::connection is that dropping references alone never destroys a connected Connection: disconnect() breaks the cycles, and every teardown path now calls it before releasing its reference. The _Serve remote-disconnect handler now looks its connection up through a weak_ptr and removes it from m_incoming_connections by value instead of capturing a list iterator. This fixes a latent use-after-free: the handler runs from the event loop task set, so disconnectIncoming() could destroy the connection and invalidate the captured iterator between the handler being queued and running. Ipc::disconnectIncoming() behavior is unchanged; it now erases connections from the list in its first sync (keeping them alive via collected references), drains them, and then drops the references. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Remove the per-client disconnect tracking from ProxyClientBase: client objects no longer register a cleanup callback with their Connection, and a disconnect no longer eagerly releases their m_client capability handles or nulls their connection pointers. Neither is necessary now that proxy objects share ownership of their Connection. The connection pointer stays valid after a disconnect because the Connection outlives its proxies, and keeping the capability handle is safe: Cap'n Proto's per-connection state is refcounted and outlives the RPC system as long as handles reference it, with calls on handles of a disconnected connection failing cleanly with DISCONNECTED errors. The handle is simply released (on the event loop thread, since capability refcounts are not thread safe) whenever the client object is eventually destroyed, and clientInvoke checks the connection's m_disconnected flag instead of a nulled pointer, throwing the same 'IPC client method called after disconnect' error as before. This deletes the detach machinery from ~ProxyClientBase, including the FIXME'd duplicate-cleanup code path. Connection::addSyncCleanup remains for its one other user, the per-thread connection maps (see SetThread), which the next commit converts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Update comments to reflect that after the previous commit, the sync cleanup callback list has exactly one remaining purpose: eagerly removing a disconnected connection's ProxyClient<Thread> entries from the thread_local per-thread connection maps (ThreadContext::request_threads / callback_threads) via callbacks registered by SetThread. Unlike interface clients, these entries cannot simply be left alive across a disconnect: they are owned by other threads that may never touch their maps again, and a surviving entry would hold the disconnected Connection object -- and through its EventLoopRef the event loop -- alive indefinitely, preventing the loop from ever exiting. (Replacing the callbacks with lazy garbage collection in SetThread was tried and hangs mptest for exactly this reason: entries owned by long-lived threads pin the loop after their connection is gone.) So this per-object disconnect tracking is retained by design, now clearly documented as thread-map-specific rather than a general client-object mechanism. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Now that proxy objects hold shared ownership of their Connection, a ProxyServer object kept alive by an in-flight call can no longer outlive the Connection, so ~ProxyServerBase can always reach the tracker through m_context.connection. Drop the shared_ptr indirection that existed to keep the tracker valid past the Connection's death, and the separate tracker handle member on ProxyServerBase. No behavior change; Connection::waitDrained() semantics are identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Needed because changing m_incoming_connections type is an API change.
Collaborator
Author
|
Rebased a067599 -> 5187179 ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Use reference counting to manage Connection object lifetimes. This implements an old idea from #176 (comment) and has two benefits:
disconnectandwaitDrainedmethods #335 for server objects, and this PR extends it to treat client and server objects symmetrically.ProxyClientobjects register with Connections, soConnectionobjects no longer need to store lists ofProxyClientobjects and can just use use counts instead.This is based on #335. The non-base commits are:
aa49a11proxy-io: manage Connection lifetime with shared_ptr5ad8807proxy-io: keep client capability handles across disconnect17b17f3doc: scope Connection cleanup callbacks to per-thread map entriesb13aae1proxy-io: make server object tracker a plain Connection member5187179ci: Check out bitcoin/bitcoin PR #35932 instead of master