Terminate TBB deterministically before MATLAB unloads the MEX file - #206
Open
DLuminary wants to merge 2 commits into
Open
Terminate TBB deterministically before MATLAB unloads the MEX file#206DLuminary wants to merge 2 commits into
DLuminary wants to merge 2 commits into
Conversation
MATLAB crashes on exit after a nonlinear optimization when GTSAM is built with TBB, on macOS. See borglab/gtsam#2489. The crash does not depend on TBB_NUM_THREADS, is unaffected by explicitly deleting wrapped objects, and disappears with -DGTSAM_WITH_TBB=OFF -- so it is not object lifetime. This matches oneTBB issue #977: a library links TBB and uses a parallel construct, the host loads it dynamically, and the process crashes at exit. Reported there on Linux and macOS but not Windows, and noted as racy. The oneTBB maintainer's guidance is that dynamic library unload is a difficult moment for oneTBB, because worker threads are still finishing their routine and may touch addresses in the unmapped image; the recommendation is task_scheduler_handle with finalize() before the unload. _deleteAllObjects is registered with mexAtExit and runs just before MATLAB clears the MEX file, which is the right point to wind the scheduler down. finalize() blocks until the workers have exited. The call is placed after the collector delete loop so that any destructor which itself uses TBB has already run. Version guard ------------- task_scheduler_handle, tbb::attach and finalize() are oneTBB APIs; the attach tag in particular is not present in early oneTBB. GTSAM's HandleTBB.cmake accepts TBB from 4.4 onwards, so the block is guarded on TBB_INTERFACE_VERSION >= 12060 (oneTBB 2021.6) and compiles out entirely on older TBB, leaving those builds byte-identical. Builds without TBB are unaffected via GTSAM_USE_TBB. Verified in isolation, not against MATLAB: - finalize() is idempotent; calling it repeatedly is harmless - the std::nothrow overload returns false rather than throwing if the scheduler is still referenced elsewhere, so it cannot disturb another holder -- relevant because MATLAB ships its own TBB - TBB re-initializes transparently if the module is used again after a clear, so this does not affect the clear/recompile/reload workflow that mexLock() would break - the guarded block compiles both with the guard active and with it forced inactive
Member
|
Oopsies, CI seems to crash |
Author
|
Should be passing now. |
Member
|
I'll let @ProfFan review and merge |
Member
|
@ProfFan ready to go? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MATLAB crashes on exit after a nonlinear optimization when GTSAM is built with TBB, on macOS. See borglab/gtsam#2489. The crash does not depend on TBB_NUM_THREADS, is unaffected by explicitly deleting wrapped objects, and disappears with -DGTSAM_WITH_TBB=OFF -- so it is not object lifetime.
This matches oneTBB issue #977: a library links TBB and uses a parallel construct, the host loads it dynamically, and the process crashes at exit. Reported there on Linux and macOS but not Windows, and noted as racy. The oneTBB maintainer's guidance is that dynamic library unload is a difficult moment for oneTBB, because worker threads are still finishing their routine and may touch addresses in the unmapped image; the recommendation is task_scheduler_handle with finalize() before the unload.
_deleteAllObjects is registered with mexAtExit and runs just before MATLAB clears the MEX file, which is the right point to wind the scheduler down. finalize() blocks until the workers have exited. The call is placed after the collector delete loop so that any destructor which itself uses TBB has already run.
Version guard
task_scheduler_handle, tbb::attach and finalize() are oneTBB APIs; the attach tag in particular is not present in early oneTBB. GTSAM's HandleTBB.cmake accepts TBB from 4.4 onwards, so the block is guarded on TBB_INTERFACE_VERSION >= 12060 (oneTBB 2021.6) and compiles out entirely on older TBB, leaving those builds byte-identical. Builds without TBB are unaffected via GTSAM_USE_TBB.
Verified in isolation, not against MATLAB: