Pin cuDSS's dlopen'd threading-layer plugin to prevent a teardown race - #1883
Pin cuDSS's dlopen'd threading-layer plugin to prevent a teardown race#1883ramakrishnap-nv wants to merge 1 commit into
Conversation
cuDSS dlopen()s its threading-layer plugin (CUDSS_MT_LIB_FILE_NAME) via cudssSetThreadingLayer() and dlclose()s it again from cudssDestroy(). Unloading a shared object that spawned its own OpenMP worker threads is unsafe while any of those threads are still parked or running inside it, and this is a reachable teardown race when the barrier solver's cuDSS handle is destroyed while other threads are still active. Open the plugin ourselves once with RTLD_NODELETE so its mapping is never actually unloaded for the life of the process, regardless of how many cudssCreate/cudssDestroy cycles run afterward. cuDSS's own dlopen()/dlclose() pairs keep working normally. Addresses the dlclose()-during-active-threads race described in NVIDIA#1219. Signed-off-by: Ramakrishna Prabhu <ramakrishnap@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
/ok to test |
Local validation notesBuilt and tested locally (conda dev env, real GPU):
So this fix is correct by construction (`RTLD_NODELETE` is the standard fix for "unsafe to `dlclose()` a library while its spawned threads may still be active"), and doesn't regress anything locally or in CI, but I wasn't able to empirically reproduce and then un-reproduce the exact #1219 crash locally. If anyone has the original repro MPS file (or capacity for a similar large concurrent-B&B MIP run), a direct before/after check there would close the loop. |
ramakrishnap-nv
added
bug
CI Test Summary✅ All 31 test job(s) passed. |
|
@ramakrishnap-nv |
|
Can we update the rocky version to 9 or 10? This will solve the issue with the old runtime |
hlinsen
left a comment
hlinsen
left a comment
There was a problem hiding this comment.
This doesn't change the fact that multiple OpenMP runtime instances are loaded in-process; it removes the specific unload-while-active-threads race that's crashing today.
The second bug is fixed in cuDSS 0.8, we'll get it when we upgrade to it.
I think the main issue is the multiple runtimes and reliance on system libgomp.
3 participants
Summary
Addresses the dual-OpenMP-runtime teardown crash in #1219 (also relevant to #1768/#1774): cuDSS
dlopen()s its threading-layer plugin (libcudss_mtlayer_gomp.so.0) viacudssSetThreadingLayer()anddlclose()s it again fromcudssDestroy(). Unloading a shared object that has spawned its own OpenMP worker threads is unsafe while any of those threads are still parked or running inside it — the code they're executing can be unmapped out from under them — which matches the GDB analysis in #1219 (crash insidelibgomp.so.1duringcudssDestroy()teardown, with other threads still active).We looked at reverting the wheel build to GNU libgomp instead of the bundled LLVM libomp (to unify the two runtimes), but the Rocky Linux 8 wheel build image's only available libgomp predates OpenMP 5.0 (
omp_fulfill_event, needed by the fast MPS parser), with no path to a newer one without vendoring a foreign binary — which just recreates the same class of conflict. So this fixes the actual reachable crash mechanism directly instead: open cuDSS's threading-layer plugin ourselves once withRTLD_NODELETE, so the mapping is never actually unloaded for the life of the process, regardless of how manycudssCreate/cudssDestroycycles run afterward. cuDSS's owndlopen()/dlclose()pairs keep working normally.This doesn't change the fact that multiple OpenMP runtime instances are loaded in-process; it removes the specific unload-while-active-threads race that's crashing today.
Test plan