_remove_obj_from_tls to fix deadlock involving _stash_obj_in_tls (#182083)#182083
_remove_obj_from_tls to fix deadlock involving _stash_obj_in_tls (#182083)#182083williamwen42 wants to merge 1 commit into
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/182083
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 024f390 with merge base 0051214 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@williamwen42 has exported this pull request. If you are a Meta employee, you can view the originating Diff in D103314406. |
|
Actually, thinking about this some more, wouldn't this be subject to the same deadlock/race described in #180701? |
…2083) Summary: Fix deadlock caused by #173568, which stashes a contextvars.Context as a SafePyObject in C++ thread_local storage during every backward() call. When a C++ thread exits, __call_tls_dtors destroys the thread_local and the SafePyObject destructor tries to acquire the GIL — deadlocking if the GIL holder is blocked on that thread (as in UP's GpuThreadGroup). The old cleanup (_stash_obj_in_tls("context", None)) replaced the entry with a SafePyObject(None) but still left a SafePyObject in the map. This adds _remove_obj_from_tls which erases the key entirely, so no SafePyObject survives to thread exit. Test Plan: Green signals Differential Revision: D103314406
3fd07d4 to
f997a4f
Compare
…2083) Summary: Fix deadlock caused by #173568, which stashes a contextvars.Context as a SafePyObject in C++ thread_local storage during every backward() call. When a C++ thread exits, __call_tls_dtors destroys the thread_local and the SafePyObject destructor tries to acquire the GIL — deadlocking if the GIL holder is blocked on that thread (as in UP's GpuThreadGroup). The old cleanup (_stash_obj_in_tls("context", None)) replaced the entry with a SafePyObject(None) but still left a SafePyObject in the map. This adds _remove_obj_from_tls which erases the key entirely, so no SafePyObject survives to thread exit. Test Plan: Green signals test_correctness_fsdp_semi_frozen and test_correctness_one_pp_clip_mixed under UP tests passing locally Reviewed By: malfet Differential Revision: D103314406
f997a4f to
6b9bc6e
Compare
…2083) Summary: Fix deadlock caused by #173568, which stashes a contextvars.Context as a SafePyObject in C++ thread_local storage during every backward() call. When a C++ thread exits, __call_tls_dtors destroys the thread_local and the SafePyObject destructor tries to acquire the GIL — deadlocking if the GIL holder is blocked on that thread (as in UP's GpuThreadGroup). The old cleanup (_stash_obj_in_tls("context", None)) replaced the entry with a SafePyObject(None) but still left a SafePyObject in the map. This adds _remove_obj_from_tls which erases the key entirely, so no SafePyObject survives to thread exit. Test Plan: Green signals test_correctness_fsdp_semi_frozen and test_correctness_one_pp_clip_mixed under UP tests passing locally Reviewed By: malfet Differential Revision: D103314406
6b9bc6e to
f682352
Compare
…2083) Summary: Fix deadlock caused by #173568, which stashes a contextvars.Context as a SafePyObject in C++ thread_local storage during every backward() call. When a C++ thread exits, __call_tls_dtors destroys the thread_local and the SafePyObject destructor tries to acquire the GIL — deadlocking if the GIL holder is blocked on that thread (as in UP's GpuThreadGroup). The old cleanup (_stash_obj_in_tls("context", None)) replaced the entry with a SafePyObject(None) but still left a SafePyObject in the map. This adds _remove_obj_from_tls which erases the key entirely, so no SafePyObject survives to thread exit. Test Plan: Green signals test_correctness_fsdp_semi_frozen and test_correctness_one_pp_clip_mixed under UP tests passing locally Reviewed By: malfet Differential Revision: D103314406
f682352 to
024f390
Compare
|
@pytorchbot merge (Initiating merge automatically since Phabricator Diff has merged) |
Merge failedReason: This PR needs a If not, please add the To add a label, you can comment to pytorchbot, for example For more information, see Details for Dev Infra teamRaised by workflow job |
|
@pytorchmergebot merge -f "all looks good" |
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Summary:
Fix deadlock caused by #173568, which stashes a contextvars.Context as a SafePyObject in C++ thread_local storage during every backward() call. When a C++ thread exits, __call_tls_dtors destroys the thread_local and the SafePyObject destructor tries to acquire the GIL — deadlocking if the GIL holder is blocked on that thread (as in UP's GpuThreadGroup).
The old cleanup (_stash_obj_in_tls("context", None)) replaced the entry with a SafePyObject(None) but still left a SafePyObject in the map. This adds _remove_obj_from_tls which erases the key entirely, so no SafePyObject survives to thread exit.
Test Plan:
Green signals
test_correctness_fsdp_semi_frozen and test_correctness_one_pp_clip_mixed under UP tests passing locally
Reviewed By: malfet
Differential Revision: D103314406