-
-
Notifications
You must be signed in to change notification settings - Fork 32.3k
gh-132869: Fix crash due to memory ordering problem in dictobject und… #133593
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
It looks like 3.15 was just branched and all the presubmits are red. I guess if I retry in a bit things may work again. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Most changes to python, including crash bug fixes require a news entry.
…ct under free threading. Currently the reads and writes to a dictionary entry and to its contents are unordered. For example in `do_lookup` we have the following code: ``` for (;;) { ix = dictkeys_get_index(dk, i); if (ix >= 0) { int cmp = check_lookup(mp, dk, ep0, ix, key, hash); ``` where `dictkeys_get_index` performs a relaxed atomic read of `dk_indices[i]`, where the `check_lookup` function might be, say, compare_unicode_unicode which makes a relaxed load of the me_key value in that index. ``` PyDictUnicodeEntry *ep = &((PyDictUnicodeEntry *)ep0)[ix]; PyObject *ep_key = FT_ATOMIC_LOAD_PTR_RELAXED(ep->me_key); assert(ep_key != NULL); ``` However, the writer also does not order these two writes appropriately; for example `insert_combined_dict` does the following: ``` Py_ssize_t hashpos = find_empty_slot(mp->ma_keys, hash); dictkeys_set_index(mp->ma_keys, hashpos, mp->ma_keys->dk_nentries); if (DK_IS_UNICODE(mp->ma_keys)) { PyDictUnicodeEntry *ep; ep = &DK_UNICODE_ENTRIES(mp->ma_keys)[mp->ma_keys->dk_nentries]; STORE_KEY(ep, key); STORE_VALUE(ep, value); } else { PyDictKeyEntry *ep; ep = &DK_ENTRIES(mp->ma_keys)[mp->ma_keys->dk_nentries]; STORE_KEY(ep, key); STORE_VALUE(ep, value); STORE_HASH(ep, hash); } mp->ma_version_tag = new_version; ``` where the `dk_indices` value is set first, followed by setting the `me_key`, both as relaxed writes. This is problematic because the write to `dk_indices` may be ordered first, either by the program order on x86, or allowed by the relaxed memory ordering semantics of ARM. The reader above will be able to observe a state where the index has been set but the key value is still null, leading to a crash in the reproducer of python#132869. The fix is two-fold: * order the index write after the the write to its contents, and * use sequentially consistent reads and writes. It would suffice to use load-acquire and store-release here but those atomic operations do not exist in the CPython atomic headers at the necessary types. I was only able to reproduce the crash under CPython 3.13, but I do not see any reason the bug is fixed on the 3.14 branch either since the code does not seem to have changed. Fixes python#132869
Done. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was only able to reproduce the crash under CPython 3.13, but I do not see any reason the bug is fixed on the 3.14 branch either since the code does not seem to have changed.
Could there be other possibilities for 3.14 not failing? If so, this PR might not be necessary for the 3.14 branch, only the 3.13 branch
Thanks for the bug report and repro. I'm not sure we should go with this approach:
I think It looks like the bug still exists in 3.14 in |
Sure, makes sense to me. I'm not that familiar with this code. Do you want to make that change? |
Sure, I'll make the change. After looking at the code more, I don't think the bug exists in 3.14. The |
Closing per discussion above. |
See #133700 |
…er free threading.
Currently the reads and writes to a dictionary entry and to its contents are unordered. For example in
do_lookup
we have the following code:where
dictkeys_get_index
performs a relaxed atomic read ofdk_indices[i]
, where thecheck_lookup
function might be, say, compare_unicode_unicode which makes a relaxed load of the me_key value in that index.However, the writer also does not order these two writes appropriately; for example
insert_combined_dict
does the following:where the
dk_indices
value is set first, followed by setting theme_key
, both as relaxed writes. This is problematic because the write todk_indices
may be ordered first, either by the program order on x86, or allowed by the relaxed memory ordering semantics of ARM. The reader above will be able to observe a state where the index has been set but the key value is still null, leading to a crash in the reproducer of #132869.The fix is two-fold:
I was only able to reproduce the crash under CPython 3.13, but I do not see any reason the bug is fixed on the 3.14 branch either since the code does not seem to have changed.
Fixes #132869