Thanks to visit codestin.com
Credit goes to github.com

Skip to content

gh-132869: Fix crash due to memory ordering problem in dictobject und… #133593

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

hawkinsp
Copy link
Contributor

@hawkinsp hawkinsp commented May 7, 2025

…er free threading.

Currently the reads and writes to a dictionary entry and to its contents are unordered. For example in do_lookup we have the following code:

    for (;;) {
        ix = dictkeys_get_index(dk, i);
        if (ix >= 0) {
            int cmp = check_lookup(mp, dk, ep0, ix, key, hash);

where dictkeys_get_index performs a relaxed atomic read of dk_indices[i], where the check_lookup function might be, say, compare_unicode_unicode which makes a relaxed load of the me_key value in that index.

    PyDictUnicodeEntry *ep = &((PyDictUnicodeEntry *)ep0)[ix];
    PyObject *ep_key = FT_ATOMIC_LOAD_PTR_RELAXED(ep->me_key);
    assert(ep_key != NULL);

However, the writer also does not order these two writes appropriately; for example insert_combined_dict does the following:

    Py_ssize_t hashpos = find_empty_slot(mp->ma_keys, hash);
    dictkeys_set_index(mp->ma_keys, hashpos, mp->ma_keys->dk_nentries);

    if (DK_IS_UNICODE(mp->ma_keys)) {
        PyDictUnicodeEntry *ep;
        ep = &DK_UNICODE_ENTRIES(mp->ma_keys)[mp->ma_keys->dk_nentries];
        STORE_KEY(ep, key);
        STORE_VALUE(ep, value);
    }
    else {
        PyDictKeyEntry *ep;
        ep = &DK_ENTRIES(mp->ma_keys)[mp->ma_keys->dk_nentries];
        STORE_KEY(ep, key);
        STORE_VALUE(ep, value);
        STORE_HASH(ep, hash);
    }
    mp->ma_version_tag = new_version;

where the dk_indices value is set first, followed by setting the me_key, both as relaxed writes. This is problematic because the write to dk_indices may be ordered first, either by the program order on x86, or allowed by the relaxed memory ordering semantics of ARM. The reader above will be able to observe a state where the index has been set but the key value is still null, leading to a crash in the reproducer of #132869.

The fix is two-fold:

  • order the index write after the the write to its contents, and
  • use sequentially consistent reads and writes. It would suffice to use load-acquire and store-release here but those atomic operations do not exist in the CPython atomic headers at the necessary types.

I was only able to reproduce the crash under CPython 3.13, but I do not see any reason the bug is fixed on the 3.14 branch either since the code does not seem to have changed.

Fixes #132869

@hawkinsp
Copy link
Contributor Author

hawkinsp commented May 7, 2025

@colesbury

@colesbury colesbury self-requested a review May 7, 2025 17:36
@hawkinsp
Copy link
Contributor Author

hawkinsp commented May 7, 2025

It looks like 3.15 was just branched and all the presubmits are red. I guess if I retry in a bit things may work again.

Copy link
Contributor

@sharktide sharktide left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Most changes to python, including crash bug fixes require a news entry.

…ct under free threading.

Currently the reads and writes to a dictionary entry and to its contents are
unordered. For example in `do_lookup` we have the following code:
```
    for (;;) {
        ix = dictkeys_get_index(dk, i);
        if (ix >= 0) {
            int cmp = check_lookup(mp, dk, ep0, ix, key, hash);
```
where `dictkeys_get_index` performs a relaxed atomic read of `dk_indices[i]`,
where the `check_lookup` function might be, say, compare_unicode_unicode which
makes a relaxed load of the me_key value in that index.
```
    PyDictUnicodeEntry *ep = &((PyDictUnicodeEntry *)ep0)[ix];
    PyObject *ep_key = FT_ATOMIC_LOAD_PTR_RELAXED(ep->me_key);
    assert(ep_key != NULL);
```

However, the writer also does not order these two writes appropriately; for
example `insert_combined_dict` does the following:

```
    Py_ssize_t hashpos = find_empty_slot(mp->ma_keys, hash);
    dictkeys_set_index(mp->ma_keys, hashpos, mp->ma_keys->dk_nentries);

    if (DK_IS_UNICODE(mp->ma_keys)) {
        PyDictUnicodeEntry *ep;
        ep = &DK_UNICODE_ENTRIES(mp->ma_keys)[mp->ma_keys->dk_nentries];
        STORE_KEY(ep, key);
        STORE_VALUE(ep, value);
    }
    else {
        PyDictKeyEntry *ep;
        ep = &DK_ENTRIES(mp->ma_keys)[mp->ma_keys->dk_nentries];
        STORE_KEY(ep, key);
        STORE_VALUE(ep, value);
        STORE_HASH(ep, hash);
    }
    mp->ma_version_tag = new_version;
```

where the `dk_indices` value is set first, followed by setting the `me_key`,
both as relaxed writes. This is problematic because the write to
`dk_indices` may be ordered first, either by the program order on x86, or
allowed by the relaxed memory ordering semantics of ARM. The reader above
will be able to observe a state where the index has been set but the key value
is still null, leading to a crash in the reproducer of python#132869.

The fix is two-fold:
* order the index write after the the write to its contents, and
* use sequentially consistent reads and writes. It would suffice to use
  load-acquire and store-release here but those atomic operations do not exist
  in the CPython atomic headers at the necessary types.

I was only able to reproduce the crash under CPython 3.13, but I do not
see any reason the bug is fixed on the 3.14 branch either since the code
does not seem to have changed.

Fixes python#132869
@hawkinsp
Copy link
Contributor Author

hawkinsp commented May 8, 2025

Most changes to python, including crash bug fixes require a news entry.

Done.

Copy link
Contributor

@sharktide sharktide left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was only able to reproduce the crash under CPython 3.13, but I do not see any reason the bug is fixed on the 3.14 branch either since the code does not seem to have changed.

Could there be other possibilities for 3.14 not failing? If so, this PR might not be necessary for the 3.14 branch, only the 3.13 branch

@colesbury
Copy link
Contributor

Thanks for the bug report and repro. I'm not sure we should go with this approach:

  • compare_unicode_unicode is intended to be called with the dictionary locked. We have compare_unicode_unicode_threadsafe and similar functions for when there may be concurrent modifications
  • I don't think this handles deletion from a dict properly. I think you can still end up seeing a NULL key here if somebody deletes from the dictionary concurrently (see delitem_common).

I think _PyDictKeys_StringLookup() either needs to use lock the dict (or keys in a split dict) or use unicodekeys_lookup_unicode_threadsafe.

It looks like the bug still exists in 3.14 in specialize.c, but not in _PyObject_TryGetInstanceAttribute which the repro uses because that now calls the safe function _PyDictKeys_StringLookupSplit.

@hawkinsp
Copy link
Contributor Author

hawkinsp commented May 8, 2025

Sure, makes sense to me. I'm not that familiar with this code. Do you want to make that change?

@colesbury
Copy link
Contributor

Sure, I'll make the change.

After looking at the code more, I don't think the bug exists in 3.14. The _PyDictKeys_StringLookup calls all happen when the dictionary is locked.

@hawkinsp
Copy link
Contributor Author

hawkinsp commented May 8, 2025

Closing per discussion above.

@hawkinsp hawkinsp closed this May 8, 2025
@colesbury
Copy link
Contributor

See #133700

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Crash due to racy read in dictobject do_lookup under free threading
3 participants