I recently added a test in test_io.test_memoryio using _testcapi.set_nomemory() to inject a MemoryError: commit 23eda00. The test was flagged as "data race" by clang ThreadSanitizer on a GHA CI run.
- The main thread calls
_testcapi.set_nomemory() which calls PyMem_SetAllocator() which is implemented as set_allocator_unlocked(): its writes into _PyRuntime.allocators.standard.raw (type PyMemAllocatorEx).
- The faulthandler thread calls
PyMem_RawFree() which reads from _PyRuntime.allocators.standard.raw.
PyMem_SetAllocator() acquires a lock, but PyMem_RawFree() doesn't acquire this lock.
This race condition was already seen with tracemalloc:
Using _testcapi.set_nomemory() in a multithreaded application is used, but this function should only be used for testing purpose. So I propose skipping the test if Python is built with TSAN (./configure --with-thread-sanitizer with clang).
Logs:
WARNING: ThreadSanitizer: data race (pid=15518)
Write of size 8 at 0x55b013af2f80 by main thread:
#0 __tsan_memcpy <null> (python+0xfc452) (BuildId: 77d4ccbb4bf7bb80d66adcdbb8a43d9cd1f555bc)
#1 set_allocator_unlocked /home/runner/work/cpython/cpython/Objects/obmalloc.c (python+0x38417f) (BuildId: 77d4ccbb4bf7bb80d66adcdbb8a43d9cd1f555bc)
#2 PyMem_SetAllocator /home/runner/work/cpython/cpython/Objects/obmalloc.c:1143:5 (python+0x38417f)
#3 fm_setup_hooks /home/runner/work/cpython/cpython/./Modules/_testcapi/mem.c:160:5 (_testcapi.cpython-316d-x86_64-linux-gnu.so+0x353e9) (BuildId: 97ab4e2a64a9d371fd1394f000a763262bd7b835)
#4 fm_set_nomemory /home/runner/work/cpython/cpython/./Modules/_testcapi/mem.c:189:5 (_testcapi.cpython-316d-x86_64-linux-gnu.so+0x353e9)
#5 set_nomemory /home/runner/work/cpython/cpython/./Modules/_testcapi/mem.c:202:5 (_testcapi.cpython-316d-x86_64-linux-gnu.so+0x34590) (BuildId: 97ab4e2a64a9d371fd1394f000a763262bd7b835)
#6 cfunction_call /home/runner/work/cpython/cpython/Objects/methodobject.c:575:18 (python+0x34e22f) (BuildId: 77d4ccbb4bf7bb80d66adcdbb8a43d9cd1f555bc)
#7 _PyObject_MakeTpCall /home/runner/work/cpython/cpython/Objects/call.c:242:18 (python+0x29318b) (BuildId: 77d4ccbb4bf7bb80d66adcdbb8a43d9cd1f555bc)
#8 _PyObject_VectorcallTstate /home/runner/work/cpython/cpython/./Include/internal/pycore_call.h:142:16 (python+0x292ac9) (BuildId: 77d4ccbb4bf7bb80d66adcdbb8a43d9cd1f555bc)
#9 PyObject_Vectorcall /home/runner/work/cpython/cpython/Objects/call.c:327:12 (python+0x294410) (BuildId: 77d4ccbb4bf7bb80d66adcdbb8a43d9cd1f555bc)
(...)
Previous read of size 8 at 0x55b013af2f80 by thread T1:
#0 PyMem_RawFree /home/runner/work/cpython/cpython/Objects/obmalloc.c:1239:5 (python+0x384578) (BuildId: 77d4ccbb4bf7bb80d66adcdbb8a43d9cd1f555bc)
#1 pythread_wrapper /home/runner/work/cpython/cpython/Python/thread_pthread.h:234:5 (python+0x66c445) (BuildId: 77d4ccbb4bf7bb80d66adcdbb8a43d9cd1f555bc)
Location is global '_PyRuntime' of size 266672 at 0x55b013af2b70 (python+0xa46f80)
Thread T1 (tid=15538, running) created by main thread at:
#0 pthread_create <null> (python+0x1010de) (BuildId: 77d4ccbb4bf7bb80d66adcdbb8a43d9cd1f555bc)
#1 do_start_joinable_thread /home/runner/work/cpython/cpython/Python/thread_pthread.h:283:14 (python+0x66b1db) (BuildId: 77d4ccbb4bf7bb80d66adcdbb8a43d9cd1f555bc)
#2 PyThread_start_new_thread /home/runner/work/cpython/cpython/Python/thread_pthread.h:338:9 (python+0x66b28a) (BuildId: 77d4ccbb4bf7bb80d66adcdbb8a43d9cd1f555bc)
#3 faulthandler_dump_traceback_later_impl /home/runner/work/cpython/cpython/./Modules/faulthandler.c:883:9 (python+0x6a4c64) (BuildId: 77d4ccbb4bf7bb80d66adcdbb8a43d9cd1f555bc)
#4 faulthandler_dump_traceback_later /home/runner/work/cpython/cpython/./Modules/clinic/faulthandler.c.h:439:20 (python+0x6a4c64)
(...)
Linked PRs
I recently added a test in
test_io.test_memoryiousing_testcapi.set_nomemory()to inject a MemoryError: commit 23eda00. The test was flagged as "data race" by clang ThreadSanitizer on a GHA CI run._testcapi.set_nomemory()which callsPyMem_SetAllocator()which is implemented asset_allocator_unlocked(): its writes into_PyRuntime.allocators.standard.raw(typePyMemAllocatorEx).PyMem_RawFree()which reads from_PyRuntime.allocators.standard.raw.PyMem_SetAllocator()acquires a lock, butPyMem_RawFree()doesn't acquire this lock.This race condition was already seen with
tracemalloc:test_tracemalloc_track_racesegfault on buildbot #143143 (comment)tracemallocaborts when run from threads in no-gil #126315 (comment)Using
_testcapi.set_nomemory()in a multithreaded application is used, but this function should only be used for testing purpose. So I propose skipping the test if Python is built with TSAN (./configure --with-thread-sanitizerwith clang).Logs:
Linked PRs