This branch adds support for the free-threaded (nogil) build of CPython. It includes the work done by Lysandros (use strong references, critical sections for Message objects). I excluded the changes to the "cpp" backend since I think we should focus on support for upb only at this time. These changes are designed such that they shouldn't affect the behavior or performance of the default (non-free-threaded) build.

Lysandraos's PRs superseded by this one:

• Fix thread safety issues when sharing Message objects across threads #23056
• Hold a per-object mutex when modifying PyUpb_WeakMaps #23024
• Use strong reference CPython C APIs under free-threading #22736

Additional changes I made:

• Add recursive mutex implementation, essentially the same as threading.RLock()
• Add locking for the ObjCache structure.
• Add a critical section for the DescriptorPool object.
• Allocate the c_descriptor_symtab data on module init, avoiding thread-safety issue.
• Change PyUpb_WeakMap to use strong references.
• Modify the py_wheel function in python/dist/BUILD.bazel so that free-threaded wheel files and .so files have to correct names (free-threaded build does not yet have a stable ABI).

I have some limited multi-threaded testing that I've run with the TSAN build. These changes fix all TSAN warnings and the tests complete without crashing (previously they caused a SEGV).

I've also tested the multi-threaded scaling of the free-threading build of the modified library. Unfortunately, it doesn't scale well when used from multiple parallel threads. The ObjCache structure is a bottleneck since it is updated on every Message creation and deletion. I think that could be fixed by having some lock-free hash map operatations for ObjCache (i.e. make the fast path where the object already exists not need to acquire the lock). It's not clear to me if upb_inttable_* is thread-safe without locking. I think that scaling work could come later, after we have it working without crashing.

Instructions on testing this:

• Install bazelisk
• Have a Python 3.14t build somewhere, preferably with TSAN and --with-pydebug options on.
• Add the "bin" folder for that Python to your PATH
• I use the following shell script to build with bazel, using Clang as the compiler:

#!/bin/sh
export PATH=$HOME/tmp/py-3.14t-tsan/bin:/usr/local/bin:/usr/bin:/bin
CC=clang CXX=clang++ bazelisk build  --config=tsan --compilation_mode=dbg --noenable_bzlmod //python/dist:binary_wheel

• Install the package: python3 -m pip install --force-reinstall bazel-bin/python/dist/protobuf-6.33.0-cp314-cp314t-linux_x86_64.whl

• Run tests, e.g. python3 pb_threaded.py

Attached is the pb_threaded.py script I was using to test. It is too simple yet and we need some better testing. Perhaps we can use pytest-run-parallel and run the existing unit tests.

pb_threaded.py.txt