Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Build error on macOS ARM no GIL #857

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
jaraco opened this issue Mar 5, 2025 · 6 comments
Open

Build error on macOS ARM no GIL #857

jaraco opened this issue Mar 5, 2025 · 6 comments

Comments

@jaraco
Copy link

jaraco commented Mar 5, 2025

Attempting to build on Python 3.14t on macOS on ARM, I get the following failure:

      building '_sodium' extension
      creating build/temp.macosx-15.1-arm64-cpython-314t/build/temp.macosx-15.1-arm64-cpython-314t
      gcc -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall -I/opt/python/include/python3.14t -Ibuild/temp.macosx-15.1-arm64-cpython-314t/include -c build/temp.macosx-15.1-arm64-cpython-314t/_sodium.c -o build/temp.macosx-15.1-arm64-cpython-314t/build/temp.macosx-15.1-arm64-cpython-314t/_sodium.o
      In file included from build/temp.macosx-15.1-arm64-cpython-314t/_sodium.c:57:
      /opt/python/include/python3.14t/Python.h:51:4: error: "The limited API is not currently supported in the free-threaded build"
         51 | #  error "The limited API is not currently supported in the free-threaded build"
            |    ^
      In file included from build/temp.macosx-15.1-arm64-cpython-314t/_sodium.c:57:
      In file included from /opt/python/include/python3.14t/Python.h:72:
      /opt/python/include/python3.14t/object.h:146:5: error: unknown type name 'PyMutex'
        146 |     PyMutex ob_mutex;           // per-object lock
            |     ^
      /opt/python/include/python3.14t/object.h:299:5: error: call to undeclared function '_Py_atomic_store_ssize_relaxed'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        299 |     _Py_atomic_store_ssize_relaxed(&ob->ob_size, size);
            |     ^
      In file included from build/temp.macosx-15.1-arm64-cpython-314t/_sodium.c:57:
      In file included from /opt/python/include/python3.14t/Python.h:73:
      /opt/python/include/python3.14t/refcount.h:97:26: error: call to undeclared function '_Py_atomic_load_uint32_relaxed'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
         97 |         uint32_t local = _Py_atomic_load_uint32_relaxed(&ob->ob_ref_local);
            |                          ^
      /opt/python/include/python3.14t/refcount.h:101:29: error: call to undeclared function '_Py_atomic_load_ssize_relaxed'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        101 |         Py_ssize_t shared = _Py_atomic_load_ssize_relaxed(&ob->ob_ref_shared);
            |                             ^
      /opt/python/include/python3.14t/refcount.h:115:13: error: call to undeclared function '_Py_atomic_load_uint32_relaxed'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        115 |     return (_Py_atomic_load_uint32_relaxed(&op->ob_ref_local) ==
            |             ^
      /opt/python/include/python3.14t/refcount.h:115:63: warning: comparison of integers of different signs: 'int' and 'unsigned int' [-Wsign-compare]
        115 |     return (_Py_atomic_load_uint32_relaxed(&op->ob_ref_local) ==
            |             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
        116 |             _Py_IMMORTAL_REFCNT_LOCAL);
            |             ~~~~~~~~~~~~~~~~~~~~~~~~~
      /opt/python/include/python3.14t/refcount.h:146:9: error: call to undeclared function '_Py_IsOwnedByCurrentThread'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        146 |     if (_Py_IsOwnedByCurrentThread(ob)) {
            |         ^
      /opt/python/include/python3.14t/refcount.h:241:22: error: call to undeclared function '_Py_atomic_load_uint32_relaxed'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        241 |     uint32_t local = _Py_atomic_load_uint32_relaxed(&op->ob_ref_local);
            |                      ^
      /opt/python/include/python3.14t/refcount.h:248:9: error: call to undeclared function '_Py_IsOwnedByCurrentThread'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        248 |     if (_Py_IsOwnedByCurrentThread(op)) {
            |         ^
      /opt/python/include/python3.14t/refcount.h:249:9: error: call to undeclared function '_Py_atomic_store_uint32_relaxed'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        249 |         _Py_atomic_store_uint32_relaxed(&op->ob_ref_local, new_local);
            |         ^
      /opt/python/include/python3.14t/refcount.h:252:9: error: call to undeclared function '_Py_atomic_add_ssize'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        252 |         _Py_atomic_add_ssize(&op->ob_ref_shared, (1 << _Py_REF_SHARED_SHIFT));
            |         ^
      /opt/python/include/python3.14t/refcount.h:335:22: error: call to undeclared function '_Py_atomic_load_uint32_relaxed'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        335 |     uint32_t local = _Py_atomic_load_uint32_relaxed(&op->ob_ref_local);
            |                      ^
      /opt/python/include/python3.14t/refcount.h:341:9: error: call to undeclared function '_Py_IsOwnedByCurrentThread'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        341 |     if (_Py_IsOwnedByCurrentThread(op)) {
            |         ^
      /opt/python/include/python3.14t/refcount.h:343:9: error: call to undeclared function '_Py_atomic_store_uint32_relaxed'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        343 |         _Py_atomic_store_uint32_relaxed(&op->ob_ref_local, local);
            |         ^
      /opt/python/include/python3.14t/refcount.h:345:13: error: call to undeclared function '_Py_MergeZeroLocalRefcount'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        345 |             _Py_MergeZeroLocalRefcount(op);
            |             ^
      /opt/python/include/python3.14t/refcount.h:349:9: error: call to undeclared function '_Py_DecRefShared'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
        349 |         _Py_DecRefShared(op);
            |         ^
      1 warning and 16 errors generated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error orig

This may or may not be related to #834.
@alex
Copy link
Member

alex commented Mar 5, 2025

CFFI does not support freethreaded Python at this point, so this is not expected to work, see: python-cffi/cffi#119

@ngoldbaum
Copy link
Contributor

ngoldbaum commented Apr 2, 2025 •
edited
Loading

If I force the build to use our fork of CFFI:

diff --git a/pyproject.toml b/pyproject.toml
index d6cc581..5c91e55 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -3,7 +3,7 @@
 requires = [
     "setuptools>=61.0.0,!=74.0.0",
     "wheel",
-    "cffi>=1.4.1; platform_python_implementation != 'PyPy'",
+    "cffi @ git+https://github.com/quansight-labs/cffi; platform_python_implementation != 'PyPy'",
 ]
 build-backend = "setuptools.build_meta"

After updating from tox to nox: #862

I can build and run the tests on the free-threaded build. I don't see any test failures.

It would probably be helpful for our effort to upstream the changes in our fork of cffi if we could come up with multithreaded tests. Mostly to stress-test cffi.

@ngoldbaum
Copy link
Contributor

I tried building pynacl with TSAN support but hit a build error in libsodium's make check:

      make[3]: Leaving directory '/work/pynacl/build/temp.linux-x86_64-cpython-314t/test/default'
      make  check-TESTS
      make[3]: Entering directory '/work/pynacl/build/temp.linux-x86_64-cpython-314t/test/default'
      make[4]: Entering directory '/work/pynacl/build/temp.linux-x86_64-cpython-314t/test/default'
      PASS: aead_aegis128l
      PASS: aead_aegis256
      PASS: aead_aes256gcm
      PASS: aead_aes256gcm2
      PASS: aead_chacha20poly1305
      PASS: aead_chacha20poly13052
      PASS: aead_xchacha20poly1305
      PASS: auth
      PASS: auth2
      PASS: auth3
      PASS: auth5
      PASS: auth6
      PASS: auth7
      PASS: box
      PASS: box2
      PASS: box7
      PASS: box8
      PASS: box_easy
      PASS: box_easy2
      PASS: box_seal
      PASS: box_seed
      PASS: chacha20
      PASS: codecs
      PASS: core1
      PASS: core2
      PASS: core3
      PASS: core4
      PASS: core5
      PASS: core6
      PASS: ed25519_convert
      PASS: generichash
      PASS: generichash2
      PASS: generichash3
      PASS: hash
      PASS: hash3
      PASS: kdf
      PASS: keygen
      PASS: kx
      PASS: metamorphic
      PASS: misuse
      PASS: onetimeauth
      PASS: onetimeauth2
      PASS: onetimeauth7
      PASS: pwhash_argon2i
      PASS: pwhash_argon2id
      PASS: randombytes
      PASS: scalarmult
      PASS: scalarmult2
      PASS: scalarmult5
      PASS: scalarmult6
      PASS: scalarmult7
      PASS: scalarmult8
      PASS: secretbox
      PASS: secretbox2
      PASS: secretbox7
      PASS: secretbox8
      PASS: secretbox_easy
      PASS: secretbox_easy2
      PASS: secretstream_xchacha20poly1305
      PASS: shorthash
      PASS: sign
      PASS: sodium_core
      PASS: sodium_utils
      PASS: sodium_version
      PASS: stream
      PASS: stream2
      PASS: stream3
      PASS: stream4
      PASS: verify1
      PASS: sodium_utils2
      FAIL: sodium_utils3
      PASS: core_ed25519
      PASS: core_ristretto255
      PASS: kdf_hkdf
      PASS: pwhash_scrypt
      PASS: pwhash_scrypt_ll
      PASS: scalarmult_ed25519
      PASS: scalarmult_ristretto255
      PASS: siphashx24
      PASS: xchacha20
      ============================================================================
      Testsuite summary for libsodium 1.0.20
      ============================================================================
      # TOTAL: 80
      # PASS:  79
      # SKIP:  0
      # XFAIL: 0
      # FAIL:  1
      # XPASS: 0
      # ERROR: 0
      ============================================================================
      See test/default/test-suite.log for debugging.
      Some test(s) failed.  Please report this to https://github.com/jedisct1/libsodium/issues,
      together with the test-suite.log file (gzipped) and your system
      information.  Thanks.
      ============================================================================

I don't see an upstream issue mentioning TSAN so maybe no one has hit this before.

@reaperhulk
Copy link
Member

That test file has a warning in it if you try to compile with asan and appears to be doing a bunch of signal handler testing.

@ngoldbaum
Copy link
Contributor

I opened jedisct1/libsodium#1450. I might hack the libsodium build to ignore that error to see if I can run a test I wrote under TSAN that encrypts a message in chunks in a thread pool as well as the full test suite under pytest-run-parallel. The latter is imperfect because it will skip tests using hypothesis because hypothesis isn't thread-safe.

@ngoldbaum
Copy link
Contributor

I ended up finding some races in our fork of CFFI (Quansight-Labs/cffi#2 (comment)). Nothing worrying and nothing that's problematic on the GIL-enabled build. Thanks for sanity checking what I was seeing earlier @reaperhulk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@alex @reaperhulk @jaraco @ngoldbaum