Enable CPython free-threaded wheel builds #301
Conversation
|
Thanks for this. I have kicked off the tests. I plan to merge #300 ahead of this, so we'll probably need to rebase this one at that oint. |
6362419 to
739aaec
Compare
| run: | | ||
| echo "CIBW_BEFORE_TEST=pip install pytest pytest-run-parallel" >> "$GITHUB_ENV" | ||
| echo "CIBW_ENVIRONMENT=PYLZ4_USE_SYSTEM_LZ4=False PYTEST_ADDOPTS=--parallel-threads=4" >> "$GITHUB_ENV" | ||
| echo "CIBW_TEST_COMMAND=tox -x testenv.deps+=pytest-run-parallel -x testenv.pass_env+=PYTEST_ADDOPTS -c {project}" >> "$GITHUB_ENV" |
There was a problem hiding this comment.
Wouldn't these be better set with an env stanza? If they must be set this way, please add a comment explaining why.
There was a problem hiding this comment.
The echo "VAR=VALUE" >> "$GITHUB_ENV" is the standard preferred way to set variables in Github Actions workflows: https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/store-information-in-variables#passing-values-between-steps-and-jobs-in-a-workflow
| env: | ||
| CIBW_ENVIRONMENT: PYLZ4_USE_SYSTEM_LZ4="False" |
There was a problem hiding this comment.
Why remove this for non free threading builds? We definitely don't want to use the system LZ4 library in all cases, so I am a little confused on the thinking here?
There was a problem hiding this comment.
The variable was absorbed by the redefinition over at line 71
| CIBW_BUILD: ${{ matrix.cibw_build }} | ||
| CIBW_SKIP: "cp*-musllinux*" | ||
| CIBW_TEST_COMMAND: "tox -c {project}" |
There was a problem hiding this comment.
Why is this removed in the general case?
There was a problem hiding this comment.
The actual command was moved to its own definition over at lines 73 and 79
tests/block/test_block_0.py
Outdated
| def test_2(data, mode, store_size, dictionary): | ||
| (c_kwargs, d_kwargs) = setup_kwargs(mode, store_size) | ||
|
|
||
| data_in = [data for i in range(32)] | ||
| def copy_buf(data): | ||
| data_x = data |
There was a problem hiding this comment.
This should really go into an else clause on the following conditional.
|
This all looks pleasantly clean, it'd be nice to move this forward.
This seems to be the one open TODO item - were you able to try this by any chance @andfoy? |
|
@rgommers @jonathanunderwood I've just checked, and there are no TSAN errors when running the test suite in parallel |
Fixes #292
This PR enables support for free-threading builds of
python-lz4, in particular this diff achieves the following goals:pytest-run-parallel, which executes each test in a concurrent fashion. Such check did not raise any major concurrency issues related neither to race conditions nor memory manipulation that could lead to segfaults. The only tests marked to be single-threaded were the ones related to check for memory usage.Overall, this package does not have any major C threading issues, given that the underlying library is thread safe lz4/lz4#887 (comment) (at least when each thread owns its own compression/decompression context).