Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Declare free-threaded Python support on the python_function plugin.#6289

Merged
rostan-t merged 4 commits into
NVIDIA:mainfrom
rostan-t:python_function_free_threaded_support
Apr 13, 2026
Merged

Declare free-threaded Python support on the python_function plugin.#6289
rostan-t merged 4 commits into
NVIDIA:mainfrom
rostan-t:python_function_free_threaded_support

Conversation

@rostan-t
Copy link
Copy Markdown
Collaborator

Category:

Other (e.g. Documentation, Tests, Configuration)

Description:

The DALI backend Pybind11 extension already declares support for free-threaded Python. This PR does the same for python_function_plugin. This is the missing piece for full free-threaded Python support.

Since python_function_plugin is always transitively imported, not declaring free-threaded support results in the GIL always being re-enabled for DALI:

frozen importlib._bootstrap>:491: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'nvidia.dali.python_function_plugin', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.

Additional information:

Affected modules and functionalities:

Key points relevant for the review:

Tests:

  • Existing tests apply
  • New tests added
    • Python tests
    • GTests
    • Benchmark
    • Other
  • N/A

Checklist

Documentation

  • Existing documentation applies
  • Documentation updated
    • Docstring
    • Doxygen
    • RST
    • Jupyter
    • Other
  • N/A

DALI team only

Requirements

  • Implements new requirements
  • Affects existing requirements
  • N/A

REQ IDs: N/A

JIRA TASK: N/A

@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented Apr 13, 2026

Greptile Summary

This PR completes DALI's free-threaded Python support by adding py::mod_gil_not_used() to the python_function_plugin pybind11 module — the last DALI extension that lacked this declaration. Without it, importing python_function_plugin (which is transitively always imported) would re-enable the GIL and emit a RuntimeWarning, defeating the purpose of a free-threaded build. The PR also removes the now-unnecessary PYTHON_GIL environment variable plumbing from build/test scripts, adds a test_gil_disabled() sanity test, and expands the free-threading CI test body to include experimental_mode.test_multithreading.

Confidence Score: 5/5

This PR is safe to merge; all changes are correct, minimal, and consistent with the existing pattern in backend_impl.cc.

The core C++ change mirrors a pattern already in production. Script cleanup is straightforward. No P0/P1 issues identified; all remaining observations are P2 or lower and do not block merge.

No files require special attention.

Important Files Changed

Filename Overview
dali/operators/python_function/dltensor_function.cc Core change: adds py::mod_gil_not_used() to PYBIND11_MODULE, matching the pattern already used in backend_impl.cc. Clean and correct.
dali/test/python/free-threading/test_multithreading.py Adds test_gil_disabled() to verify the GIL stays off after DALI import; removes debug print; import reorganization follows PEP 8.
dali/test/python/experimental_mode/test_multithreading.py Fixes assert_raises glob pattern from "EvalContext" to "*EvalContext*" to make the wildcard intent explicit; functionally equivalent given the glob_to_regex implementation.
qa/setup_test_common.sh Removes now-redundant PYTHON_GIL export and save/restore logic in enable_conda/disable_conda; replaces with a direct python3 -c exit(...) idiom for the version-string conditional.
docker/build_helper.sh Removes PYTHON_GIL export; no longer needed since python_function_plugin now declares GIL-not-used.
qa/TL0_free-threading/test_body.sh Adds experimental_mode.test_multithreading to the free-threading CI test body; drops the -s flag for positional directory arguments.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["Python free-threaded build\n(Py_GIL_DISABLED=1)"] --> B["import nvidia.dali"]
    B --> C["backend_impl.cc\nPYBIND11_MODULE(..., py::mod_gil_not_used())"]
    B --> D["python_function_plugin\nPYBIND11_MODULE(..., py::mod_gil_not_used())"]
    C --> E{"GIL re-enabled?"}
    D --> E
    E -- "Before this PR\n(plugin missing declaration)" --> F["⚠️ GIL RE-ENABLED\nRuntimeWarning emitted"]
    E -- "After this PR\n(both declare GIL-not-used)" --> G["✅ GIL stays DISABLED"]
    G --> H["test_gil_disabled()\nassert not sys._is_gil_enabled()"]
Loading

Reviews (2): Last reviewed commit: "Add a check making sure that DALI doesn'..." | Re-trigger Greptile

@JanuszL JanuszL self-assigned this Apr 13, 2026
@mzient mzient self-assigned this Apr 13, 2026
@rostan-t rostan-t requested a review from JanuszL April 13, 2026 12:28
@rostan-t
Copy link
Copy Markdown
Collaborator Author

!build

@rostan-t rostan-t requested a review from mzient April 13, 2026 12:32
@dali-automaton
Copy link
Copy Markdown
Collaborator

CI MESSAGE: [48400360]: BUILD STARTED

@dali-automaton
Copy link
Copy Markdown
Collaborator

CI MESSAGE: [48400360]: BUILD PASSED

@rostan-t rostan-t merged commit 294e5ea into NVIDIA:main Apr 13, 2026
7 checks passed
@rostan-t rostan-t deleted the python_function_free_threaded_support branch April 13, 2026 16:58
stiepan pushed a commit that referenced this pull request Apr 16, 2026
…6289)

* Declare free-threaded Python support on the python_function plugin
* Stop setting PYTHON_GIL=0 to run the tests
* Add dynamic mode multithreading tests to the free-threading test suite
* Add a test making sure that DALI doesn't force-enable the GIL
---------

Signed-off-by: Rostan Tabet <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants