Initial version of `cuda.pathfinder._find_nvidia_headers` for `nvshmem` #661

rwgk · 2025-05-28T17:43:59Z

Description

Bump cuda-pathfinder version to TODO

Minimal viable product, currently not a public API (by way of using an underscore prefix: _find_nvidia_headers).

To support nvshmem4py.

Commit 673c38c was transferred from PR #864.

copy-pr-bot · 2025-05-28T17:44:02Z

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

rwgk · 2025-05-28T17:45:11Z

/ok to test

ZzEeKkAa · 2025-07-18T01:35:03Z

Just wondering what is the status for this PR?

rwgk · 2025-07-18T16:53:39Z

I still need to fix this up, after merging #723 a couple days ago.

isVoid · 2025-07-28T23:29:56Z

Is there some mechanism so that user can override the path returned by the api? Or that's largely a library duty to maintain?

…modernize new code

…more complete commands for setting up manual testing.

rwgk · 2025-08-12T06:23:19Z

/ok to test

ZzEeKkAa · 2025-08-12T14:59:30Z

Will it tolerate CUDA_HOME and CUDA_PATH ?

rwgk · 2025-08-12T16:32:11Z

Will it tolerate CUDA_HOME and CUDA_PATH ?

The code added in this PR ignores those completely. I've only tested these situations (copy-pasted from the current cuda_pathfinder/tests/test_find_nvidia_headers.py):

    # pip install nvidia-nvshmem-cu12
    # pip install nvidia-nvshmem-cu13

    # conda create -y -n nvshmem python=3.12
    # conda activate nvshmem
    # conda install -y conda-forge::libnvshmem3 conda-forge::libnvshmem-dev

    # wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
    # sudo dpkg -i cuda-keyring_1.1-1_all.deb
    # sudo apt update
    # sudo apt install libnvshmem3-cuda-12 libnvshmem3-dev-cuda-12
    # sudo apt install libnvshmem3-cuda-13 libnvshmem3-dev-cuda-13

Offline @leofang wrote that getting headers in general is now the top priority (I just created #832 to track the work). That will bring in CUDA_HOME / CUDA_PATH for headers. The next step will be to get an overview of where the headers live.

Coming back to nvshmem: Can those headers also be installed into CUDA_HOME / CUDA_PATH? (How is that usually done?)

…headers.py

…ind_nvidia_headers.py

rwgk · 2025-09-04T00:35:32Z

/ok to test

leofang

I have not reviewed tests, just the implementation.

cuda_pathfinder/cuda/pathfinder/_headers/find_nvidia_headers.py

cuda_pathfinder/cuda/pathfinder/_utils/conda_env.py

cuda_pathfinder/cuda/pathfinder/_headers/find_nvidia_headers.py

cuda_pathfinder/cuda/pathfinder/_utils/env_vars_for_include.py

…e from find_nvidia_headers.py" This reverts commit c90c393.

…_nvidia_headers.py" This reverts commit eb2e78a.

cuda_pathfinder/cuda/pathfinder/_headers/find_nvidia_headers.py

kkraus14 · 2025-09-05T17:43:01Z

cuda_pathfinder/cuda/pathfinder/_headers/find_nvidia_headers.py

+    # Installed from a wheel
+    nvidia_sub_dirs = ("nvidia", "nvshmem", "include")
+    hdr_dir: str  # help mypy
+    for hdr_dir in find_sub_dirs_all_sitepackages(nvidia_sub_dirs):


We should do it in a follow up, but I think we should really move towards using an importlib based resolution method instead of walking the sitepackages ourselves.

I thought we discussed and agreed walking the paths is acceptable since on the PyPI side they are predictable? Any reason to prefer importlib?

That's a good point, tx, I created issue #949 to track this.

(I wrote this code before @ZzEeKkAa pointed me to importlib while working on PR #864)

I thought we discussed and agreed walking the paths is acceptable since on the PyPI side they are predictable? Any reason to prefer importlib?

Lines crossed here. Let's review under #949 when we get to it.

Oops, sorry, copy-paste mishap. Corrected (949).

cuda_pathfinder/cuda/pathfinder/_headers/find_nvidia_headers.py

kkraus14 · 2025-09-05T17:52:32Z

cuda_pathfinder/tests/test_find_nvidia_headers.py

+                    "site-packages",  # pip install
+                    "dist-packages",  # apt install


Do we need to do anything for conda packages here?

This is gated by have_nvidia_nvshmem_package(). ... But I just realized, there is no nvshmem .deb that installs into dist-packages. So I removed dist-packages from here to not implicitly suggest that it exists: commit 2426260

While at it, I added additional test (low-hanging fruits): commit b74a84c

Interactive testing with conda and "all_must_work" set:

$ pytest -ra -s -vv tests/test_find_nvidia_headers.py

========================================================================= test session starts ========================================================================== platform linux -- Python 3.12.11, pytest-8.4.1, pluggy-1.6.0 -- /home/rgrossekunst/miniforge3/envs/nvshmem/bin/python3.12 cachedir: .pytest_cache rootdir: /home/rgrossekunst/forked/cuda-python/cuda_pathfinder configfile: pyproject.toml collected 2 items tests/test_find_nvidia_headers.py::test_unknown_libname PASSED tests/test_find_nvidia_headers.py::test_find_libname_nvshmem PASSED ============================================================================= INFO summary ============================================================================= INFO test_find_libname_nvshmem: hdr_dir='/home/rgrossekunst/miniforge3/envs/nvshmem/include' ========================================================================== 2 passed in 0.01s ===========================================================================

Without wheel or conda ("all_must_work" set):

$ pytest -ra -s -vv tests/test_find_nvidia_headers.py

========================================================================= test session starts ========================================================================== platform linux -- Python 3.12.3, pytest-8.4.2, pluggy-1.6.0 -- /home/rgrossekunst/JunkVenv/bin/python3 cachedir: .pytest_cache rootdir: /home/rgrossekunst/forked/cuda-python/cuda_pathfinder configfile: pyproject.toml collected 2 items tests/test_find_nvidia_headers.py::test_unknown_libname PASSED tests/test_find_nvidia_headers.py::test_find_libname_nvshmem PASSED ============================================================================= INFO summary ============================================================================= INFO test_find_libname_nvshmem: hdr_dir='/usr/include/nvshmem_13' ========================================================================== 2 passed in 0.02s ===========================================================================

I also did negative tests for both, by using bad expected paths, and they fail as expected.

… a .deb that installs into dist-packages does not exist.

rwgk · 2025-09-05T18:46:42Z

/ok to test

github-actions · 2025-09-05T19:36:25Z

Doc Preview CI
Preview removed because the pull request was closed or merged.

rwgk · 2025-09-05T19:56:06Z

For easy pre-release testing, I uploaded this to TestPyPI:

https://test.pypi.org/project/cuda-pathfinder/1.2.2.dev202509051239/

It's exactly the same as under this PR, except that I appended .dev202509051239 to the version number (version numbers have to be unique for PyPI, even TestPyPI, and cannot be overwritten).

$ git diff
diff --git a/cuda_pathfinder/cuda/pathfinder/_version.py b/cuda_pathfinder/cuda/pathfinder/_version.py
index 70aa6255c..6a6745b50 100644
--- a/cuda_pathfinder/cuda/pathfinder/_version.py
+++ b/cuda_pathfinder/cuda/pathfinder/_version.py
@@ -1,4 +1,4 @@
 # SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 # SPDX-License-Identifier: Apache-2.0

-__version__ = "1.2.2"
+__version__ = "1.2.2.dev202509051239"

$ rm -rf dist/
$ python -m build
$ twine check --strict dist/cuda_pathfinder-*-py3-none-any.whl
Checking dist/cuda_pathfinder-1.2.2.dev202509051239-py3-none-any.whl: PASSED
$ twine upload --verbose -r testpypi dist/*
View at:
https://test.pypi.org/project/cuda-pathfinder/1.2.2.dev202509051239/

…m` (NVIDIA#661) * find_nvidia_headers.py initial version (untested). * Add tests/test_path_finder_find_headers.py, with hard-coded paths. * Better error message: UNKNOWN libname='unknown-libname' * if libname == "nvshmem" and IS_WINDOWS: return None * Move find_nvidia_headers.py → _headers/find_nvidia_headers.py * test_find_nvidia_headers.py: removed hard-wired paths, comments with more complete commands for setting up manual testing. * Make _find_nvidia_header_directory private for now. * test_find_nvidia_headers.py: Move comments with installation commands up. * Add `have_nvidia_nvshmem_package()` function to enable `assert hdr_dir is not None` * Add nvidia-nvshmem-cu12,13 in pyproject.toml * assert site-packages or dist-packages * Add CUDA_PATHFINDER_TEST_FIND_NVIDIA_HEADERS_STRICTNESS * Transfer `ci/`, `.github/` changes from PR NVIDIA#864 * Add CUDA_PATHFINDER_TEST_FIND_NVIDIA_HEADERS_STRICTNESS in `ci/`, `.github/` * reverse=True in sorting of "/usr/include/nvshmem_*" (find newest first) * Fix: assert site-packages or dist-packages only if have_nvidia_nvshmem_package() * pytest.skip("nvshmem has no Windows support.") * Add new cuda/pathfinder/_utils/conda_env.py and use from find_nvidia_headers.py * Add new cuda/pathfinder/_utils/env_vars_for_include.py and use from find_nvidia_headers.py * Revert "Add new cuda/pathfinder/_utils/env_vars_for_include.py and use from find_nvidia_headers.py" This reverts commit c90c393. * Revert "Add new cuda/pathfinder/_utils/conda_env.py and use from find_nvidia_headers.py" This reverts commit eb2e78a. * Bump pathfinder version to 1.2.2 and add release/1.2.2-notes.rst * Remove os.path.isdir() tests that are not strictly needed. * test_find_nvidia_headers.py: remove check for `dist-packages` because a .deb that installs into dist-packages does not exist. * Additional testing

rwgk added 4 commits May 27, 2025 16:41

find_nvidia_headers.py initial version (untested).

d174f5a

Add tests/test_path_finder_find_headers.py, with hard-coded paths.

5ed86c5

Better error message: UNKNOWN libname='unknown-libname'

a98de70

if libname == "nvshmem" and IS_WINDOWS: return None

6a93fe7

rwgk self-assigned this May 28, 2025

This comment has been minimized.

Sign in to view

leofang self-requested a review May 28, 2025 22:26

leofang added P0 High priority - Must do! feature New feature or request cuda.bindings Everything related to the cuda.bindings module labels May 28, 2025

leofang added this to the cuda-pathfinder first release milestone Jul 2, 2025

github-project-automation bot added this to CCCL Jul 2, 2025

github-project-automation bot moved this to Todo in CCCL Jul 2, 2025

leofang added cuda.pathfinder Everything related to the cuda.pathfinder module and removed cuda.bindings Everything related to the cuda.bindings module labels Jul 18, 2025

leofang modified the milestones: cuda-pathfinder first release, cuda-pathfinder parking lot Jul 21, 2025

isVoid mentioned this pull request Aug 8, 2025

[FEA]: Support for finding nvshmem host library via cuda-pathfinder #823

Open

1 task

rwgk added 3 commits August 11, 2025 22:02

Merge branch 'main' into find_nvidia_headers_nvshmem and move/adjust/…

3ab9643

…modernize new code

Move find_nvidia_headers.py → _headers/find_nvidia_headers.py

c1e9385

test_find_nvidia_headers.py: removed hard-wired paths, comments with …

c3464b3

…more complete commands for setting up manual testing.

rwgk changed the title ~~Initial version of path_finder find_nvidia_headers.py (Minimal Viable Product)~~ [cuda_pathfinder] Initial version of find_nvidia_headers.py for nvshmem (Minimal Viable Product) Aug 12, 2025

rwgk added 2 commits September 3, 2025 12:02

Merge branch 'main' into find_nvidia_headers_nvshmem

dc4de43

Add new cuda/pathfinder/_utils/conda_env.py and use from find_nvidia_…

eb2e78a

…headers.py

rwgk force-pushed the find_nvidia_headers_nvshmem branch from b1ceca6 to eb2e78a Compare September 3, 2025 23:31

Add new cuda/pathfinder/_utils/env_vars_for_include.py and use from f…

c90c393

…ind_nvidia_headers.py

rwgk force-pushed the find_nvidia_headers_nvshmem branch from c822c15 to c90c393 Compare September 4, 2025 00:30

leofang reviewed Sep 5, 2025

View reviewed changes

rwgk added 4 commits September 5, 2025 10:05

Revert "Add new cuda/pathfinder/_utils/env_vars_for_include.py and us…

cee717a

…e from find_nvidia_headers.py" This reverts commit c90c393.

Revert "Add new cuda/pathfinder/_utils/conda_env.py and use from find…

a50da30

…_nvidia_headers.py" This reverts commit eb2e78a.

Merge branch 'main' into find_nvidia_headers_nvshmem

a185f03

Bump pathfinder version to 1.2.2 and add release/1.2.2-notes.rst

7cfcbfe

leofang previously approved these changes Sep 5, 2025

View reviewed changes

cuda_pathfinder/cuda/pathfinder/_headers/find_nvidia_headers.py Show resolved Hide resolved

github-project-automation bot moved this from Todo to In Review in CCCL Sep 5, 2025

Remove os.path.isdir() tests that are not strictly needed.

510f470

rwgk dismissed leofang’s stale review via 510f470 September 5, 2025 17:44

kkraus14 reviewed Sep 5, 2025

View reviewed changes

This was referenced Sep 5, 2025

Prefer importlib over find_sub_dirs_all_sitepackages #949

Open

[FEA]: Enable getting headers from CTK or other 1st party libs #832

Closed

rwgk added 3 commits September 5, 2025 11:28

test_find_nvidia_headers.py: remove check for dist-packages because…

2426260

… a .deb that installs into dist-packages does not exist.

Additional testing

b74a84c

Merge branch 'main' into find_nvidia_headers_nvshmem

6ee6529

leofang approved these changes Sep 5, 2025

View reviewed changes

rwgk merged commit d64120b into NVIDIA:main Sep 5, 2025
49 checks passed

github-project-automation bot moved this from In Review to Done in CCCL Sep 5, 2025

rwgk deleted the find_nvidia_headers_nvshmem branch September 5, 2025 19:21

		"site-packages", # pip install
		"dist-packages", # apt install

Initial version of cuda.pathfinder._find_nvidia_headers for nvshmem #661

Initial version of cuda.pathfinder._find_nvidia_headers for nvshmem #661

Uh oh!

Conversation

rwgk commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

copy-pr-bot bot commented May 28, 2025

Uh oh!

rwgk commented May 28, 2025

Uh oh!

This comment has been minimized.

ZzEeKkAa commented Jul 18, 2025

Uh oh!

rwgk commented Jul 18, 2025

Uh oh!

isVoid commented Jul 28, 2025

Uh oh!

rwgk commented Aug 12, 2025

Uh oh!

ZzEeKkAa commented Aug 12, 2025

Uh oh!

rwgk commented Aug 12, 2025

Uh oh!

rwgk commented Sep 4, 2025

Uh oh!

leofang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kkraus14 Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

leofang Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

rwgk Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

rwgk Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leofang Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

rwgk Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kkraus14 Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

rwgk Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rwgk commented Sep 5, 2025

Uh oh!

Uh oh!

github-actions bot commented Sep 5, 2025

Uh oh!

rwgk commented Sep 5, 2025

Uh oh!

Uh oh!

Initial version of `cuda.pathfinder._find_nvidia_headers` for `nvshmem` #661

Initial version of `cuda.pathfinder._find_nvidia_headers` for `nvshmem` #661

rwgk commented May 28, 2025 •

edited

Loading

rwgk Sep 5, 2025 •

edited

Loading

rwgk Sep 5, 2025 •

edited

Loading