Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Add path used by pip's build isolation procedure to DLL search #150013

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

Add path used by pip's build isolation procedure to DLL search #150013

wants to merge 1 commit into from

Conversation

Luthaf
Copy link
Contributor

@Luthaf Luthaf commented Mar 26, 2025 •
edited
Loading

Re-opening a new PR after #131340 and #140535 where closed for being stale without a review.

Without this, trying to import torch in a downstream setup.py file would result in

The specified module could not be found. Error loading "C:\...\pip-build-env-himl3xh3\normal\Lib\site-packages\torch\lib\shm.dll" or one of its dependencies."

This seems to be because pip does not use a full virtualenv for build isolation, instead creating directories and manually adding them to sys.path. The same issue does not seem to apply when using python -m build.

To reproduce, you can create a directory with two files:

# pyproject.toml
[project]
name = "windows-torch-mkl-pip"
version = "0.0.0"

[build-system]
requires = [
    "setuptools",
    "torch"
]
# setup.py
from setuptools import setup

import torch


setup()

Then, trying to build a wheel with pip install . will give some output similar to:

Installing collected packages: tbb, mpmath, intel-openmp, typing-extensions, sympy, numpy, networkx, mkl, MarkupSafe, fsspec, filelock, jinja2, torch
      Creating C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-himl3xh3\normal\Scripts
    Successfully installed MarkupSafe-2.1.5 filelock-3.14.0 fsspec-2024.6.0 intel-openmp-2021.4.0 jinja2-3.1.4 mkl-2021.4.0 mpmath-1.3.0 networkx-3.3 numpy-1.26.4 sympy-1.12.1 tbb-2021.12.0 torch-2.3.1+cpu typing-extensions-4.12.2
    Created temporary directory: C:\Users\runneradmin\AppData\Local\Temp\pip-modern-metadata-ascqww5w
    Preparing metadata (pyproject.toml): started
    Running command Preparing metadata (pyproject.toml)
    Traceback (most recent call last):
      File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-7yztij8w\cp312-win_amd64\build\venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
        main()
      File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-7yztij8w\cp312-win_amd64\build\venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
        json_out['return_val'] = hook(**hook_input['kwargs'])
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "C:\Users\runneradmin\AppData\Local\Temp\cibw-run-7yztij8w\cp312-win_amd64\build\venv\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 149, in prepare_metadata_for_build_wheel
        return hook(metadata_directory, config_settings)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-himl3xh3\overlay\Lib\site-packages\setuptools\build_meta.py", line 366, in prepare_metadata_for_build_wheel
        self.run_setup()
      File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-himl3xh3\overlay\Lib\site-packages\setuptools\build_meta.py", line 311, in run_setup
        exec(code, locals())
      File "<string>", line 295, in <module>
      File "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-himl3xh3\normal\Lib\site-packages\torch\__init__.py", line 143, in <module>
        raise err
    OSError: [WinError 126] The specified module could not be found. Error loading "C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-himl3xh3\normal\Lib\site-packages\torch\lib\shm.dll" or one of its dependencies.
    error: subprocess-exited-with-error
    
    Preparing metadata (pyproject.toml) did not run successfully.
    exit code: 1
    
    See above for output.

Torch is properly installed in C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-himl3xh3\normal\Lib\site-packages\torch\ and all the mkl libraries are in C:\Users\runneradmin\AppData\Local\Temp\pip-build-env-himl3xh3\normal\Library\bin, but this directory is not covered by existing DLL paths.

This is similar to #125109, and the fix is similar to #125684. Ping @atalman and @malfet since you fixed & reviewed the previous similar fix.

@pytorch-bot PyTorch Bot
Copy link

pytorch-bot bot commented Mar 26, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/150013

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 00a6eff with merge base 0d17029 (image):

NEW FAILURE - The following job has failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@github-actions GitHub Actions
Copy link
Contributor

This PR needs a release notes: label

If your changes are user facing and intended to be a part of release notes, please use a label starting with release notes:.

If not, please add the topic: not user facing label.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "topic: not user facing"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@soulitzer soulitzer requested a review from malfet March 26, 2025 23:05
@soulitzer soulitzer added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Mar 26, 2025
@soulitzer soulitzer requested a review from a team March 26, 2025 23:05
Skylion007
Skylion007 reviewed May 12, 2025
View reviewed changes
torch/__init__.py Outdated
Copy link
Collaborator

@Skylion007 Skylion007 May 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use shutil.which to find the path of MKL somehow?

Copy link
Contributor Author

@Luthaf Luthaf May 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think so, since the issue is that MKL is NOT in the PATH to begin with.

Looking at this code again, I think I made a copy/paste mistake though, this should read os.path.dirname(wheel.__file__), "..", "..": MKL gets installed in <tmp-pip-prefix>/Library/bin, and the only way I found to get this <tmp-pip-prefix> was to look where other build-time dependency such as wheel are installed.

EDIT: no, this is fine, I got confused: I need to workaround this at build-time for my own packages (which import torch in the setup.py, but here we are inside torch already … 😅

@Luthaf Luthaf force-pushed the load-mkl-pip-build-isolation branch from 7bbd017 to dc0ab5b Compare May 12, 2025 13:31
@github-actions GitHub Actions
Copy link
Contributor

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

@github-actions github-actions bot added the Stale label Jul 11, 2025
@Luthaf
Add path used by pip's build isolation procedure to DLL search
00a6eff 
Without this, trying to `import torch` in a downstream `setup.py` file
would result in

```
The specified module could not be found. Error loading "C:\...\pip-build-env-himl3xh3\normal\Lib\site-packages\torch\lib\shm.dll" or one of its dependencies."
```

This seems to be because pip does not use a full virtualenv for build
isolation, instead creating directories and manually adding them to
`sys.path`
@Luthaf Luthaf force-pushed the load-mkl-pip-build-isolation branch from dc0ab5b to 00a6eff Compare July 11, 2025 14:39
@Luthaf
Copy link
Contributor Author

Luthaf commented Jul 11, 2025

@malfet @Skylion007 could this get a review?

@Luthaf Luthaf requested a review from Skylion007 July 11, 2025 14:40
@github-actions github-actions bot closed this Aug 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@malfet malfet Awaiting requested review from malfet

@Skylion007 Skylion007 Awaiting requested review from Skylion007

Assignees

No one assigned

Labels

open source Stale triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Luthaf @Skylion007 @soulitzer @pytorchbot