Thanks to visit codestin.com
Credit goes to github.com

Skip to content

BUG: ifort executable path being split at whitespaces and then used incorrectly even after already being parsed and used correctly #20316

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
crswong888 opened this issue Nov 6, 2021 · 3 comments
Closed

BUG: ifort executable path being split at whitespaces and then used incorrectly even after already being parsed and used correctly #20316

crswong888 opened this issue Nov 6, 2021 · 3 comments
Labels
00 - Bug component: numpy.distutils component: numpy.f2py

Comments

@crswong888
Copy link

crswong888 commented Nov 6, 2021 •
edited
Loading

Describe the issue:

Hello, this is my first numpy ticket, so hopefully, I can follow the basic guidelines. I searched through the existing issues, and while there are others similar, I think mine has not yet been raised.

I'm trying to wrap some Fortran using F2PY via numpy.distutils, and the generated package and all of its modules are working great. Still, I'm making every effort to reduce the number of tasks invoked by the build_ext command and clean up the console output. But something is incorrectly appending executables and source files that I did not specify. I believe it is clear that the problem is that somewhere the code is splitting the executable path at whitespaces, even after it correctly interprets the same path beforehand.

I'm on Windows 10 and have the ifort compiler provided by Intel Parallel Studio XE 2019 installed at C:\Program Files (x86)\IntelSWTools. There are a few lines in the output from my setup.py script that indicate the problem:

running build_ext
running build_src
build_src
building extension "..py_alchemyCore" sources
f2py options: []
  adding 'build\src.win-amd64-3.9\build\src.win-amd64-3.9\fortranobject.c' to sources.
  adding 'build\src.win-amd64-3.9\build\src.win-amd64-3.9\' to include_dirs.
  adding 'build\src.win-amd64-3.9\py_alchemyCore-f2pywrappers2.f90' to sources.
build_src: building npy-pkg config files
No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils
customize MSVCCompiler
customize MSVCCompiler using build_ext
customize IntelEM64VisualFCompiler
Found executable C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64\ifort.exe
Could not locate executable C:\Program
customize IntelEM64VisualFCompiler using build_ext
building '..py_alchemyCore' extension
compiling C sources
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DF2PY_REPORT_ON_ARRAY_COPY=1 -Ibuild\src.win-amd64-3.9\build\src.win-amd64-3.9\ -IC:\Users\cwong\.venv\alchemy-env\lib\site-packages\numpy\core\include -IC:\Users\cwong\.venv\alchemy-env\include -IC:\Python39\include -IC:\Python39\include -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.16.27023\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.16.27023\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt /Tcbuild\src.win-amd64-3.9\py_alchemyCoremodule.c /Fobuild\temp.win-amd64-3.9\Release\build\src.win-amd64-3.9\py_alchemyCoremodule.obj
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DF2PY_REPORT_ON_ARRAY_COPY=1 -Ibuild\src.win-amd64-3.9\build\src.win-amd64-3.9\ -IC:\Users\cwong\.venv\alchemy-env\lib\site-packages\numpy\core\include -IC:\Users\cwong\.venv\alchemy-env\include -IC:\Python39\include -IC:\Python39\include -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.16.27023\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.16.27023\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt /Tcbuild\src.win-amd64-3.9\build\src.win-amd64-3.9\fortranobject.c /Fobuild\temp.win-amd64-3.9\Release\build\src.win-amd64-3.9\build\src.win-amd64-3.9\fortranobject.obj
compiling Fortran sources
Fortran f77 compiler: C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64\ifort.exe Files (x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64\ifort.exe /nologo /MD /nbs /names:lowercase /assume:underscore /recursive /O1 /QaxSSE3
Fortran f90 compiler: C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64\ifort.exe Files (x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64\ifort.exe /nologo /MD /nbs /names:lowercase /assume:underscore /recursive /O1 /QaxSSE3
Fortran fix compiler: C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64\ifort.exe Files (x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64\ifort.exe /nologo /MD /nbs /names:lowercase /assume:underscore /recursive /O1 /QaxSSE3
compile options: '-DF2PY_REPORT_ON_ARRAY_COPY=1 -Ibuild\src.win-amd64-3.9\build\src.win-amd64-3.9\ -IC:\Users\cwong\.venv\alchemy-env\lib\site-packages\numpy\core\include -IC:\Users\cwong\.venv\alchemy-env\include -IC:\Python39\include -IC:\Python39\include -c'
extra f90 options: '/free'
ifort.exe:f90: ..\current\alchemy_core.for
ifort: command line warning #10161: unrecognized source type 'Files'; object file assumed
ifort: command line warning #10161: unrecognized source type '(x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64\ifort.exe'; object file assumed
ifort: command line warning #10006: ignoring unknown option '/c '
ifort: warning #10145: no action performed for file 'Files'
ifort: warning #10145: no action performed for file '(x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64\ifort.exe'
extra f90 options: '/free'
ifort.exe:f90: dummy_code.f90
ifort: command line warning #10161: unrecognized source type 'Files'; object file assumed
ifort: command line warning #10161: unrecognized source type '(x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64\ifort.exe'; object file assumed
ifort: command line warning #10006: ignoring unknown option '/c '
ifort: warning #10145: no action performed for file 'Files'
ifort: warning #10145: no action performed for file '(x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64\ifort.exe'
extra f90 options: '/free'
ifort.exe:f90: build\src.win-amd64-3.9\py_alchemyCore-f2pywrappers2.f90
ifort: command line warning #10161: unrecognized source type 'Files'; object file assumed
ifort: command line warning #10161: unrecognized source type '(x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64\ifort.exe'; object file assumed
ifort: command line warning #10006: ignoring unknown option '/c '
ifort: warning #10145: no action performed for file 'Files'
ifort: warning #10145: no action performed for file '(x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64\ifort.exe'
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\compiler\lib\intel64 /LIBPATH:C:\Users\cwong\.venv\alchemy-env\libs /LIBPATH:C:\Python39\libs /LIBPATH:C:\Python39 /LIBPATH:C:\Users\cwong\.venv\alchemy-env\PCbuild\amd64 /LIBPATH:C:\Users\cwong\.venv\alchemy-env\libs /LIBPATH:C:\Python39\libs /LIBPATH:C:\Python39 /LIBPATH:C:\Users\cwong\.venv\alchemy-env\PCbuild\amd64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.16.27023\ATLMFC\lib\x64 /LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.16.27023\lib\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\lib\um\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\ucrt\x64 /LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\um\x64 /EXPORT:PyInit_py_alchemyCore build\temp.win-amd64-3.9\Release\build\src.win-amd64-3.9\py_alchemyCoremodule.obj build\temp.win-amd64-3.9\Release\build\src.win-amd64-3.9\build\src.win-amd64-3.9\fortranobject.obj build\temp.win-amd64-3.9\Release\alchemy_core\current\alchemy_core.o build\temp.win-amd64-3.9\Release\dummy_code.o build\temp.win-amd64-3.9\Release\build\src.win-amd64-3.9\py_alchemyCore-f2pywrappers2.o /OUT:py_alchemyCore.cp39-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.9\Release\build\src.win-amd64-3.9\py_alchemyCore.cp39-win_amd64.lib

First, after it correctly locates my compiler executable, it mysteriously tries to find another one C:\Program:

Found executable C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64\ifort.exe
Could not locate executable C:\Program

Second, it tries to run ifort with two files named Files and (x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64\ifort.exe:

Fortran f90 compiler: C:\Program Files (x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64\ifort.exe Files (x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64\ifort.exe /nologo /MD /nbs /names:lowercase /assume:underscore /recursive /O1 /QaxSSE3

And finally, the compiler complains that it doesn't recognize these files and ignores them, thankfully. Somehow, the executable path is split like C:\Program, Files, (x86)\IntelSWTools\compilers_and_libraries\windows\bin\intel64\ifort.exe, and then passed around all willy-nilly. I'm not sure if this is a numpy issue, but it seems somewhat known. The commonly accepted solution appears to be to use a different C compiler like mingw or intelemw, albeit I prefer to stick with the default msvc one.

Reproduce the code example:

import os, sys
from numpy.distutils.core import setup
from numpy.distutils.misc_util import Configuration

def configuration(parent_package='', top_path=None):
    arch = 'intel64'
    intel_dir = os.path.join('C:', os.sep, 'Program Files (x86)', 'IntelSWTools',
                             'compilers_and_libraries', 'windows')
    os.environ['Path'] += ';' + os.path.join(intel_dir, 'bin', arch)

    config = Configuration('.', parent_package, top_path)

    config.add_extension(
        name='py_alchemyCore',
        sources=[os.path.join('..', 'current', 'alchemy_core.for'), 'dummy_code.f90'],
        library_dirs=[os.path.join(intel_dir, 'compiler', 'lib', arch)],
        define_macros=[('F2PY_REPORT_ON_ARRAY_COPY', 1)],
        extra_f90_compile_args=['/free']
    )

    return config

if __name__ == '__main__':
    default_args = ['build_ext', '--inplace', '--force', '--fcompiler=intelvem']
    setup(script_args=default_args if len(sys.argv) == 1 else None, configuration=configuration)

Error message:

No response

NumPy/Python version information:

1.19.5 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)]
@crswong888 crswong888 changed the title BUG: <Please write a comprehensive title after the 'BUG: ' prefix> BUG: ifort executable path being split at whitespaces and then used incorrectly even after already being parsed and used correctly Nov 6, 2021
@eric-wieser
Copy link
Member

I think there might be another open issue about this. Whitespace handling is a mess, especially since what users deem correct depends what operating system they're on.

@crswong888
Copy link
Author

@eric-wieser yeah, I was searching for a while and noticed several issues falling under the "whitespace handling is a mess" category - closed and open ones.

I've seen the discussions, and I can tell you guys are well aware of this general issue. I totally get it if you close this one with reference to some other. Luckily, it's not fatal in my case but does lead to some silly behavior.

@HaoZeke
Copy link
Member

HaoZeke commented Sep 5, 2023

This should be fixed by the new meson backend (or moves it out of numpy and into meson if its still present..

@HaoZeke HaoZeke closed this as completed Sep 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
00 - Bug component: numpy.distutils component: numpy.f2py
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@WarrenWeckesser @eric-wieser @HaoZeke @crswong888