BUG: f2py compile error with full path of signature file on Windows #22211

jmrohwer · 2022-09-06T09:49:36Z

Describe the issue:

I am trying to port the build system for a python package (https://github.com/PySCeS/pysces) from numpy.distutils to use scikit-build because of the deprecation of distutils (the package contains 2 Fortran extension modules wrapped with f2py). To get to know the system I’ve started with a minimal example. I have been getting compile failures on Windows (on Linux and macOS everything works).

I have posted this on SO and on the Fortran discourse group but have not had any replies despite quite a number of reads. Links for cross-reference:

During compilation of the module with python setup.py build the build fails. In tracking it down I saw that cmake expands the full path of the source files and this is where it fails. To ensure that it is not a problem with the skbuild build system I verified that the failure can be reproduced on the command line using f2py on its own as follows - save the Fortran files in the code block below to the same folder and then run in a command prompt:

(skbuild) z:\Documents\Python\skbuild_test\simple\hello\shout>f2py.exe -m hi -c hi.pyf hi.f

This works as expected and generates the .pyd and .dll files. However, when specifying the full path as follows (and as it is called in skbuild), it fails:

(skbuild) z:\Documents\Python\skbuild_test\simple\hello\shout>f2py.exe -m hi -c z:/Documents/Python/skbuild_test/simple/hello/shout/hi.pyf z:/Documents/Python/skbuild_test/simple/hello/shout/hi.f

The traceback produced is appended below. The problem seems to be with the creating C:Users part leading to the malformed filename in the last line. Specifying only the filename for hi.pyf but the full path for hi.f does work strangely enough, but as soon as the full path for hi.pyf is specified it fails. As mentioned, on Linux and macOS all works fine.

Note: I could of course skip the .pyf file and then it works. However, the Fortran code I’m wrapping is F77 and needs to be redistributed in unmodified form due to licensing conditions, so I do need a custom .pyf file to integrate it into my Python module.

Reproduce the code example:

**hi.f**

      SUBROUTINE HELLO()
      PRINT*,"Hello from fortran"
      END

**hi.pyf**

!    -*- f90 -*-
! Note: the context of this file is case sensitive.

python module hi ! in 
    interface  ! in :hi
        subroutine hello ! in :hi:hi.f
        end subroutine hello
    end interface 
end python module hi

! This file was auto-generated with f2py (version:1.23.2).
! See:
! https://web.archive.org/web/20140822061353/http://cens.ioc.ee/projects/f2py2e

Error message:

running build
running config_cc
INFO: unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
INFO: unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
INFO: build_src
INFO: building extension "hi" sources
creating C:
creating C:Users
creating C:Users\jr
creating C:Users\jr\AppData
creating C:Users\jr\AppData\Local
creating C:Users\jr\AppData\Local\Temp
creating C:Users\jr\AppData\Local\Temp\tmpdbudw6jh
creating C:Users\jr\AppData\Local\Temp\tmpdbudw6jh\src.win-amd64-3.10
creating C:Users\jr\AppData\Local\Temp\tmpdbudw6jh\src.win-amd64-3.10\Documents
creating C:Users\jr\AppData\Local\Temp\tmpdbudw6jh\src.win-amd64-3.10\Documents\Python
creating C:Users\jr\AppData\Local\Temp\tmpdbudw6jh\src.win-amd64-3.10\Documents\Python\skbuild_test
creating C:Users\jr\AppData\Local\Temp\tmpdbudw6jh\src.win-amd64-3.10\Documents\Python\skbuild_test\simple
creating C:Users\jr\AppData\Local\Temp\tmpdbudw6jh\src.win-amd64-3.10\Documents\Python\skbuild_test\simple\hello
creating C:Users\jr\AppData\Local\Temp\tmpdbudw6jh\src.win-amd64-3.10\Documents\Python\skbuild_test\simple\hello\shout
INFO: f2py options: []
INFO: f2py: Z:/Documents/Python/skbuild_test/simple/hello/shout/hi.pyf
error: [WinError 123] The filename, directory name, or volume label syntax is incorrect: 'C:C:\\Users\\jr\\AppData\\Local\\Temp\\tmpdbudw6jh\\src.win-amd64-3.10\\Documents\\Python\\skbuild_test\\simple\\hello\\shout'

NumPy/Python version information:

Windows 10 64-bit
MSVC 2022 build tools
mingw 11.2.0.07112021
python 3.10.6
numpy 1.23.2

Context for the issue:

Any project utilizing f2py to wrap Fortran code, and which includes a custom .pyf file, cannot be built with scikit-build which becomes problematic once distutils and numpy.distutils are no longer available.

The text was updated successfully, but these errors were encountered:

jmrohwer · 2022-09-06T10:17:16Z

I've tracked down the origin of this to the expansion of tempfile.mkdtemp() here:

numpy/numpy/f2py/f2py2e.py

Lines 514 to 525 in 5bc5976

    
           remove_build_dir = 0 
        
           try: 
        
               i = sys.argv.index('--build-dir') 
        
           except ValueError: 
        
               i = None 
        
           if i is not None: 
        
               build_dir = sys.argv[i + 1] 
        
               del sys.argv[i + 1] 
        
               del sys.argv[i] 
        
           else: 
        
               remove_build_dir = 1 
        
               build_dir = tempfile.mkdtemp()

If I manually specify a --build-dir on the command line, or in the CMakeLists.txt call to f2py in the project build, then it works. This is how I'm currently working around the issue for PySCeS, e.g. here:

https://github.com/PySCeS/pysces/blob/7aedc3e1f0fa060fb414052474457f09d9cc30e9/pysces/nleq2/CMakeLists.txt#L21-L28

jmrohwer · 2022-09-06T10:22:10Z

Incidentially, I see that the f2py compile process itself makes use of numpy.distutils extensively, e.g.

numpy/numpy/f2py/f2py2e.py

Line 607 in 5bc5976

from numpy.distutils.command.build_src import get_f2py_modulename

numpy/numpy/f2py/f2py2e.py

Line 636 in 5bc5976

from numpy.distutils.core import setup, Extension

What is the numpy strategy going forward with this, with the deprecation of distutils and numpy.distutils?

rgommers · 2022-09-07T18:27:25Z

Incidentially, I see that the f2py compile process itself makes use of numpy.distutils extensively, e.g.

This is only for f2py -c, and that is being ported to Meson as part of @NamamiShanker's GSoC project: https://blogs.python-gsoc.org/en/namamishankers-blog/

What is the numpy strategy going forward with this, with the deprecation of distutils and numpy.distutils?

Note that the core functionality of f2py (generating bindings) is independent of build system, and there are docs for multiple build systems (including scikit-build) at https://numpy.org/devdocs/f2py/buildtools/index.html.

rgommers · 2023-09-05T18:53:54Z

@jmrohwer I just saw the comment on gh-22225, and that reminded me of this issue. Is it still relevant?

I'll note that you should not be using f2py -c at all if you're using CMake (scikit-build / scikit-build-core) or Meson/meson-python. You want to use f2py as a transpiler only, and integrate the resulting .c/.f/.h files into your regular CMake or Meson targets.

jmrohwer · 2023-09-18T20:04:16Z

@jmrohwer I just saw the comment on gh-22225, and that reminded me of this issue. Is it still relevant?

@rgommers I am happy to close this. Apologies for the delayed reply, I was travelling.

I'll note that you should not be using f2py -c at all if you're using CMake (scikit-build / scikit-build-core) or Meson/meson-python. You want to use f2py as a transpiler only, and integrate the resulting .c/.f/.h files into your regular CMake or Meson targets.

Is there any good documentation or examples on this I could use as a reference or starting point? I am not an expert in these build systems and the project (PySCeS) started out using f2py -c with a custom .pyf file using numpy.distutils. I did get this to work with scikit-build, still calling f2py-c but am happy to give it a stab to do it "properly" if I have a pointer where to start. Note that we use a .pyf file to only expose a small subset of the Fortran methods in Python.

HaoZeke · 2023-11-22T19:07:22Z

@jmrohwer I just saw the comment on gh-22225, and that reminded me of this issue. Is it still relevant?

@rgommers I am happy to close this. Apologies for the delayed reply, I was travelling.

I'll note that you should not be using f2py -c at all if you're using CMake (scikit-build / scikit-build-core) or Meson/meson-python. You want to use f2py as a transpiler only, and integrate the resulting .c/.f/.h files into your regular CMake or Meson targets.

Is there any good documentation or examples on this I could use as a reference or starting point? I am not an expert in these build systems and the project (PySCeS) started out using f2py -c with a custom .pyf file using numpy.distutils. I did get this to work with scikit-build, still calling f2py-c but am happy to give it a stab to do it "properly" if I have a pointer where to start. Note that we use a .pyf file to only expose a small subset of the Fortran methods in Python.

Tracking in #25199, so I'll close this here 😄 .

jmrohwer added the 00 - Bug label Sep 6, 2022

rgommers added the component: numpy.f2py label Sep 7, 2022

jmrohwer mentioned this issue Sep 19, 2022

ENH: Meson build system support for F2PY's f2py -c command #22225

Closed

jmrohwer mentioned this issue Oct 25, 2023

BUG: f2py -c does not work on Windows with meson (Python 3.12) #25000

Closed

HaoZeke mentioned this issue Nov 22, 2023

DOC: Explain distribution for f2py with meson-python #25199

Open

HaoZeke closed this as completed Nov 22, 2023

jmrohwer mentioned this issue Dec 15, 2023

Implementing the meson-python build backend PySCeS/pysces#94

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

BUG: f2py compile error with full path of signature file on Windows #22211

BUG: f2py compile error with full path of signature file on Windows #22211

jmrohwer commented Sep 6, 2022

jmrohwer commented Sep 6, 2022

Uh oh!

jmrohwer commented Sep 6, 2022

Uh oh!

rgommers commented Sep 7, 2022

Uh oh!

rgommers commented Sep 5, 2023

Uh oh!

jmrohwer commented Sep 18, 2023

Uh oh!

HaoZeke commented Nov 22, 2023

Uh oh!

Uh oh!

BUG: f2py compile error with full path of signature file on Windows #22211

BUG: f2py compile error with full path of signature file on Windows #22211

Comments

jmrohwer commented Sep 6, 2022

Describe the issue:

Reproduce the code example:

Error message:

NumPy/Python version information:

Context for the issue:

jmrohwer commented Sep 6, 2022

Uh oh!

jmrohwer commented Sep 6, 2022

Uh oh!

rgommers commented Sep 7, 2022

Uh oh!

rgommers commented Sep 5, 2023

Uh oh!

jmrohwer commented Sep 18, 2023

Uh oh!

HaoZeke commented Nov 22, 2023

Uh oh!