BUG: f2py appears to ignore Fortran compiler flags #25784

warrickball · 2024-02-07T15:50:37Z

Describe the issue:

I'm collaborating on a code that compiles some fixed-form Fortran subroutines using f2py. My collaborators are using Python 3.11 and earlier, whereas I'm failing to build the subroutines on Python 3.12, presumably because I have to use the Meson backend (which I installed through my package manager).

Specifically, an example subroutine fails to build with gfortran without the -ffixed-form flag. But even if I pass this to f2py, I get the same error that gfortran gives me without the flag, as if it's not being passed.

I'm not 100% sure how all these form fields are going to render in the issue, but taking the Fortran source below as mwe.f95, I get ($ represents the terminal prompt):

$ gfortran -c mwe.f95 
mwe.f95:1:22:

    1 |       subroutine test(
      |                      1
Error: Invalid character in name at (1)
mwe.f95:2:7:

    2 |      & x)
      |       1
Error: Invalid character in name at (1)
mwe.f95:4:9:

    4 |       end subroutine test
      |         1
Error: Expecting END PROGRAM statement at (1)
f951: Error: Unexpected end of file in ‘mwe.f95’
$ gfortran -ffixed-form -c mwe.f95  # no output
$ f2py -c --f77flags=-ffixed-form --f90flags=-ffixed-form -m mwe mwe.f95
< full error in the appropriate field but relevant detail is... >
...

[3/6] Compiling Fortran object mwe.cpython-312-x86_64-linux-gnu.so.p/mwe.f95.o
FAILED: mwe.cpython-312-x86_64-linux-gnu.so.p/mwe.f95.o 
gfortran -Imwe.cpython-312-x86_64-linux-gnu.so.p -I. -I.. -I/home/wball/.local/lib/python3.12/site-packages/numpy/core/include -I/home/wball/.local/lib/python3.12/site-packages/numpy/f2py/src -I/usr/include/python3.12 -fvisibility=hidden -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -O3 -fPIC -Jmwe.cpython-312-x86_64-linux-gnu.so.p -o mwe.cpython-312-x86_64-linux-gnu.so.p/mwe.f95.o -c ../mwe.f95
../mwe.f95:1:22:

    1 |       subroutine test(
      |                      1
Error: Invalid character in name at (1)
../mwe.f95:2:7:

    2 |      & x)
      |       1
Error: Invalid character in name at (1)
../mwe.f95:4:9:

    4 |       end subroutine test
      |         1
Error: Expecting END PROGRAM statement at (1)
f951: Error: Unexpected end of file in ‘../mwe.f95’
...
subprocess.CalledProcessError: Command '['meson', 'compile', '-C', 'bbdir']' returned non-zero exit status 1.

Note that the build succeeds if I change the source file extension to .f, perhaps because the fixed-form is inferred.

Reproduce the code example:

! mwe.f95
      subroutine test(
     & x)
          real*8 x
      end subroutine test

Error message:

Cannot use distutils backend with Python>=3.12, using meson backend instead.
Using meson backend
Will pass --lower to f2py
See https://numpy.org/doc/stable/f2py/buildtools/meson.html
Reading fortran codes...
	Reading file 'mwe.f95' (format:free)
Line #1 in mwe.f95:"      subroutine test( "
	analyzeline: No name/args pattern found for line.
Post-processing...
	Block: mwe
			Block: unknown_subroutine
Applying post-processing hooks...
  character_backward_compatibility_hook
Post-processing (stage 2)...
Building modules...
    Building module "mwe"...
    Generating possibly empty wrappers"
    Maybe empty "mwe-f2pywrappers.f"
        Constructing wrapper function "unknown_subroutine"...
          unknown_subroutine()
    Wrote C/API module "mwe" to file "./mwemodule.c"
The Meson build system
Version: 1.3.1
Source dir: /tmp/tmpjodywqy7
Build dir: /tmp/tmpjodywqy7/bbdir
Build type: native build
Project name: mwe
Project version: 0.1
Fortran compiler for the host machine: gfortran (gcc 13.2.1 "GNU Fortran (GCC) 13.2.1 20231205 (Red Hat 13.2.1-6)")
Fortran linker for the host machine: gfortran ld.bfd 2.40-14
C compiler for the host machine: cc (gcc 13.2.1 "cc (GCC) 13.2.1 20231205 (Red Hat 13.2.1-6)")
C linker for the host machine: cc ld.bfd 2.40-14
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program /usr/bin/python3 found: YES (/usr/bin/python3)
Found pkg-config: YES (/usr/bin/pkg-config) 1.9.5
Run-time dependency python found: YES 3.12
Library quadmath found: YES
Build targets in project: 1

Found ninja-1.11.1 at /usr/bin/ninja
INFO: autodetecting backend as ninja                                                                                                                                                          
INFO: calculating backend command to run: /usr/bin/ninja -C /tmp/tmpjodywqy7/bbdir
ninja: Entering directory `/tmp/tmpjodywqy7/bbdir'
[3/6] Compiling Fortran object mwe.cpython-312-x86_64-linux-gnu.so.p/mwe.f95.o
FAILED: mwe.cpython-312-x86_64-linux-gnu.so.p/mwe.f95.o 
gfortran -Imwe.cpython-312-x86_64-linux-gnu.so.p -I. -I.. -I/home/wball/.local/lib/python3.12/site-packages/numpy/core/include -I/home/wball/.local/lib/python3.12/site-packages/numpy/f2py/src -I/usr/include/python3.12 -fvisibility=hidden -fdiagnostics-color=always -D_FILE_OFFSET_BITS=64 -Wall -O3 -fPIC -Jmwe.cpython-312-x86_64-linux-gnu.so.p -o mwe.cpython-312-x86_64-linux-gnu.so.p/mwe.f95.o -c ../mwe.f95
../mwe.f95:1:22:

    1 |       subroutine test(
      |                      1
Error: Invalid character in name at (1)
../mwe.f95:2:7:

    2 |      & x)
      |       1
Error: Invalid character in name at (1)
../mwe.f95:4:9:

    4 |       end subroutine test
      |         1
Error: Expecting END PROGRAM statement at (1)
f951: Error: Unexpected end of file in ‘../mwe.f95’
[5/6] Compiling C object mwe.cpython-312-x86_64-linux-gnu.so.p/15b943306aebd8c2a7f428d911374b526ccafd56_.._.._f2py_src_fortranobject.c.o
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/usr/bin/f2py", line 33, in <module>
    sys.exit(load_entry_point('numpy==1.24.4', 'console_scripts', 'f2py')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wball/.local/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 766, in main
    run_compile()
  File "/home/wball/.local/lib/python3.12/site-packages/numpy/f2py/f2py2e.py", line 738, in run_compile
    builder.compile()
  File "/home/wball/.local/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 178, in compile
    self.run_meson(self.build_dir)
  File "/home/wball/.local/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 173, in run_meson
    self._run_subprocess_command(compile_command, build_dir)
  File "/home/wball/.local/lib/python3.12/site-packages/numpy/f2py/_backends/_meson.py", line 167, in _run_subprocess_command
    subprocess.run(command, cwd=cwd, check=True)
  File "/usr/lib64/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['meson', 'compile', '-C', 'bbdir']' returned non-zero exit status 1.

Python and NumPy Versions:

1.26.3
3.12.1 (main, Dec 18 2023, 00:00:00) [GCC 13.2.1 20231205 (Red Hat 13.2.1-6)]

Runtime Environment:

[{'numpy_version': '1.26.3',
'python': '3.12.1 (main, Dec 18 2023, 00:00:00) [GCC 13.2.1 20231205 (Red '
'Hat 13.2.1-6)]',
'uname': uname_result(system='Linux', node='localhost.localdomain', release='6.7.3-200.fc39.x86_64', version='#1 SMP PREEMPT_DYNAMIC Thu Feb 1 03:29:52 UTC 2024', machine='x86_64')},
{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
'found': ['SSSE3',
'SSE41',
'POPCNT',
'SSE42',
'AVX',
'F16C',
'FMA3',
'AVX2'],
'not_found': ['AVX512F',
'AVX512CD',
'AVX512_KNL',
'AVX512_KNM',
'AVX512_SKX',
'AVX512_CLX',
'AVX512_CNL',
'AVX512_ICL']}},
{'architecture': 'Haswell',
'filepath': '/home/wball/.local/lib/python3.12/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so',
'internal_api': 'openblas',
'num_threads': 4,
'prefix': 'libopenblas',
'threading_layer': 'pthreads',
'user_api': 'blas',
'version': '0.3.23.dev'},
{'filepath': '/usr/lib64/libgomp.so.1.0.0',
'internal_api': 'openmp',
'num_threads': 4,
'prefix': 'libgomp',
'user_api': 'openmp',
'version': None}]
None

Context for the issue:

This appears to be a regression in f2py, in that something that was working under defaults in Python 3.11 (with distutils) appears to broken in Python 3.12 (with Meson). I suspect this is actually a Meson issue but I have limited experience with it.

I have access to a few different systems to try a few tests. If the intention isn't to support what I'm trying, it should be documented. I couldn't (at least in the ~20 minutes I spent paging through) find anything relevant in the f2py docs.

The text was updated successfully, but these errors were encountered:

kamischi · 2024-04-11T20:34:26Z

I encountered the same problem. I pass flags to the fortran compiler to silence some warnings. This stopped working with 3.12. I found a work around with settings FFLAGS before the f2py command, like

FFLAGS=-Wno-maybe-uninitialized f2py ...

h-larsson · 2024-05-31T21:37:48Z

Setting FFLAGS leads to problems when meson does "Sanity testing Fortran compiler" so this workaround does not seem to help here.

Sanity testing Fortran compiler: gfortran
Is cross compiler: False.
Sanity check compiler command line: gfortran sanitycheckf.f90 -o sanitycheckf.exe -ffixed-form -ffixed-form
Sanity check compile stdout:

-----
Sanity check compile stderr:
sanitycheckf.f90:1:2:

    1 | program main; print *, "Fortran compilation is working."; end program
      |  1
Error: Non-numeric character in statement label at (1)

HaoZeke · 2024-06-13T00:56:51Z

Setting FFLAGS leads to problems when meson does "Sanity testing Fortran compiler" so this workaround does not seem to help here.

Sanity testing Fortran compiler: gfortran
Is cross compiler: False.
Sanity check compiler command line: gfortran sanitycheckf.f90 -o sanitycheckf.exe -ffixed-form -ffixed-form
Sanity check compile stdout:

-----
Sanity check compile stderr:
sanitycheckf.f90:1:2:

    1 | program main; print *, "Fortran compilation is working."; end program
      |  1
Error: Non-numeric character in statement label at (1)

This should be reported upstream to meson since this is definitely a bug at their end (reported mesonbuild/meson#13319), the current (post distutils) way to pass flags is via FFLAGS...

In general I'll close this as out of scope for F2Py, the options being:

Pass flags via FFLAGS (recommended)
Change the file extension to let F2PY infer fixed form (not really a great idea, the whole inference of standards is messy)

h-larsson · 2024-06-13T01:46:05Z

@HaoZeke Thanks for your help. However, f2py --help shows me

 --f77flags=          Specify F77 compiler flags
  --f90flags=          Specify F90 compiler flags
  --opt=               Specify optimization flags [NO_MESON]

So according to the documentation, --f77flags and --f90flags should work with meson, which is not the case.

kamischi · 2024-06-13T08:16:23Z

Misleading docs are a nightmare, undocumented work arounds waste endless time and cause frustration. So, there is still a need to do something on the side of numpy, even if it is an upstream bug with meson. Unless a fix in meson can be expected in short time.

HaoZeke · 2024-06-13T08:53:06Z

Thanks @h-larsson and @kamischi, I was a bit hasty closing this, it turns out there's still a bug in the meson template, it gets passed the flags but doesn't use them (they should be set as f_args or maybe c_args on the compiled extension).

Also, the help text needs updating. Will probably get to both this weekend.

Backport of numpy#26703, numpy#26706, and numpy#26812. Closes numpy#25784. Background and notes: - Although the CLI was passing `fc_flags` to the Meson backend they weren't being used - The behavior currently is to (redundantly) collect the unique set of flags passed via either `--f77flags` or `--f90flags` - They all map to `fortran_args` for the `meson.build` (`distutils` used to have separate handling for the two cases) - Eventually these should be merged into a single `--fflags` argument (basically since they're now redundant post-distutils) - This is likely only when we have 3.12 as the lowest supported version, since `distutils` will need to have both CLI options - The test is specially crafted to get an error without `-ffixed-form` but is otherwise junk - Adding a continuation character at the end of the line as well will both get `f2py` and `gfortran` to compile without `-ffixed-form` - Or putting the same code in a `.f` file, thereby signalling to F2PY that this is a Fortran 77 file It does make sense to handle `fortran_args` this way though, since it is more robust than setting `FFLAGS` (which anyway won't work due to a `meson` bug, mesonbuild/meson#13327). Squashed commit of the following: commit bbc198b48b79417f210bd38de6c60ab98d18857d Author: Rohit Goswami <[email protected]> Date: Fri Jun 28 19:43:41 2024 +0200 CI,MAINT: Bump gfortran version [wheel build] Co-authored-by: ngoldbaum <[email protected]> commit fffa296b7a611de1ab25f85619e92e6d6c5185a4 Author: Rohit Goswami <[email protected]> Date: Fri Jun 28 14:54:50 2024 +0200 CI,TST: Fix meson tests needing gfortran [wheel build] commit 807e898e80e544e720512f9feccc9c897cd55f08 Author: Rohit Goswami <[email protected]> Date: Sun Jun 16 09:40:06 2024 +0000 TST: Skip an f2py module test on Windows commit cbcb6c2c4340f5bbca46e2c6d8474bbed7d90664 Author: Rohit Goswami <[email protected]> Date: Sun Jun 16 04:06:16 2024 +0000 MAINT: Be more robust wrt f2py flags commit 09e3eb004b0e076271817384a92d489b3b0cc0eb Author: Rohit Goswami <[email protected]> Date: Sun Jun 16 03:44:17 2024 +0000 TST: Add one for passing arguments to f2py Co-authored-by: warrickball <[email protected]> commit 04d53b44b75537fc5b411cf9e54c2f169198419f Author: Rohit Goswami <[email protected]> Date: Sun Jun 16 00:14:54 2024 +0000 BUG: Use fortran args from f2py in meson commit 6f736b0cbe977ba9270c45f7cc22b7ae8ce06ed2 Author: Rohit Goswami <[email protected]> Date: Sat Jun 15 23:36:55 2024 +0000 MAINT,BUG: Correctly skip distutils options

warrickball added the 00 - Bug label Feb 7, 2024

HaoZeke added the component: numpy.f2py label Jun 13, 2024

HaoZeke closed this as completed Jun 13, 2024

HaoZeke mentioned this issue Jun 13, 2024

BUG: Fixed form flags for Fortran compilers break sanity checks mesonbuild/meson#13319

Closed

HaoZeke reopened this Jun 13, 2024

HaoZeke mentioned this issue Jun 16, 2024

BUG: Handle --f77flags and --f90flags for meson #26703

Merged

mattip closed this as completed in #26703 Jun 16, 2024

charris mentioned this issue Jul 5, 2024

BUG: Handle --f77flags and --f90flags for meson [wheel build] #26870

Merged

DWesl mentioned this issue Apr 15, 2025

BUG: f2py meson issue with -D and #ifdef #28648

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

BUG: f2py appears to ignore Fortran compiler flags #25784

BUG: f2py appears to ignore Fortran compiler flags #25784

warrickball commented Feb 7, 2024

kamischi commented Apr 11, 2024

Uh oh!

h-larsson commented May 31, 2024

Uh oh!

HaoZeke commented Jun 13, 2024 •

edited

Loading

Uh oh!

h-larsson commented Jun 13, 2024

Uh oh!

kamischi commented Jun 13, 2024

Uh oh!

HaoZeke commented Jun 13, 2024

Uh oh!

Uh oh!

BUG: f2py appears to ignore Fortran compiler flags #25784

BUG: f2py appears to ignore Fortran compiler flags #25784

Comments

warrickball commented Feb 7, 2024

Describe the issue:

Reproduce the code example:

Error message:

Python and NumPy Versions:

Runtime Environment:

Context for the issue:

kamischi commented Apr 11, 2024

Uh oh!

h-larsson commented May 31, 2024

Uh oh!

HaoZeke commented Jun 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

h-larsson commented Jun 13, 2024

Uh oh!

kamischi commented Jun 13, 2024

Uh oh!

HaoZeke commented Jun 13, 2024

Uh oh!

HaoZeke commented Jun 13, 2024 •

edited

Loading