ping @HaoZeke

Ran into a similar issue recently, a workaround is to add

contains
subroutine has_module_lanedata()
end subroutine has_module_lanedata

before the end module lanedata (and similarly before end module types). NumPy assumes that data-only modules used by other Fortran code are for Fortran reference purposes only, and are not to be exported to Python.

@DWesl Thank you. Yes, that this is a workaround for the current behaviour, as I wrote.

However,

• this is not what f2py did in the past, so breaks backward compatibility w/o need.
• this assumption would be very unreasonable as I have a lot more moldes with subroutines that are used internally only then the other way round.

If it was desired to exclude modules, a f2py directive to hide modules are parts thereof should be introduced.

Things I do not want in the interface I compile in external libraries. There used to be f2py commands to include/exclude specific subroutines/symbols on interface creation. Things I put in the interface and for which I use f2py to make the pyf file are there because I explicitly want them in the interface on the first place.

FWIW I don't think this was an intended change. f2py is complicated, has poor test coverage, and few people understand the internals. We would have left it alone, but the removal of distutils by Python forced us to make some changes and you're seeing the fallout.

It's not a perfect situation at all. If you want to help to fix this issue that would be very appreciated.

Thanks everyone for the discussion here, I've been thinking of how best to resolve this and will have a PR up this weekend most likely. As far as I can remember this (poorly documented) breaking change had to do with the way F2PY was extended to handle derived types.

The solution is better docs + an explicit hiding directive to handle declaration only derived type modules.

@HaoZeke is one of the people who have a lot of knowledge of f2py internals, so I'll defer to them :)

@HaoZeke You have been always very helpful fixing things swiftly. Thank you so much for that!

My recent small patch suggestions to allow the setup for my f2py build framework to continue work with Python 3.12 after the discontinuation of distutils (at least one of them seems a real bug, the other was adding a simple option for backward compatibility that would not have broken anything else or had side effects other than having to "maintain" three extra lines of code), however, were rejected, so I found it futile to suggest further patches myself.

What do people do in this case, maybe need to make a branch and rebase every time there is a numpy update? (For matplotlib I also just have a mplfixes module where obviously useful natural extensions were not accepted.)

@HaoZeke You have been always very helpful fixing things swiftly. Thank you so much for that!

Thanks, sorry this one took a while to get to.

My recent small patch suggestions to allow the setup for my f2py build framework to continue work with Python 3.12 after the discontinuation of distutils (at least one of them seems a real bug, the other was adding a simple option for backward compatibility that would not have broken anything else or had side effects other than having to "maintain" three extra lines of code), however, were rejected, so I found it futile to suggest further patches myself.

Sorry could you refresh my memory on this? Generally I'd love to try to get either existing working patchsets in or equivalent changes (i.e. those which pass the original test failure)

What do people do in this case, maybe need to make a branch and rebase every time there is a numpy update? (For matplotlib I also just have a mplfixes module where obviously useful natural extensions were not accepted.)

There's (IMO terrible) precedent set in the form of f90wrap which has a hard dependency on f2py but is technically a different project; and eventually diverges and winds down..

For build systems, it is probable that a separate extension module will be used (since NumPy will not vendor any distutils code anymore), but making a branch and rebasing is not ergonomic (and confusing to everyone using it too).

@HaoZeke OK, I may try to cast these into real pull requests.

It is related to the thread #24874 with items
#24874 (comment)
and after
#24874 (comment)
as well as
#24874 (comment)
It does not seem obvious to me why not just allow users pass their own flags if they really want to, even if it is not in the spirit of usual package distribution; some --- possibly many --- people just want to develop and compile locally for their own use, no distribution intended.

I dug into this a bit, and it certainly is an interesting one, thanks for reporting! For starters, it seems to only show up in certain cases, for instance, this works:

module types

  implicit none

  INTEGER, PARAMETER :: int32 = SELECTED_INT_KIND(8)
  INTEGER, PARAMETER :: int64 = SELECTED_INT_KIND(16)
  INTEGER, PARAMETER :: real64 = SELECTED_REAL_KIND(15)
end module types

module lanedata

  use types, only: &
       real64, int32

  implicit none

  save

  integer(kind=int32), parameter :: &
       maxdata = 2**20-1

  real(kind=real64), dimension(:,:), allocatable :: &
       theta

  integer(kind=int32) :: &
       ndata

end module lanedata

in blah.f90 with f2py -m ltest blah.f90 -c works fine:

In [1]: import ltest

In [2]: dir(ltest)
Out[2]: 
['__doc__',
 '__f2py_numpy_version__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__spec__',
 '__version__',
 '_ltest_error',
 'lanedata']

In [3]: dir(ltest.lanedata)
Out[3]: 
['__call__',
 '__class__',
 '__delattr__',
 '__dir__',
 '__doc__',
 '__eq__',
 '__format__',
 '__ge__',
 '__getattribute__',
 '__getstate__',
 '__gt__',
 '__hash__',
 '__init__',
 '__init_subclass__',
 '__le__',
 '__lt__',
 '__ne__',
 '__new__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__setattr__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 'maxdata',
 'ndata']

In [4]: ltest.lanedata.ndata
Out[4]: array(0, dtype=int32)

In [5]: ltest.lanedata.maxdata
Out[5]: array(1048575, dtype=int32)

However, the bug is reproducible, if a subroutine outside a module is present (which a situation we don't currently have tests for), if there is more than one module present in a single file.

module types

  implicit none

  INTEGER, PARAMETER :: int32 = SELECTED_INT_KIND(8)
  INTEGER, PARAMETER :: int64 = SELECTED_INT_KIND(16)
  INTEGER, PARAMETER :: real64 = SELECTED_REAL_KIND(15)

end module types

module lanedata

  use types, only: &
       real64, int32

  implicit none

  save

  integer(kind=int32), parameter :: &
       maxdata = 2**20-1

  real(kind=real64), dimension(:,:), allocatable :: &
       theta

  integer(kind=int32) :: &
       ndata

end module lanedata

subroutine freelanedata()

  use lanedata, only: &
       ndata, theta

  implicit none

  if (allocated(theta)) deallocate(theta)
  ndata = -1

end subroutine freelanedata

Where now, with f2py -m ltest blah.f90 -c:

In [1]: import ltest

In [2]: dir(ltest)
Out[2]: 
['__doc__',
 '__f2py_numpy_version__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__spec__',
 '__version__',
 '_ltest_error',
 'freelanedata']

In [3]: 

There are a few things I need to understand conceptually about what the intended behavior is though. The reason this happens is because when there are subroutines they get exported into a module of the same name as requested on the command line (ltest in this case), and there is likely a bug in the way the modules are (not) being exported. Should have a fix soon.

Bizarrely, I think there's something more specific to the modules reported here by @2sn; since this also works:

module datonly

  implicit none

  integer, parameter :: max_value = 100

  real, dimension(:), allocatable :: data_array

end module datonly


module dat

  implicit none

  integer, parameter :: max_= 1009

end module dat

subroutine simple_subroutine(arg)
  integer, intent(inout) :: arg
  arg = arg * 5
end subroutine simple_subroutine

This also works..

In [1]: import datonly

In [2]: datonly.datonly.max_value
Out[2]: array(100, dtype=int32)

In [3]: datonly.datonly.max_value
Out[3]: array(100, dtype=int32)

In [4]: datonly.dat.max_
Out[4]: array(1009, dtype=int32)

In [5]: dir(datonly)
Out[5]: 
['__doc__',
 '__f2py_numpy_version__',
 '__file__',
 '__loader__',
 '__name__',
 '__package__',
 '__spec__',
 '__version__',
 '_datonly_error',
 'dat',
 'datonly',
 'simple_subroutine']

@HaoZeke Thank you very much for looking into this.

I am not sure whether your question about the intent/use case is directed at me. What my Python code dies is trying to access the date in module lanedata directly. Obviously, one could add a function to return the data, but such calls also tend to be rather expensive.

I do not know how access to module data is done internally by f2py, maybe there is a similar implicit function call in the first place, but it is certainly easier in terms of syntax (unless one creates a fancy Python class wrapping such function calls).

Do you plan to have a f2py directive to hide or explicitly include modules (there used to be some f2py command line flags to include or exclude symbols from wapping), or just include all? Except for a small bit of overhead at compile time (and maybe small increase in module size), what would be reasons to not include all module and their public members by default? Hiding private implementation details? Looking at hour case study, it seems what is included and what not is based upon some guesses what the user may likely have intended and desired, but it seems hard to anticipate actual use cases in general.

@HaoZeke Thank you very much for looking into this.

I am not sure whether your question about the intent/use case is directed at me. What my Python code dies is trying to access the date in module lanedata directly. Obviously, one could add a function to return the data, but such calls also tend to be rather expensive.

I do not know how access to module data is done internally by f2py, maybe there is a similar implicit function call in the first place, but it is certainly easier in terms of syntax (unless one creates a fancy Python class wrapping such function calls).

Do you plan to have a f2py directive to hide or explicitly include modules (there used to be some f2py command line flags to include or exclude symbols from wapping), or just include all? Except for a small bit of overhead at compile time (and maybe small increase in module size), what would be reasons to not include all module and their public members by default? Hiding private implementation details? Looking at hour case study, it seems what is included and what not is based upon some guesses what the user may likely have intended and desired, but it seems hard to anticipate actual use cases in general.

Include all should definitely be the default (and is again after #27695). There are tests for the hide directives, so that should be OK.

I'm thinking of adding some examples on using derived types soon, but otherwise, mostly just thinking of bugfixes / docs in the short term.