Thanks to visit codestin.com
Credit goes to github.com

Skip to content

TST: Windows f2py tests hang indefinitely with msys gcc-fortran on PATH #9342

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
matthew-brett opened this issue Jul 1, 2017 · 11 comments
Open

TST: Windows f2py tests hang indefinitely with msys gcc-fortran on PATH #9342

matthew-brett opened this issue Jul 1, 2017 · 11 comments
Labels
05 - Testing component: CI component: numpy.f2py

Comments

@matthew-brett
Copy link
Contributor

With Appveyor file:

# vim ft=yaml

build_script:
  - pip install numpy

test_script:
  # Adding this line causes the tests to hang forever
  # - PATH=%PATH%;c:\msys64\usr\bin
  # Unless you uninstall gcc-fortran like this
  # - pacman -R --noconfirm gcc-fortran
  - pip install nose
  - python -c "import numpy.f2py; numpy.f2py.test(verbose=4)"

Without adding msys64 to the path, install + f2py tests pass in 30 seconds or so:

https://ci.appveyor.com/project/matthew-brett/windows-wheel-builder/build/1.0.53#L29

Adding msys64 to the path - hangs until build timeout (I canceled this one after 90 seconds):

https://ci.appveyor.com/project/matthew-brett/windows-wheel-builder/build/1.0.52#L16

Adding msys64 to the path, but uninstalling gcc-fortran, tests pass again:

https://ci.appveyor.com/project/matthew-brett/windows-wheel-builder/build/1.0.51#L40
matthew-brett added a commit to matthew-brett/windows-wheel-builder that referenced this issue Jul 1, 2017
@matthew-brett
Test without msys2 on the PATH
8fdf590 
See: numpy/numpy#9342
@carlkl
Copy link
Member

carlkl commented Jun 16, 2022

@matthew-brett, is this issue still valid?

@carlkl
Copy link
Member

carlkl commented Jun 16, 2022

@matthew-brett, is this issue still topical?

@carlkl
Copy link
Member

carlkl commented Jun 18, 2022

I successfully tested this with Windows 10 and

Python 3.9.4 (tags/v3.9.4:1f2e308, Apr  6 2021, 13:40:21) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

and the UCRT64 GCC toolchain from msys2:

gcc version 12.1.0 (Rev2, Built by MSYS2 project)

9342.log

However, a lot of tests are skipped right now due to #9673, so I leave this open for now.

@carlkl
Copy link
Member

carlkl commented Jun 19, 2022

With un-skipped test (#9673) I got a lot of errors, but not hang or segfault with python -c "import numpy.f2py; numpy.f2py.test(verbose=4)"

@matthew-brett
Copy link
Contributor Author

@carlkl - I get the same as you, after laboriously editing out the decorators for skipping the tests:

433 passed, 36 deselected, 122 errors in 149.94s (0:02:29)

@matthew-brett
Copy link
Contributor Author

Errors appear to be of form:

ImportError: DLL load failed while importing _test_ext_module_5430: The specified module could not be found.

@carlkl
Copy link
Member

carlkl commented Jun 20, 2022 •
edited
Loading

If I manually add '--compiler=mingw32' to f2py_opts in f2py/tests/utils.py I get rid of most of the errors. I think this is due to the fact, that the '--compiler=mingw32' flag is mandatory if using mingw-w64. This means f2py/tests/utils.py needs some extra logic for the mingw-w64 case.

However, there are 39 errors left, also with i.e. ImportError: DLL load failed while importing _test_ext_module_5403. For this specific case I manually edited part of the log from python -m pytest -l -k "test_callback" --verbosity=4:

_test_ext_module_5403
---------------------

...
INFO: compiling Fortran sources
INFO: Fortran f77 compiler: C:\\Users\\devel\\AppData\\Local\\pool\\msys2\\ucrt64\\bin\\gfortran.exe -Wall -g -ffixed-form -fno-second-underscore -O3 -funroll-loops
Fortran f90 compiler: C:\\Users\\devel\\AppData\\Local\\pool\\msys2\\ucrt64\\bin\\gfortran.exe -Wall -g -fno-second-underscore -O3 -funroll-loops
Fortran fix compiler: C:\\Users\\devel\\AppData\\Local\\pool\\msys2\\ucrt64\\bin\\gfortran.exe -Wall -g -ffixed-form -fno-second-underscore -Wall -g -fno-second-underscore -O3 -funroll-loops
creating C:\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmpqakzci6_\\Release\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmpxtytyh1g
INFO: compile options: '-IC:\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmpqakzci6_\\src.win-amd64-3.9 
  -IC:\\Users\\devel\\AppData\\Local\\tmp\\Python39\\Lib\\site-packages\\numpy\\core\\include 
  -IC:\\Users\\devel\\AppData\\Local\\tmp\\Python39\\include 
  -IC:\\Users\\devel\\AppData\\Local\\tmp\\Python39\\Include -cINFO: g"
fortran.exe:f77: C:\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmpxtytyh1g\\tmp2fg4qn3k.f
INFO: C:\\Users\\devel\\AppData\\Local\\pool\\msys2\\ucrt64\\bin\\gfortran.exe 
  -Wall -g -Wall -g -shared 
  C:\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmpqakzci6_\\Release\\users\\devel\\appdata\\local\\msys2\\tmp\\tmpqakzci6_\\src.win-amd64-3.9\\_test_ext_module_5403module.o 
  C:\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmpqakzci6_\\Release\\users\\devel\\appdata\\local\\msys2\\tmp\\tmpqakzci6_\\src.win-amd64-3.9\\fortranobject.o 
  C:\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmpqakzci6_\\Release\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmpxtytyh1g\\tmp2fg4qn3k.o 
  -LC:\\Users\\devel\\AppData\\Local\\pool\\msys2\\ucrt64\\lib\\gcc\\x86_64-w64-mingw32\\12.1.0 
  -LC:\\Users\\devel\\AppData\\Local\\tmp\\Python39\\libs 
  -LC:\\Users\\devel\\AppData\\Local\\tmp\\Python39 
  -LC:\\Users\\devel\\AppData\\Local\\tmp\\Python39\\PCbuild\\amd64 
  -lpython39 -lgfortran 
  -o  .\\_test_ext_module_5403.cp39-win_amd64.pyd

Removing build directory C:\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmpqakzci6_

        p          = <Popen: returncode: 0 args: ['C:\\Users\\devel\\AppData\\Local\\tmp...>
        skip       = []
        source_files = ['C:\\Users\\devel\\AppData\\Local\\msys2\\tmp\\tmp2fg4qn3k.f']
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

name = '_test_ext_module_5403', package = None

    def import_module(name, package=None):
        """Import a module.
    
        The 'package' argument is required when performing a relative import. It
        specifies the package to use as the anchor point from which to resolve the
        relative import to an absolute import.
    
        """
        level = 0
        if name.startswith('.'):
            if not package:
                msg = ("the 'package' argument is required to perform a relative "
                       "import for {!r}")
                raise TypeError(msg.format(name))
            for character in name:
                if character != '.':
                    break
                level += 1
>       return _bootstrap._gcd_import(name[level:], package, level)
E       ImportError: DLL load failed while importing _test_ext_module_5403: Das angegebene Modul wurde nicht gefunden.

It seems, that `.\_test_ext_module_5403.cp39-win_amd64.pyd was successfully build, but the build directory has been removed before running the test?

@HaoZeke
Copy link
Member

HaoZeke commented Nov 11, 2023

@carlkl or @matthew-brett, it'd be really great to see if these errors are still present since the transition to meson (and after #25111).

@HaoZeke HaoZeke changed the title Windows f2py tests hang indefinitely with msys gcc-fortran on PATH TST: Windows f2py tests hang indefinitely with msys gcc-fortran on PATH Nov 11, 2023
@carlkl
Copy link
Member

carlkl commented Nov 12, 2023

I can certainly investigate this, but it will take some time ...

@carlkl
Copy link
Member

carlkl commented Nov 15, 2023 •
edited
Loading

Short test with Windows 10 64-bit

python -c "import numpy.f2py; numpy.f2py.test(verbose=4)"

C:\Users\dev\AppData\Local\home\devel\Python311\Lib\site-packages\numpy\_pytesttester.py:143: DeprecationWarning:

  `numpy.distutils` is deprecated since NumPy 1.23.0, as a result
  of the deprecation of `distutils` itself. It will be removed for
  Python >= 3.12. For older Python versions it will remain present.
  It is recommended to use `setuptools < 60.0` for those Python versions.
  For more details, see:
    https://numpy.org/devdocs/reference/distutils_status_migration.html


  from numpy.distutils import cpuinfo
NumPy version 2.0.0.dev0+git20231114.0f61000
NumPy CPU features:  SSE SSE2 SSE3 SSSE3* SSE41* POPCNT* SSE42* AVX* F16C* FMA3* AVX2* AVX512F* AVX512CD* AVX512_SKX* AVX512_CLX? AVX512_CNL? AVX512_ICL?
================================================= test session starts =================================================
platform win32 -- Python 3.11.6, pytest-7.4.2, pluggy-1.3.0 -- C:\Users\dev\AppData\Local\home\devel\Python311\python.exe
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(WindowsPath('C:/Users/dev/AppData/Local/home/tmp3/np_PR25111/numpy_CI_runs_6872272007_job_18690474876/patch/.hypothesis/examples'))
rootdir: C:\Users\dev\AppData\Local\home\tmp3\np_PR25111\numpy_CI_runs_6872272007_job_18690474876\patch
plugins: hypothesis-6.88.1, cov-4.1.0, timeout-2.2.0, xdist-3.3.1
collected 803 items / 36 deselected / 767 selected

=============================================== short test summary info ===============================================
FAILED tests/test_f2py2e.py::test_gen_pyf_no_overwrite - OSError: [WinError 6] Das Handle ist ungültig
================== 1 failed, 59 passed, 704 skipped, 36 deselected, 3 xfailed, 22 warnings in 35.34s ==================

so must of the the tests are skipped, is this intended? test_gen_pyf_no_overwrite fails with an OSError.

@HaoZeke
Copy link
Member

HaoZeke commented Nov 15, 2023

Short test with Windows 10 64-bit

Block (27 lines)

python -c "import numpy.f2py; numpy.f2py.test(verbose=4)"

C:\Users\dev\AppData\Local\home\devel\Python311\Lib\site-packages\numpy\_pytesttester.py:143: DeprecationWarning:

  `numpy.distutils` is deprecated since NumPy 1.23.0, as a result
  of the deprecation of `distutils` itself. It will be removed for
  Python >= 3.12. For older Python versions it will remain present.
  It is recommended to use `setuptools < 60.0` for those Python versions.
  For more details, see:
    https://numpy.org/devdocs/reference/distutils_status_migration.html


  from numpy.distutils import cpuinfo
NumPy version 2.0.0.dev0+git20231114.0f61000
NumPy CPU features:  SSE SSE2 SSE3 SSSE3* SSE41* POPCNT* SSE42* AVX* F16C* FMA3* AVX2* AVX512F* AVX512CD* AVX512_SKX* AVX512_CLX? AVX512_CNL? AVX512_ICL?
================================================= test session starts =================================================
platform win32 -- Python 3.11.6, pytest-7.4.2, pluggy-1.3.0 -- C:\Users\dev\AppData\Local\home\devel\Python311\python.exe
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(WindowsPath('C:/Users/dev/AppData/Local/home/tmp3/np_PR25111/numpy_CI_runs_6872272007_job_18690474876/patch/.hypothesis/examples'))
rootdir: C:\Users\dev\AppData\Local\home\tmp3\np_PR25111\numpy_CI_runs_6872272007_job_18690474876\patch
plugins: hypothesis-6.88.1, cov-4.1.0, timeout-2.2.0, xdist-3.3.1
collected 803 items / 36 deselected / 767 selected

=============================================== short test summary info ===============================================
FAILED tests/test_f2py2e.py::test_gen_pyf_no_overwrite - OSError: [WinError 6] Das Handle ist ungültig
================== 1 failed, 59 passed, 704 skipped, 36 deselected, 3 xfailed, 22 warnings in 35.34s ==================

so must of the the tests are skipped, is this intended? test_gen_pyf_no_overwrite fails with an OSError.

Ah, sorry, jumped the gun, testing these should be after #25134..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
05 - Testing component: CI component: numpy.f2py
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@matthew-brett @pearu @HaoZeke @carlkl