BUG: Fixes for building on Cygwin. #16246
Conversation
Leaving it later can lead to some headers getting confused, because they're included with one set of feature flags, but the headers they depend on were included with a different set of feature flags.
…rebase. Cygwin needs each DLL to have a unique address for fork() emulation to work. I'm hoping that calling rebase on a complete list of modules compiled in this session, plus those installed globally, will allow the test suite to get an accurate result for more tests.
… on cygwin. Most likely I should be testing for newlib (the C runtime, roughly takes the place of glibc, I think), not cygwin (the emulation layer on Windows), but I have no idea how to do that from within python. If someone working on embedded systems runs into this issue, this hopefully gives them some idea where to start.
…win. See two commits back for an explanation of why fork() fails on cygwin. Alternately, see the much better explanation at: https://cygwin.com/cygwin-ug-net/highlights.html#ov-hi-process with hints on error messages and workarounds at: https://cygwin.com/faq.html#faq.using.fixing-fork-failures
|
The failure in Travis seems to be because it couldn't install gfortran, which seems unrelated. |
|
The header change seems trivial, the distutils change I do not know, maybe @rgommers can have a look at some point. @embray you seems to have looked at cygwin related things before. If you have time, any input is appreciated! |
|
Looking at |
|
To clarify, this is Cygwin, not MinGW. I can try to cross-compile or check wheels from somewhere if necessary, but nothing here is related to MinGW. As an interesting side note, compiling latest master works just fine, and seems to have fewer failures than this branch. I'm suspecting I did something wrong with the I'll try to check the problematic floating-point functions manually and add problematic ones to |
|
I'm happy to take a look at this, but could you provide a brief overview of what this fixes? I haven't personally had any problems lately with compiling Numpy on Cygwin. Is this fixing any build issues? Otherwise it seems to be mostly concerned with fixing the test suite, and setting a number of tests to xfail. I admit I haven't tried running the full test suite on Cygwin in a while. |
|
A few months ago (#14787 (comment), though I think I had noted the problem earlier), numpy wasn't compiling on Cygwin unless I made I was getting a lot of Several tests failed and still fail due to what seemed like tiny floating-point differences, sometimes nonexistent differences as the branch cut tests don't always report which side of the branch they're trying to test. My first approach was to ignore them, as I saw no relevant differences. That is the third commit. Earlier comments in this thread directed me to There were still tests with A question for the maintainers: should I try to include the modified |
|
@DWesl Out of curiosity what versions of Cygwin and GCC are you currently using? |
|
@embray Out of curiosity, what AVX-related errors are you seeing? I ran into some a few months ago and reported those in #14787, where other people mentioned possible solutions. Edit: Just saw #16290, which looks like the same problem, and you already found the solution I ended up using. |
These changes made some sense when compiling only for cygwin. They do not belong in NumPy master.
I don't know if the list should be the same as one of the Windows lists, or if this should instead be a Newlib list. I know how to make a Cygwin list, so I did that. It didn't always work, though.
The generic "this doesn't work" was confusing people. Specifying what "this" is and what happed works better.
Mark tests suddenly passing, and also document how I expect them to fail.
This was actually making the situation worse, I think because I forgot to include the NumPy dlls themselves. Removing this made for many fewer fork() failures when I tried it. `python3 -m pip show numpy --files | grep dll` will give a list of dlls, with at least a relative path. This can be done in python with subprocess (will require adding pip as a runtime dependency), but I'm not sure that's in scope here.
I found that when building the latest master branch on Cygwin, while testing #16246, that thousands of warnings were generated at build time like: numpy/core/src/npysort/binsearch.c.src: In function ‘binsearch_left_bool’: numpy/core/src/npysort/binsearch.c.src:82:1: warning: visibility attribute not supported in this configuration; ignored [-Wattributes] Granted this is just a warning, so I don't think it's a serious issue. It seems the test that was supposed to check for __attribute__ support was not working as expected. The #pragmas only take effect if I provide a function body--they are ignored for bare declarations. I don't know if that's by intent, or if it's a GCC issue. For reference: $ gcc --version gcc (GCC) 7.4.0
These should primarily be on test functions.
I found that when building the latest master branch on Cygwin, while testing numpy#16246, that thousands of warnings were generated at build time like: numpy/core/src/npysort/binsearch.c.src: In function ‘binsearch_left_bool’: numpy/core/src/npysort/binsearch.c.src:82:1: warning: visibility attribute not supported in this configuration; ignored [-Wattributes] Granted this is just a warning, so I don't think it's a serious issue. It seems the test that was supposed to check for __attribute__ support was not working as expected. The #pragmas only take effect if I provide a function body--they are ignored for bare declarations. I don't know if that's by intent, or if it's a GCC issue. For reference: $ gcc --version gcc (GCC) 7.4.0
I don't know enough to make that decision. |
|
For the broader question of how this is working; there were two tests that segfault on Cygwin versions between June and the present; we may want to wait until the fix is in a release before merging into main. I apparently copied over the preview of Cygwin 3.2.0, so I still have segfaults in the I also get: |
The bulk of the changes in #17548 seems to be here: The corresponding code in this PR: Only one of |
|
@DWesl I'll be making a release in a day or two. You should pick what need to be added, if anything, and do it. There may, or may not, be another release in the 1.19 series. Have you tested with current master? |
This reverts commit 6ffbcb2. Replaced by 17548.
This reverts commit d9c21bb. Replaced by numpy#17548.
This reverts commit 92eaff3. Replaced by numpy#17548.
|
I dropped my attempt to add the compiler flags, since #17548 probably got tested and I don't think this did. 38 failures on master with python3.6, many due to the ufunc errors fixed/xfailed here, and more due to the polynomial class being confused about unicode vs ascii printing. 15 failures on master with python3.7, mostly due to the ufunc errors fixed or xfailed in this PR. NumPy arrays are still marked as instances of Hashable, and there's a few other errors that aren't NumPy related. 7 failures on cygwin-fixes with python3.7, one due to not running I can't think of anything else I know how to do, unless someone has pointers for fixing the |
| // int*, int64* should be propertly aligned on ARMv7 to avoid bus error | ||
| #if !defined(NPY_STRONG_ALIGNMENT) && defined(__arm__) && !(defined(__aarch64__) || defined(_M_ARM64)) | ||
| #define NPY_STRONG_ALIGNMENT 1 | ||
| #endif |
There was a problem hiding this comment.
@seiko2plus @Qiyu8 Thoughts about this. Do you deal with this elsewhere?
There was a problem hiding this comment.
yes, there's already an opened pr for it, see #18065
|
Closing, I have a rebased version that I will push for examination. |
|
See #18102 for rebased version. |
This was suggested by @seiko2plus for debugging a segfault in the tests on Cygwin: numpy#18102 (comment) This test passes on Cygwin, and the whole testsuite has only the failures I expect from running on Cygwin (see numpy#18102 and numpy#16246).
This was suggested by @seiko2plus for debugging a segfault in the tests on Cygwin: numpy#18102 (comment) This test passes on Cygwin, and the whole testsuite has only the failures I expect from running on Cygwin (see numpy#18102 and numpy#16246).
This was suggested by @seiko2plus for debugging a segfault in the tests on Cygwin: numpy#18102 (comment) This test passes on Cygwin, and the whole testsuite has only the failures I expect from running on Cygwin (see numpy#18102 and numpy#16246).
Changes needed to get numpy to compile on Cygwin, then a few changes to try to reduce the number of test failures due to extension module overlap, then marking differences I think are due to floating-point implementation or fork() failures.
Someone should probably check whether the list of ignored floating-point failures is reasonable.
Most of those have been around since 1.16 a while back.