Thanks to visit codestin.com
Credit goes to github.com

Skip to content

ENH, DOC: Build notes and fixes for Cygwin. #18308

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 4 commits into from
Closed

ENH, DOC: Build notes and fixes for Cygwin. #18308

wants to merge 4 commits into from

Conversation

DWesl
Copy link
Contributor

@DWesl DWesl commented Feb 3, 2021

Teach the NumPy build system about Cygwin.

Someone seems to have fixed a number of failing tests, so thank you to whomever that might be.

There are twelve tests that still fail.

  • Four appear to be due to the reference implementation of np.abs(np.complex256(+-inf+nanj)) giving inf while the system cabsl(INFINITY + NAN * _IMAGINARY_I) gives nan.
  • Two more are complaining that np.abs(np.complex256(1.7e4932)) should not be inf. The system cabsl(1.1e4932 + 0j) == inf; larger numbers give me compiler warnings about the literal not fitting in a long double.
  • I think another two are related to csqrt{f,} not using the sign of zero for branch cuts.
  • Three involve zero-division warnings in numpy/core/tests/test_umath.py:TestComplexFunctions.test_loss_of_precision:check, and the divisor there is np.arcsinh(z).real. The system arcsinhl(1e-20+0j) == 0+0j
  • The last is np.power(np.complex256(2+0j), 32768+0j) not raising an overflow FPE. I don't know how to check that one.

I can include the tests, written in C, if anyone wants to see those.

Update of #18102, #16246.

@mattip
Copy link
Member

mattip commented Feb 3, 2021

Do you know how difficult it would be to set up a CI run with cygwin, at least for the 64-bit case? Is it supported by github actions/Azure?

@DWesl
Copy link
Contributor Author

DWesl commented Feb 3, 2021

I've heard of someone who's done it. I'll see if I can find their work.

The Azure WindowsFast failure seems to be a problem finding nm.exe, which I think is unrelated.

@DWesl
Copy link
Contributor Author

DWesl commented Feb 3, 2021 •
edited
Loading

egor-tensin/setup-cygwin looks promising. Do you want me to try adding that here? In another PR?

@mattip
Copy link
Member

mattip commented Feb 3, 2021

There are a few things already going on in this PR.

  • test parameterization
  • refactor a helper-function implementation for cygwin
  • blocklist some system library functions
  • check for sys.platform == 'cygwin' or sys.platform == 'win32' rather than just the latter

I think we should try to set up CI for cygwin in another PR, then circle back to this once it is working. That github action looks plausible, you would need the python package and then could follow the way pypy overrides the PATH to make sure the cygwin python and gcc get used.

@mattip
Copy link
Member

mattip commented Feb 3, 2021

@seberg @charris does that make sense as a plan?

@charris charris changed the title Cygwin build notes/updates ENH, DOC: Build notes and fixes for Cygwin. Feb 3, 2021
@DWesl
Copy link
Contributor Author

DWesl commented Feb 4, 2021

The attempt at a 64-bit Cygwin CI run using GitHub actions is up at #18330. Setting up the Cygwin environment seems to be working fine, it looks like just the running the tests bit that's having trouble. I'm hoping someone here has experience with setting up that side of CI and can help get that working.

@DWesl DWesl mentioned this pull request Feb 6, 2021
Merged
Base automatically changed from master to main March 4, 2021 02:05
@charris
Copy link
Member

charris commented Apr 13, 2021

A rebase should fix the test failures.

@DWesl DWesl force-pushed the cygwin-build-notes branch 2 times, most recently from 7da3732 to 1a9abbf Compare April 14, 2021 01:53
Carreau
Carreau reviewed Apr 17, 2021
View reviewed changes
@charris
Copy link
Member

charris commented Jul 21, 2021

@DWesl Needs rebase. How much of this is still needed?

@DWesl
Copy link
Contributor Author

DWesl commented Jul 21, 2021

I think the only bits that haven't gone in one way or another are telling the fortran compiler that Cygwin doesn't have -rpath and rewriting cyg2win32 to use a function designed for translating paths to something Windows understands.

The tests are passing without needing to be told that rpath doesn't work, so apparently that isn't necessary anymore.

cyg2win32 is only used in numpy.distutils for the Absoft compilers. Converting paths from Cygwin-style to Windows-style is needed only if calling a Windows program from Cygwin python, which may work for 32-bit C programs that don't do I/O. For 64-bit extensions, Cygwin long int being 64 bits while Windows long int is 32 bits is going to cause problems if this is an extension for the Cygwin python, rather than kind of cross compiled for a Windows python install (instead of using the Cygwin-to-WIndows cross-compilers, or using that Windows python install to compile the extensions).

Leaving cyg2win32 as it is probably won't hurt anything, but it won't be solving whatever problem it's supposed to solve as often as it could. I suppose we could just leave it until someone who wants to do the exact combination of things this function wants to help with runs into problems and switch over then.

I can do PRs for both, if people are interested. The -rpath thing is less than a line.

There's a few options for fixing cyg2win32. I can use the command-line utility designed to convert Cygwin-style paths to Windows-style paths, or I can use the C API function the utility wraps using ctypes or Cython, or I can try rolling my own conversion of all absolute paths, parsing /etc/fstab to figure out what needs to turn into a drive letter. fork() is slow on Cygwin and sometimes fails, but I'm not sure the other options are worth the bother.

DWesl added 4 commits July 21, 2021 07:13
@DWesl
TST: Add Cygwin to the x86 feature tests.
e61028b 
This was suggested by @seiko2plus for debugging a segfault in the
tests on Cygwin:
numpy#18102 (comment)

This test passes on Cygwin, and the whole testsuite has only the
failures I expect from running on Cygwin (see numpy#18102 and numpy#16246).
@DWesl
BLD: Tell fortran compiler Cygwin doesn't support rpath.
2f20c93 
Cygwin inherits Windows's lack of RPATH.
@DWesl
BLD: Change cyg2win32 to use the cygpath utility.
e30548d 
The old function only handled the default `/cygdrive` prefix.  This
can be customized to different values: `/` and `/mnt` are both common.
`/proc/cygdrive` does the same thing, regardless of user customization.

`cygpath` handles all of these.  There's also a C function if you'd
prefer to avoid the `fork()` call, which tends to be slow on Cygwin.

Edit: Fix the cyg2win32 function to have correct types.

It returned bytes earlier; this should return a string.

Edit: Fix docsrting to follow numpydoc.
@DWesl
TST: Parametrize a few more tests.
a254171 
Move the for loops outside the test body so I can get more than one
error at once.

STY: Fix long lines flagged by linter.
@DWesl DWesl force-pushed the cygwin-build-notes branch from 1a9abbf to a254171 Compare July 21, 2021 11:21
@DWesl
Copy link
Contributor Author

DWesl commented Jul 21, 2021

Remaining changes rebased. These are:

  • Run the x86 feature tests on Cygwin as well as Linux. The Cygwin tests still pass with this additional test, so I may as well add them.
  • Tell numpy.distutils.fcompiler.gnu that Cygwin doesn't have rpath. The compiler seems to have figured that out without the hint, since the tests passed before I told it about this. Leaving this out probably wouldn't be a big loss.
  • Have cyg2win32 use Cygwin's utilities for converting paths from Cygwin-style to Windows style. I have no idea what problem this is trying to solve, but if NumPy still wants to solve whatever problem that is, I can make that solution more general.
  • Parametrize a few more tests. This allows reporting multiple failures from the parametrized tests, and allows more precise error reports when using pytest-forked to work around a segfault. Since I've already got the Cygwin tests passing, I suppose the main reason to add this would be making the tests more idomatic for pytest.

I can make separate PRs for these to clarify discussion.

On one CI runner, the test test_build_import, which is skipped if sys.platform != "win32", fails with

subprocess.CalledProcessError: Command '['nm.exe', '--help']' returned non-zero exit status 4294967295.

I don't think that's something I did.

This was referenced Jul 21, 2021
Merged
Merged
Merged
Merged
@DWesl
Copy link
Contributor Author

DWesl commented Jul 29, 2021

And the last four bits of this PR went in with PRs #19534, #19535, #19542, and #19547, so this can be closed.

@DWesl DWesl closed this Jul 29, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Carreau Carreau Carreau left review comments

Assignees
No one assigned
Labels
01 - Enhancement 04 - Documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@DWesl @mattip @charris @Carreau