Just to say, that I can run the 64-bit build in the following way. I have a script c:\tmp\do_build.sh like this:

export LDFLAGS="-lucrt -static -static-libgcc"
export OPENBLAS_COMMIT="bfd9c1b58cd3"
bash tools/build_openblas.sh "c:\\opt" 64 32

I can then run the whole build with:

cd openblas-libs
c:\rtools40\ucrt64.exe c:\tmp\do_build.sh

This puts up an msys window, where the work happens, but the command meanwhile returns - so I am not sure how to plumb that into the CI

EDIT - fixed - see below.

Ah - I think the right incantation is (for example):

$env:LDFLAGS = "-lucrt -static -static-libgcc"
$env:OPENBLAS_COMMIT = "bfd9c1b58cd3"
$env:CHERE_INVOKING = 'yes'  # Preserve the current working directory
$env:MSYSTEM = 'UCRT64'  # Start a 64 bit UCRT environment
$env:OPENBLAS_ROOT = 'c:\opt'
$env:BUILD_BITS = "64"
& C:\rtools40\usr\bin\bash -lc tools/build_openblas.sh

I'm curious as to why the staging repo says the the labels are non-standard and doesn't show the openblas libs along with the other projects.

@charris - is the stuff from the staging repo to do with this PR? I haven't changed the non-Windows build, and CI is failing as yet on Windows - I'm working on that.

is the stuff from the staging repo to do with this PR?

No, I just wanted you to be aware. It seems recent, there didn't used to be that warning.

@mattip @carlkl - I've refactored now so the builds work, but the dynamic library test is now failing. You can't see the failure in the Github Actions logs, but I believe the failure is due to the following error that I do see on my local newly installed Windows 10 machine:

$ C:\rtools40\usr\bin\bash -lc ./test_dyn.exe
C:/repos/openblas-libs/test_dyn.exe: error while loading shared libraries: api-ms-win-crt-filesystem-l1-1-0.dll: cannot open shared object file: No such file or directory

Do you have any insight as to why the dynamic build would fail in this way? Anything I can do to debug?

similar to: facebook/watchman#627

@carlkl - I can't find api-ms-win-crt-filesystem-l1-1-0.dll on my machine. I see that it is a standard Windows DLL. There are references to it in current MinGW-w64 source.

I also searched in a previous recent updated Windows install, and didn't find that DLL.

@mstorsjo - sorry to ping you here - but does MingW-w64 UCRT64 expect this file to be on all Windows systems? Or have we made some error in compilation, to trigger this requirement?

@carlkl - I can't find api-ms-win-crt-filesystem-l1-1-0.dll on my machine. I see that it is a standard Windows DLL. There are references to it in current MinGW-w64 source.

I also searched in a previous recent updated Windows install, and didn't find that DLL.

@mstorsjo - sorry to ping you here - but does MingW-w64 UCRT64 expect this file to be on all Windows systems? Or have we made some error in compilation, to trigger this requirement?

If you're running on Windows Vista/7/8 and haven't installed the UCRT redistributable package, you won't have this DLL and applications built with UCRT won't run.

If you're running on Vista/7/8 and have got the UCRT redistributable package installed, then you should have this DLL, and applications should run successfully.

If you're running on Windows 10/11, then this DLL actually is a system-built in API set DLL, and I think you might not actually find a physical DLL on disk with this file name, but executables that link against it still should run successfully. In the case of this DLL, it should redirect to ucrtbase.dll (which should exist on disk). Windows has got a bunch of such API set DLLs, see e.g. https://docs.microsoft.com/en-us/windows/win32/apiindex/api-set-loader-operation for more information on them.

@mattip @carlkl - I've refactored now so the builds work, but the dynamic library test is now failing. You can't see the failure in the Github Actions logs, but I believe the failure is due to the following error that I do see on my local newly installed Windows 10 machine:

$ C:\rtools40\usr\bin\bash -lc ./test_dyn.exe
C:/repos/openblas-libs/test_dyn.exe: error while loading shared libraries: api-ms-win-crt-filesystem-l1-1-0.dll: cannot open shared object file: No such file or directory

Do you have any insight as to why the dynamic build would fail in this way? Anything I can do to debug?

Ok, so this was the actual culprit here that I was supposed to comment on...

The issue actually is that bash intercepts the loader error (which normally would pop up a dialog box saying that it failed to load the executable, stating why and what was missing), but often this interception ends up pointing at the wrong missing DLL. I'd suggest trying to execute the binary without bash and checking with the dialog box what it really says is missing.

I tested the artifacts locally without error. (Windows 10).

artfacts: https://my.hidrive.com/share/5fn9a7aevd (send from @matthew-brett)

@mstorso - thanks - that's very helpful.

@carlkl - when I test locally I get a silent error from test_dyn.exe, like this:

PS C:\repos\openblas-libs> .\test.exe
 DPOTRF result                    1                    1  0.82785411218911054
 DPOTRF result                    2                    1  0.76974640611133693
 DPOTRF result                    2                    2   1.3552705831602272
 DPOTRF result                    3                    1  0.45205472353685744
 DPOTRF result                    3                    2   1.4664558880996119
 DPOTRF result                    3                    3  0.97387319602809452
                    1                    1   2.1529541746598611        2.1529541746598615
                    2                    1  -1.0310643946797247       -1.0310643946797260
                    2                    2   1.7789091870364546        1.7789091870364571
                    3                    1  0.48504845452853407       0.48504845452853507
                    3                    2  -1.1408752692976014       -1.1408752692976032
                    3                    3   1.0543751803797956        1.0543751803797967
 OK
PS C:\repos\openblas-libs> echo $?
True
PS C:\repos\openblas-libs> .\test_dyn.exe
PS C:\repos\openblas-libs> echo $?
False

I believe we should be getting the same output from both test*.exe calls.

Which in turn is probably due to:

(venv) PS C:\repos\openblas-libs> dlldiag.exe deps .\test_dyn.exe
DLL Diagnostic Tools version 0.0.18
Copyright (c) 2019-2021 Adam Rehn

Parsing module header and identifying direct dependencies... done.

Parsed module details:
Module:          C:\repos\openblas-libs\test_dyn.exe
Type:            Executable
Architecture:    x64

Attempting to load the module's direct dependencies:

api-ms-win-crt-convert-l1-1-0.dll                      Loaded successfully
api-ms-win-crt-environment-l1-1-0.dll                  Loaded successfully
api-ms-win-crt-filesystem-l1-1-0.dll                   Loaded successfully
api-ms-win-crt-heap-l1-1-0.dll                         Loaded successfully
api-ms-win-crt-locale-l1-1-0.dll                       Loaded successfully
api-ms-win-crt-math-l1-1-0.dll                         Loaded successfully
api-ms-win-crt-private-l1-1-0.dll                      Loaded successfully
api-ms-win-crt-runtime-l1-1-0.dll                      Loaded successfully
api-ms-win-crt-stdio-l1-1-0.dll                        Loaded successfully
api-ms-win-crt-string-l1-1-0.dll                       Loaded successfully
api-ms-win-crt-time-l1-1-0.dll                         Loaded successfully
api-ms-win-crt-utility-l1-1-0.dll                      Loaded successfully
KERNEL32.dll                                           Loaded successfully
libopenblas64__v0.3.20-140-gbfd9c1b5-gcc_10_3_0.dll    Error 126: The specified module could not be found.

I wonder whether this was always happening in our CI, it's just that we had not previously detected the error.

Thanks everyone - this is now ready for review. See edited introductory comment for a summary.

For unrelated test failure - see OpenMathLib/OpenBLAS#3659

This looks like very nice progress, thanks @matthew-brett & all.

A short summary to make sure I understand the impact:

• This will now do all Windows builds (32-bit, and ILP flavor of 64-bit) with Mingw-w64 instead of with MSVC
• We want to use this build with both NumPy and SciPy
• It does not change the dll's we have to bundle into wheels (UCRT is present, we'll still need to add the gfortran runtime dll).
• It should not make a significant difference performance-wise (did we test this?)
• (and for the future:) we'd be able to distribute this as a separate wheel on PyPI once we'd bundle 32-bit and 64-bit together into a wheel

* This will now do all Windows builds (32-bit, and ILP flavor of 64-bit) with Mingw-w64 instead of with MSVC

Unless I got very confused, we were always building with mingw - the difference in this PR was only to drop the stray use of the lib.exe VC binary to build a .lib export library, and instead use dlltool for the same job. This is because I believe Nathaniel's fix to binutils an age ago will allow modern dlltool to do the job. And, I did do a test build against Numpy with VC++.

* We want to use this build with both NumPy and SciPy

Yes.

* It does not change the dll's we have to bundle into wheels (UCRT is present, we'll still need to add the `gfortran` runtime dll).

I think that's right - but it will depend if -static-libgcc affects the Gfortran libraries, and on any other changes in Mingw-w64 / gfortran.

* It should not make a significant difference performance-wise (did we test this?)

It may improve performance, if the UCRT builds pick up the MS standard library math routines - but no - I haven't tested.

* (and for the future:) we'd be able to distribute this as a separate wheel on PyPI once we'd bundle 32-bit and 64-bit together into a wheel

Yes - I can't see why not - though I don't think this PR will affect that.

I have a little patch laying around for OpenBLAS to allow for debug;unstripped / non-debug;stripped linkage and to suppress the unnecessary linkage to one single symbol from libquadmath:

diff -rupN OpenBLAS/exports/Makefile OpenBLAS_tmp/exports/Makefile
--- OpenBLAS/exports/Makefile	2018-05-19 21:22:05.606912400 +0200
+++ OpenBLAS_tmp/exports/Makefile	2018-05-19 22:43:16.556514800 +0200
@@ -93,11 +93,21 @@ dll  : ../$(LIBDLLNAME)
 # in their import table.  By instead using a stable name it is possible to
 # upgrade between library versions, without needing to re-link an application.
 # For more details see: https://github.com/xianyi/OpenBLAS/issues/127.
+ifeq ($(DEBUG), 1)
 ../$(LIBDLLNAME) : ../$(LIBNAME) $(LIBPREFIX).def dllinit.$(SUFFIX)
 	$(RANLIB) ../$(LIBNAME)
 	$(CC) $(CFLAGS) $(LDFLAGS) $(LIBPREFIX).def dllinit.$(SUFFIX) \
-	-shared -o ../$(LIBDLLNAME) -Wl,--out-implib,../$(IMPLIBNAME) \
+	-shared -o ../$(LIBDLLNAME) -Wl,--defsym,quadmath_snprintf=snprintf \
 	-Wl,--whole-archive ../$(LIBNAME) -Wl,--no-whole-archive $(FEXTRALIB) $(EXTRALIB)
+	dlltool -D $(LIBDLLNAME) -d $(LIBPREFIX).def -l ../$(LIBDLLNAME).a
+else
+../$(LIBDLLNAME) : ../$(LIBNAME) $(LIBPREFIX).def dllinit.$(SUFFIX)
+	$(RANLIB) ../$(LIBNAME)
+	$(CC) $(CFLAGS) $(LDFLAGS) $(LIBPREFIX).def dllinit.$(SUFFIX) \
+	-shared -o ../$(LIBDLLNAME) -Wl,--defsym,quadmath_snprintf=snprintf -Wl,-gc-sections -Wl,-s \
+	-Wl,--whole-archive ../$(LIBNAME) -Wl,--no-whole-archive $(FEXTRALIB) $(EXTRALIB)
+	dlltool -D $(LIBDLLNAME) -d $(LIBPREFIX).def -l ../$(LIBDLLNAME).a
+endif
 
 $(LIBPREFIX).def : gensymbol
 	perl ./gensymbol win2k    $(ARCH) dummy $(EXPRECISION) $(NO_CBLAS) $(NO_LAPACK) $(NO_LAPACKE) $(NEED2UNDERSCORES) $(ONLY_CBLAS) "$(SYMBOLPREFIX)" "$(SYMBOLSUFFIX)" $(BUILD_LAPACK_DEPRECATED) > $(@F)

Carl - do you think it's safe to apply your patch to our builds?

Hm, the patch is not fresh, so I would not bet on this.

I can build OpenBLAS locally for test, but not today.

@carlkl - can we apply this in post-processing, without patching OpenBLAS?

@carlkl - sorry to keep on with the questions - but your patch only applies to the Windows build. What benefit do we get from avoiding the quadmath_snprintf symbol? I believe we can, with the current OpenBLAS builds, already build e.g. Numpy without the libquadmath library being available. Is that not correct?

@matthew-brett, quadmath_snprintf is the only missing symbol if the quadmath library is not linked in. There is no technical concern linking in libquadmath, it's only the question if you want to have code inlcuded with LGPL licence. In case of scipy I guess you want to avoid that. See OpenMathLib/OpenBLAS#2685 and https://scipy.github.io/old-wiki/pages/License_Compatibility.html

If libquadmath is linked statically then the binary code of quadmath_snprintf will be part of libqudmath.dll.

However, quadmath_snprintf won't ever be used by OpenBLAS, so you simple can replace it with some mockup code.

Found my comment from 2019 again numpy/numpy#15049 (comment)

OpenBLAS should be linked without quadmath dependency to avoid possible problems with licensing
(LGPL without exeptions).

but no one was interested in this...

OK - I've put your patch into this PR. Would you mind reviewing? I agree it's a good idea in general to avoid the libquadmath dependency. I guess we need similar patches for the Unix builds?

Aha - @carlkl - your patch gives an error on the 32-bit build only:

2022-06-23T20:44:23.4930843Z C:/rtools40/mingw32/bin/../lib/gcc/i686-w64-mingw32/8.3.0/../../../../i686-w64-mingw32/bin/ld.exe:--defsym:1: undefined symbol `snprintf' referenced in expression

OK - I pushed your (@carlkl) patch out to #85, for further reflection. I think this one is ready to merge as-is.

I don't think we need that quadmath patch for Unix builds. I'm not a lawyer, but I think it's about what are you going to deploy with the OpenBLAS binary For Linux you only link against existing libraries. But this argument could be wrong. To be sure one could ask FSF i.e.

I will test it locally and the push the quadmath patch to the OpenBLAS repo. If they accept it you can remove it later on in this repo.

What can I do to make this one suitable for merge? I think it's ready from my side.

Any idea what is up with the macOS builds?

Edit: macOS arm64 builds

I think that's just because the fix to add back the cross-compile flags is not in this branch. I've rebased.

I think I need this one for #89 since windows no longer builds the current master.

Thanks @matthew-brett

Unfortunately something else changed and the upload failed for win32.

Uploading OpenBLAS v0.3.20-140-gbfd9c1b5 to anaconda.org staging:
-rw-r--r-- 1 runneradmin None 23M Nov  2 19:12 builds/openblas-v0.3.20-140-gbfd9c1b5-win32-gcc_8_3_0.zip
tools/upload_to_anaconda_staging.sh: line 18: anaconda: command not found

I see I can use the conda in the image via the conda github action. I will try that on v0.3.21