It does seem to be a problem with OpenBLAS and threading on windows. NumPy 1.26.4 uses v0.3.23-293. Numpy 2.0 and 2.0.1 use v0.3.27.44. Setting OPENBLAS_NUM_THREADS=1 makes the test pass. I didn't see any relevant issues searching for windows and threading on the OpenBLAS issue tracker, but maybe I need to search differently. @martin-frbg does this sound familiar?

I wonder how our CI passed ☹️

I don't think this is related to the kernels chosen. The test still fails when I set OPENBLAS_CORETYPE=PRESCOTT (it was HASWELL).

what does 0.3.27.44 correspond to, git-wise ? If post-0.3.27, any chance this could be related to the NAN handling changes in SCAL that I merged on June 29 ?

For what it's worth, sunpy has this test because 4 years ago we encountered intermittent failures with OpenBLAS and multithreading on macOS (see sunpy/sunpy#4290 (comment)). Those failures went away eventually with a subsequent release of OpenBLAS.

Importantly, there were and maybe still are bad interactions when a threaded program used OpenBLAS that was configured to use threads as well. Hence, that's why setting OpenBLAS to use a single thread "fixed" issues.

Windows gets a lot less attention, and there were a few PRs merged since 0.3.23 (whatever build your -293 is) and 0.3.27-whatever that (supposedly) improved multithreading on that platform. Notably OpenMathLib/OpenBLAS#4359 - not trying to blame anyone, just that this would be the most serious change (merged in December, so debuting in 0.3.26)

@mseminatore

Is it possible to test this against an earlier (pre 0.3.26) OpenBLAS version to isolate whether this could be related to the Windows threading changes? @martin-frbg do we have existing validation tests that would/should show this issue?

Does this show up for any NTHREADS > 1?

Do we have a C repro for this yet? If not I will attempt to translate the Python to C and see if I can reproduce/debug it.

I have just managed to set up a virtual environment for reproducing this (with a pip-installed numpy - I did not get conda to install anything other than mkl or netlib even if it claimed to have switched to openblas). It definitely happens with as few as 4 cores (though not on every run), but I have not yet gotten it to fail with just two. (Unfortunately I haven't figured out how to downgrade the openblas version, pip tells me there are no packages that match the version requirement)

3 threads are definitely enough to reproduce it (on the fourth try), I still haven't seen a failure with 2 in about 20 runs so far

The openblas implementation is in <venv>/lib/site-packages/numpy.libs. You can try various scipy-openblas64 versions. Note that pip-installing the package does not replace the DLL used by numpy: the package is only used when building numpy from source. You can copy the DLL from the site-package/scipy_openblas/lib directory, overwriting the one in numpy.libs (be sure to preserve the mangled name).

Note I am close to uploading a newer scipy-openblas64 based on latest OpenBLAS HEAD. You can download a pre-release from https://anaconda.org/scientific-python-nightly-wheels/scipy-openblas64/files. You want scipy_openblas64-0.3.27.341.0-py3-none-win_amd64.whl.

@mattip thank you very much. No failures seen so far with 4 threads and the 0.3.24.95.1 version of the library copied over the original libscipy_openblas64_-fb1711452d4d8cee9f276fd1449ee5c7.dll in numpy.libs

@Siddharth-Latthe-07 is that a chatgpt-generated comment? Your comment does not add to the body of knowledge around the issue. We already know it is related to OpenBLAS, and that it is related to some thread-related problem inside OpenBLAS that only manifests itself on windows, so your analysis of the problem is not helpful. Converting the wheels we ship to another BLAS/LAPACK implementation is not something we are considering at this time.

I've seen this happen with dtype = float32 as well. @martin-frbg in validating that this doesn't happen with earlier versions have you by chance been able to grab a log file with server tracing enabled?

a log file with server tracing enabled

How can I help provide a version with that enabled?

@mattip if OpenBLAS is built with SMP_DEBUG defined then the thread server will send debug logging to stderr. That may help hint at what is going on.

@mseminatore have not gotten around to using an own build in that context yet, sorry. will try tonight.

@martin-frbg in looking over the Windows server code I note the merge of OpenMathLib/OpenBLAS#4577 in April which says it changes how threads are allocated and introduces new functions like adjust_thread_buffers(). The note on adjust_thread_buffers() says that it changes from local buffers to global buffers which sounds like it could have introduced a subtle new timing requirement on buffer usage.

Not trying to shift the focus elsewhere, but it would be helpful to rule out that merge as a potential source of these issues.

@mattip Is there any sense for when this issue started? IOW is the repro test case something that is run regularly or only when an issue is identified? As Martin pointed out, the most significant code change to Windows threading was my merge (4359) on Dec. 5th 2023 so that is a prime candidate and I will look over the code, but wondering if that long of a latency in discovering the issue is expected.

#4577 was merged only after the 0.3.27 release so probably/hopefully not in that build yet.
as I understood, the repro is from sunpy i.e. another level removed from numpy/scipy and was noticed there only because numpy/scipy switched from bundling OpenBLAS 0.3.23 to 0.3.27. the only other potential indication of trouble is a headsup from the conda packager that their build of 0.3.27 is surprisingly slow compared to earlier ones.

(conda issue is conda-forge/openblas-feedstock#160 (comment))

#4577 was merged only after the 0.3.27 release so probably/hopefully not in that build yet. as I understood, the repro is from sunpy i.e. another level removed from numpy/scipy and was noticed there only because numpy/scipy switched from bundling OpenBLAS 0.3.23 to 0.3.27. the only other potential indication of trouble is a headsup from the conda packager that their build of 0.3.27 is surprisingly slow compared to earlier ones.

Thank you @martin-frbg that is helpful to know! I will continue to review the code and attempt to debug the repro in pursuit of a working theory and a potential fix. Please let me know should you decide that you would prefer reverting the code given that this is not the first issue we've encountered and that this is having downstream impact. Regardless of whether we keep or revert the Windows code I hope we can expand test coverage to allow us to catch similar issues earlier in the future.

I'll see if I can come up with anything tonight. We are already close to a month past my tentative release date for 0.3.28 due to various stability problems (including with my health), so I guess it would not be a problem to add another week in case it becomes obvious where this goes wrong.

not sure if this output from a build with SMP_DEBUG is going to help (stopped the program after it reported the first couple of mismatches)
debuglog.zip

@mattip do you happen to know the magnitude of the errors? Is it just outside of epsilon or large? Trying to roughly characterize this as either lost work or error accumulation to guide the investigation.

Should confirm that src = src.copy("C") fixes it then in Python

No, I still am seeing problems even with this.

If I shrink the size of x to 300_000, I do not see the problem. Maybe a int32 overflow somewhere?

It should mean it is 44 commits past 0.3.27, but when looking at openblas-libs's commit 14ebf5d, I see the pyproject.toml version is 0.3.27.44.1 but the OpenBLAS commit is exactly v0.3.27. That is a mistake and is misleading, sorry.

The next jump in OpenBLAS version is in the 0.3.27.57 wheel, which uses OpenBLAS de465ffd.

sorry I do not understand - does that mean your 0.3.27.44 happened to be OpenBLAS-0.3.27 exactly, or that its exact commit hash past 0.3.27 can no longer be determined ? (de465ffd would appear to be shortly before 4577 landed so I think that other thread-handling change should not have any bearing if the sunpy folks used "0.3.27.44")

If I shrink the size of x to 300_000, I do not see the problem. Maybe a int32 overflow somewhere?

Good catch but that would be extra weird as mseminatore's work was entirely threading infrastructure not concerned with data sizes, and I do not recall any PR in the relevant timeframe that could have replaced a "blasint" with a standard int (rendering your regular INTERFACE64=1 build option useless). I had started bisecting, but ran into weird stability issues with the VM and/or compiler so need to redo everything...

sorry I do not understand - does that mean your 0.3.27.44 happened to be OpenBLAS-0.3.27 exactly?

Yes

If I shrink the size of x to 300_000, I do not see the problem. Maybe a int32 overflow somewhere?

Good catch but that would be extra weird as mseminatore's work was entirely threading infrastructure not concerned with data sizes, and I do not recall any PR in the relevant timeframe that could have replaced a "blasint" with a standard int (rendering your regular INTERFACE64=1 build option useless). I had started bisecting, but ran into weird stability issues with the VM and/or compiler so need to redo everything...

The last issue was a bug in queue management where work submitted re-entrantly was lost. The way the join logic works exec_blas_async_wait() will hang if we don't process all the work that was submitted. So, I would expect lost work to result in a hang as before.

I am reviewing the log file that Martin captured to see if there are any hints there.

OK, looking over the log file, I am seeing the following pattern for work submission:

Server[ 0] Started.  Mode = 0x2003 M =   2 N=500000 K=  2
Server[ 1] Started.  Mode = 0x2003 M =   2 N=500000 K=  2
Server[ 2] Started.  Mode = 0x2003 M =   2 N=500000 K=  2

This suggests that each call to cblas_dgemm() which if I understand the repro code would be called from Python with M=2, N=500000 and K=2 was not decomposed into a set of smaller sub-tasks. Help me out @martin-frbg, is that expected behavior for a level3 GEMM call?

I would expect the matrix to be converted into a number of sub-tiles/sub-tasks in a work_queue (such that each sub-tile A, B and C all fit in the L2$) but I am not familiar with the OpenBLAS L3 work model. My math says that a MxN matrix of order (2, 500000) of float64 would be 8MB. Therefore A, B, and C would be 8MB, 32b (2 * 2 * 8) and 8MB, thus 16MB in size, larger than any L2$ I've seen.

As an example, my own BLAS code decomposes this GEMM call into 1954 sub-tasks in the work_queue.

But I am dealing with a family medical situation, it's late and I'm tired so I may be thinking wrongly about it.

Saw this too, maybe it is the extreme imbalance of matrix dimensiions (and openblas failing to split on N).
I may have broken it myself in OpenMathLib/OpenBLAS#4585 - an attempt to get better utilization of threads. If I got that wrong and it makes OpenBLAS go single-threaded, there would be nothing the level3 driver could do to split the workload. This should affect all platforms equally though, and would not explain why the problem only appears on Windows.
(It is very early morning here, and I have been dealing with an ongoing medical situation in my family too in recent months. I will try to look into this particular PR first thing in the morning. Eventually the bisect should find the cause, if it isn't this one either)

sorry, so far I have only managed to trash my build environment - the ancient m2w64-gcc (5) from conda (that I am sure worked on another system on the weekend) suddenly throws internal compiler errors when building OpenBLAS.
Trying to replace it with the conda-forge::gcc (14) ends in a mess of conflicting dependencies involving the msys2-conda-epoch package as well as "make", and a separate msys2/mingw64 build of OpenBLAS appears to be incompatible with the pip-installed numpy (although symbols look correct).

Using the DLL from the 0.3.26.0.4 scipy-openblas64 wheels, this test and the one from #26643 both pass. That uses the OpenBLAS v0.3.26 tagged version.

Now I am getting myself confused. Rerunning the test with a 0.3.26 OpenBLAS fails, both when I use the DLL from the 0.3.26.4 wheel and also when I rebuild OpenBLAS v0.3.26.

I am using this to rebuild OpenBLAS with the compilers from the rtools package. This installs gcc/gfortran 10.3

rem this needs rtools installed, run this in an admin cmd.exe
rem choco install -y rtools --no-progress --force --version=4.0.0.20220206
set BASH_PATH=c:\rtools40\usr\bin\bash.exe
set CHERE_INVOKING=yes
set INTERFACE64=1
set LDFLAGS=-lucrt -static -static-libgcc
set MSYSTEM=UCRT64
%BASH_PATH% -lc make BINARY=64 DYNAMIC_ARCH=0 USE_THREAD=1 USE_OPENMP=0 NUM_THREADS=24 NO_WARMUP=1 NO_AFFINITY=1 CONSISTENT_FPCSR=1 BUILD_LAPACK_DEPRECATED=1 TARGET=PRESCOTT BUFFERSIZE=20 'LDFLAGS=-lucrt -static -static-libgcc' 'COMMON_OPT=-O2 -march=x86-64 -mtune=generic -fno-asynchronous-unwind-tables' 'FCOMMON_OPT= -O2 -march=x86-64 -mtune=generic -fno-asynchronous-unwind-tables -frecursive -ffpe-summary=invalid,zero -fdefault-integer-8' MAX_STACK_ALLOC=2048 INTERFACE64=1 SYMBOLSUFFIX=64_ LIBNAMESUFFIX=64_ SYMBOLPREFIX=scipy_ LIBNAMEPREFIX=scipy_ FIXED_LIBNAME=1

Then I can copy the DLL over the one used by numpy.

Unfortunately, the prefixing does not work correctly pre-0.3.26, so I will continue bisecting backwards from 0.3.26 using NumPy 1.26.4 (which does not have any scipy_ function and dll prefixes but does use the 64_ suffixes).

I have 0.3.26 building for NumPy 1.26.4 (using only SYMBOLSUFFIX) and it fails the test. Now bisecting further back...

Interesting, that would mean I am in the clear with my GEMM thread count change from PR 4585. Thanks for the chocolatey pointer, looks like I'm up and running with that again now.

I have a good commit with v0.3.25. Now I can bisect.

Bisecting: 91 revisions left to test after this (roughly 7 steps)"

Hmm. good is at d32f38fb, bad is at 302ca7edc7, PR OpenMathLib/OpenBLAS#4359 is inside the bisect range. I won't be able to finish the bisect today (each step takes an hour), but that is the most probable candidate of the remaining revisions. Maybe worthwhile staring at that a bit.

thanks - my bisect is four steps from completion, I am also building a current HEAD with 4359 manually reverted now "just in case".

Indeed HEAD with the changes from 4359 reverted appears to work fine - whether that means the problem is in 4359 or only uncovered by it remains to be seen. (Bisect still needs two steps but appears to be gravitating towards the same result)

Thank you @mattip abd @martin-frbg for the bisect work. I am sorry that I can’t more actively participate (for personal reasons) but will continue inspecting the code to see if I can identify a cause and fix.

I think it is equally possible that the actual problem lies somewhere in driver/level3/level3_thread.c (though it has its own CriticalSection), but I lack the Windows developer experience to assess this

I haven’t looked very closely at that code but I will take a look.

Any further thoughts about #OpenMathLib/OpenBLAS#4835 and whether NumPy should build its OpenBLAS with that PR as a patch to move the NumPy2.1 release forward?

not sure if that counts as thoughts, but my current intention is to merge 4835 in order to get the 0.3.28 release out in the next couple of days (probably not today though)

I backported the changes in OpenBLAS's 4835 in the 0.3.27.44.4 version of scipy-openblas, and used those wheels in #27140. I also added the test to prevent this from regressing again. Closing. Please reopen or open a new issue if I missed something.