This is indeed great and much needed. Thank you for this.

Recently, we added code using two ways of declaring LAPACK prototypes from C/C++ :

1. Declaring plain unmangled BLAS/LAPACK prototypes (https://github.com/scipy/scipy/blob/main/scipy/optimize/__lbfgsb.h#L26):

#include "../_build_utils/src/npy_cblas.h"
#include "../_build_utils/src/fortran_defs.h"
void dpotrf_(char* uplo, int* n, double* a, int* lda, int* info);

2. Using BLAS_FUNC name mangling from npy_cblas.h (https://github.com/scipy/scipy/blob/main/scipy/interpolate/src/__fitpack.h#L83):

#include "../_build_utils/src/npy_cblas.h"
#include "../_build_utils/src/fortran_defs.h"
#define DLARTG BLAS_FUNC(dlartg)
void DLARTG(double *f, double *g, double *cs, double *sn, double *r);

Both ways pass CI, both our regular runs and wheel builds.

Ralf pointed out offline that 1. possibly works because the SciPy build process constructs name mangled wrappers already, via https://github.com/scipy/scipy/blob/main/scipy/_build_utils/_generate_blas_wrapper.py and https://github.com/scipy/scipy/blob/main/scipy/_build_utils/meson.build#L12

So does it mean we don't need to redo mangling manually and 1. is the right way forward?

First two lines of 1. is not needed. That's my experiment I forgot to remove. I'll fix it when I am done troubleshooting #21583, (the problem size is large so it is taking some time to figure it out). The better example is #21553 for item 1 without those lines.

Solution (2) should be the right thing to use. I'm pretty sure that (1) is not robust. Fortran compiler mangling results are compiler-dependent, so the trailing underscore in void dpotrf_ isn't guaranteed to be there. CI passes because we only test with compilers that do add that underscore, plus:

Ralf pointed out offline that 1. possibly works because the SciPy build process constructs name mangled wrappers already

Note that we do this if and only if there's a symbol prefix or suffix used (so for scipy-openblas32 and for Accelerate). The reason for that is that the generated wrappers add ~1600 files and increase compile times by 10-20%. Hence, a regular LAPACK install built with a compiler that doesn't add the trailing underscore won't work right now.

Problem I encountered with (2) is that not all symbols are available (#21314). So that gives me the impression that the list is maintained and not generated.

I would prefer a solution, if possible at all, that we create some intermediate layer that separates compiler issues from the symbol names. That is to say from the application layer all LAPACK consuming routines see a single standard (with or without underscore, or usage like (2) etc. it doesn't matter what the standard is) and we handle the symbols at a central place potentially investigating the names and generating some runtime macros, somehow take care of it in one place.

I still don't have a good grip on where things happen or which tool is leaking symbols of (1) or (2) but when you are back in action let's have an offline session together with Evgeni if you folks are up to it.

I would prefer a solution, if possible at all, that we create some intermediate layer that separates compiler issues from the symbol names.

+1. We had an offline discussion that there's a header file autogenerated at some build stage, and maybe the right thing is to just commit it to the repository instead. That would be my preferred solution, too.
Will need to poke around a bit though.

let's have an offline session ... if you folks are up to it.

Definitely let's!

xref gh-22743 for some relevant discussion and updates to support ILP64.