We have 8 total failures. All of them are expected namespace mismatch failures, as array_api_extra doesn't allow disabling the namespace check.

The GPU failures demonstrate something we need to address — atol and rtol should either be explicitly rejected if they are arrays (rather than Python scalars), or explicitly accepted and converted either to NumPy arrays or a scalar type. I'm not sure exactly what we'd want yet.

The 'install into venv' failure I think can be addressed when we make the assertions public by ensuring that their module does not depend on pytest.

RuntimeError: Can't call numpy() on Tensor that has conjugate bit set. Use tensor.resolve_conj().numpy() instead.

That's probably something we want to fix over in array-api-extra.

The GPU failures demonstrate something we need to address — atol and rtol should either be explicitly rejected if they are arrays (rather than Python scalars), or explicitly accepted and converted either to NumPy arrays or a scalar type. I'm not sure exactly what we'd want yet.

I think we should try explicitly accepting 0-D arrays in array-api-extra for these parameters, but rejecting higher-dimensional arrays. If there are a bunch of failures for 1-D arrays with a single element then maybe it makes sense to accept that also.

or explicitly accepted and converted either to NumPy arrays or a scalar type.

I don't know much about pytorch tensors, but can we do

float(4e-16 * ref.integral)

In the tests to make this work?

Or : https://docs.pytorch.org/docs/2.12/tensors.html

But ofc this won't be a proper fix as we need to address these inputs in the extra module.

What I am proposing is something like (in array-api-extra):

actual_np = as_numpy_array(actual, xp=xp)
desired_np = as_numpy_array(desired, xp=xp)

if not isinstance(atol, float):
    atol = as_numpy_array(atol, xp=xp)
    if atol.ndim > 0:
        # raise error

if not isinstance(rtol, float):
    rtol = as_numpy_array(rtol, xp=xp)
    if rtol.ndim > 0:
        # raise error

np.testing.assert_allclose(actual_np, desired_np, atol, rtol, ...)

Makes sense. Let me open a PR in array_api_extra.

I think we want to add to https://github.com/data-apis/array-api-extra/blob/main/src/array_api_extra/_lib/_testing.py#L126-L127 array = array.resolve_conj().

Having a single canonical set of assertions across the ecosystem would be very valuable indeed. Now, the scipy versions have had a lot of effort and polishing. They have basically stabilized a while ago and have been shown to work in a non-trivial project (scipy).

Decisions like whether to wrap native asserions vs convert to numpy etc have already been discussed extensively, so what would be a reason to re-discover issues like pytorch conjugation bits #25143 (comment)?

The array-api-extra versions are IIUC adapted from the scipy versions, and simplified to meet internal needs of array-api-strict. Now if this effort is to make the array-api-versions be the versions for the ecosystem, I'd expect that the more polished/full-featured versions just go into array-api-strict, modulo potentially ripping out too specific parts (_0d shenanigans etc, potentially xp_capabilities).

So I'd expect that this is what happens: copy-paste the scipy versions into array-api-strict, and have a clear summary/discussion of where the line is drawn between "scipy-specific" and "generally useful".

Is this not what's going on here?

Is this not what's going on here?

Yes, that's exactly what we are trying to do. I have already discussed a few things, scipy testing functions and the array_api_extras function does differently in the PR discussion. Our current tests match very closely with _strict_checks in Scipy.

Yes, that's exactly what we are trying to do. I have already discussed a few things, scipy testing functions and the array_api_extras function does differently in the PR discussion. Our current tests match very closely with _strict_checks in Scipy.

Then #25143 (comment) shouldn't be even a thing because this was figured in the scipy versions long long time ago.

(I assume you mean array-api-extra when you say array-api-strict)

Decisions like whether to wrap native asserions vs convert to numpy etc have already been discussed extensively

I'm not so sure... I have a feeling this is basically a decision I made a few years ago that seemed alright to those involved at the time.

So I'd expect that this is what happens: copy-paste the scipy versions into array-api-strict, and have a clear summary/discussion of where the line is drawn between "scipy-specific" and "generally useful".

Is this not what's going on here?

When @crusaderky was working on adding primitive versions of these assertions for array-api-extra, I think it became clear that wrapping of native assertions was itself a scipy-specific historical quirk that we didn't have good reason to keep around. At the time, when I was getting to grips with the array API standard, I think I was too much in the mindset of 'keep everything native where possible' — now, I don't think such a heuristic applies so simply to the test-suite. I'm happy to take the blame for that, although it was one of the first things I ever worked on 😉.

Unfortunately, there is no clear summary/discussion accurate to the current state, as the development has been dispersed across multiple years. @crusaderky highlighted the main initial differences (aside from the non-use of native assertions) at data-apis/array-api-extra#17 (comment), but since then @mdhaber and @prady0t have worked on adding missing features that SciPy and scikit-learn need.

Is there anything in particular you are worried about being missing / erroneously dropped from the SciPy versions? Perhaps you are worried that we are silently dropping strictness? I'm pretty confident that we haven't missed anything crucial — Guido and Matt spent a significant amount of time on this initially, and the importance of @prady0t's work now is to help demonstrate that in CI. But the purpose of this PR is to figure out precisely if we are missing anything.

If a full write-up of the differences between the SciPy versions and the xpx versions is what is required to get this merged, then we should do that eventually once we have the public API in xpx.testing. I'm convinced, however, that starting again from scratch by copy-pasting over the implementations from SciPy would be a waste of time.

Well, we are trying to always convert to numpy with this PR. Scipy uses torch.testing for torch xp, whereas in array_api_extra we try to convert everything to a numpy array. This is something that sklearn also does for testing.

Yes, that's exactly what we are trying to do. I have already discussed a few things, scipy testing functions and the array_api_extras function does differently in the PR discussion. Our current tests match very closely with _strict_checks in Scipy.

Then #25143 (comment) shouldn't be even a thing because this was figured in the scipy versions long long time ago.

That's not true — this isn't a problem that was 'figured out' in SciPy a long time ago. It is a problem that was entirely avoided by the decision to wrap native assertions (which, to be fair, I don't know if you were involved in at all).

If you feel strongly that wrapping native assertions is the way to go, then we could have a new discussion on that (probably in a new array-api-extra issue?). But let's avoid assuming that because something was decided in SciPy years ago, it is the gold-standard against which alternatives must prove their legitimacy. I certainly trust my judgement now more than my judgement back then, at least 😅

I assume you mean array-api-extra when you say array-api-strict)

Indeed a typo, thanks!

(native assertions vs np.testing)

I have a feeling this is basically a decision I made a few years ago

Okay, did not know it was your decision back then. The decision worked quite well (kudos!), and I'm just surprised there's a need to redesign. It certainly ain't broken.
But no, this is very much something I don't want to re-litigate or re-discuss.

Taking a step back, the main feature of assertions is that they are stable and "boring", so that when porting a library or writing a new library, the author can concentrate on the functionality and rely on assertions just doing the right thing. What exactly is the right thing can be open to discussions of course, and we had way too many of those in scipy at least (I'm sure elsewhere too), which is why I thought they just work, and there's no need to rewrite then stabilize them, again.

As a data point, I'm currently working on adding array api support to scikit-image. The first thing I did was to drop the scipy assertions in-tree, with an expectation that they'll make their way to xpx and I'll be able to just replace them by from xpx.testing import ..... Apparently not without changes; unfortunate but well, okay.

So from this perspective, what's most important for porting a new library:

• strictness and knobs to turn some checks off. Strictness, accepting lists vs array-likes vs arrays only, check_* keywords etc --- this is something a library can and will customize; what's needed downstream is a way to customize the behavior without copy-pasting the whole thing, with just small wrappers on top of xpx assertions.
• default_xp: this one is large and extremely useful.
• xp_capabilities should probably best decoupled from core assertions.

Taking a step back, the main feature of assertions is that they are stable and "boring", so that when porting a library or writing a new library, the author can concentrate on the functionality and rely on assertions just doing the right thing. What exactly is the right thing can be open to discussions of course, and we had way too many of those in scipy at least (I'm sure elsewhere too), which is why I thought they just work, and there's no need to rewrite then stabilize them, again.

Yeah, part of the motivation for avoiding use of native assertions is for this kind of stability (or at least consistency) — we don't want to have to deal with bugs or nuances of various different xp.testing libraries if we can achieve it all via np.testing anyway.

I think you have a good argument if the question was "should we put in a bunch of thought to re-write these now?" But really, that work has already been done by Guido and Matt, so I think we are right to make use of it (presuming, fairly, that that work is of good quality).

Apparently not without changes; unfortunate but well, okay.

Let's check what changes would be required before we finalise the public API! Would you be willing to try these out in scikit-image and report any difficulties? I already have my eyes on wilsonrljr/sysidentpy#232 and glass-dev/glass#1076 too.

• default_xp: this one is large and extremely useful.

I think this may be the key remaining area of work.

strictness and knobs to turn some checks off. Strictness, accepting lists vs array-likes vs arrays only, check_* keywords etc --- this is something a library can and will customize; what's needed downstream is a way to customize the behavior without copy-pasting the whole thing, with just small wrappers on top of xpx assertions.

default_xp: this one is large and extremely useful.

xp_capabilities should probably best decoupled from core assertions.

I think these are useful points. Numpy also has a strict parameter; if set to True, it checks for dtypes, scalars, and shape. It could be useful for libraries that want to rely on this parameter. Can we open a tracking issue in array_api_extra for these improvements?

Numpy also has a strict parameter; if set to True, it checks for dtypes, scalars, and shape. It could be useful for libraries that want to rely on this parameter.

I would wait until it is explicitly requested, the finer grained parameters we have already are better IMO.

the finer grained parameters we have already are better IMO.

True.

All checks are passing now!

nice, I think #25143 (comment) is the next step

Only assert_array_almost_equal_nulp remains as a non-array_api_extra testing utility here. Can we port it too? If so, we can remove _strict_check and _assert_matching_namespace helper functions entirely from here.

Only assert_array_almost_equal_nulp remains as a non-array_api_extra testing utility here. Can we port it too? If so, we can remove _strict_check and _assert_matching_namespace helper functions entirely from here.

Let's leave it for now, but yes let's return to it when we bring this PR out of draft.