API: Enforce one copy for `array` when `copy=True` #26215

mtsokol · 2024-04-05T11:08:41Z

Addresses: #26208

This PR enables passing copy=True to def __array__(...) and prevents NumPy from making another copy once it can assume that copy=True was successfully passed and the copy was made by the implementor.

Following #26208 (comment) I implemented it with a copy_indicator that is set to 1, which later is used to decide whether NPY_ARRAY_ENSURECOPY flag can be removed.

I added a test test___array__copy_once that checks if only one copy is made with np.array(my_arr, copy=True). Without copy_indicator the test fails as np.array(my_arr, copy=True) doubles delta RSS (I used psutil package and added it as another test dependency).

I assumed that we want to prevent making implicit copies with __array__. Therefore the user is also responsible for properly implementing dtype argument.

class MyArray:
    def __array__(self, dtype=None, copy=None):
        return np.array([1,2,3], dtype=int)

my_arr = MyArray()
np.array(my_arr, dtype=float)  # Error! Produced ndarray has incorrect dtype! `__array__` should use dtype arg

seberg

Just two comments for now. But yeah, we may need the copy indicator unfortunately (maybe call it was_copied, though? It has to be passed by reference anyway, so it is obvious for the caller that it is an output).

It would be good to have a test that we don't pass copy=True if someone calls np.array([array_like]) (i.e. there is a list).

seberg · 2024-04-08T14:37:07Z

numpy/_core/tests/test_multiarray.py

+                else:
+                    return base_arr
+
+        process = psutil.Process()


Can you just do this without psutils: Just store copy you return, then compare do an identity/base check. Easier, and no need for psutil magic.

Ah, right - I removed psutils and stored copies/base arrays instead.

seberg · 2024-04-08T14:38:10Z

numpy/_core/src/multiarray/array_coercion.c

@@ -970,7 +972,7 @@ PyArray_DiscoverDTypeAndShape_Recursive(
        npy_intp out_shape[NPY_MAXDIMS],
        coercion_cache_obj ***coercion_cache_tail_ptr,
        PyArray_DTypeMeta *fixed_DType, enum _dtype_discovery_flags *flags,
-        int copy)
+        int copy, int *copy_indicator)


This already has the _dtype_discovery_flags I think it makes sense to use those "internally" to this part of the code.

Sure! I merged copy_indicator into _dtype_discovery_flags.

mtsokol · 2024-04-08T18:24:18Z

(maybe call it was_copied, though?

@seberg Sure - renamed!

It would be good to have a test that we don't pass copy=True if someone calls np.array([array_like]) (i.e. there is a list).

Sure! I changed implementation so that sequences don't pass copy=True to it's elements and added a test for that.

mtsokol · 2024-04-10T08:37:46Z

I did one more rebase to solve conflicts.

ngoldbaum

One request for clarification inline and one more general suggestion.

I think was_copied and COPY_WAS_CREATED should be renamed was_copied_by__array__ and COPY_WAS_CREATED_BY__ARRAY__ to emphasize that this is specifically to handle subtleties around __array__ implementations. Alternative names or just comments around where they are used explaining that it has something to do with __array__ would help readers trying to understand why we have both an ensure copy flag and a flag tracking if a copy already happened.

numpy/_core/tests/test_multiarray.py

charris · 2024-04-10T14:59:10Z

Does this need a backport?

ngoldbaum · 2024-04-10T15:16:58Z

Does this need a backport?

No this change is targeted for 2.1

mtsokol · 2024-04-10T15:18:47Z

I think was_copied and COPY_WAS_CREATED should be renamed was_copied_by__array__ and COPY_WAS_CREATED_BY__ARRAY__ to emphasize that this is specifically to handle subtleties around __array__ implementations.

These names are clearer and more specific - done!

seberg

LGTM, I suppose we could use the argument in PyArray_DiscoverDTypeAndShape for other things in principle, although I am not sure there is anything concrete, so let's use the name, it is much nicer everywhere else (and all other places will go away when the deprecation goes away).

The ENSURE_COPY flag setting logic looks incorrect, though (I admit, I didn't try it).

seberg · 2024-04-11T08:26:50Z

numpy/_core/src/multiarray/ctors.c

-        /* TODO: As of NumPy 2.0 this path is only reachable by C-API. */
-        Py_SETREF(new, PyArray_NewCopy((PyArrayObject *)new, NPY_KEEPORDER));
+    if (was_copied_by__array__ != NULL && copy == 1 &&
+        must_copy_but_copy_kwarg_unimplemented == 0) {


Could add a comment to remove the was_copied_by__array__ argument again here, but it is probably clear enough (when it happens).

seberg · 2024-04-11T08:33:12Z

numpy/_core/src/multiarray/ctors.c

@@ -1615,6 +1622,10 @@ PyArray_FromAny_int(PyObject *op, PyArray_Descr *in_descr,
        assert(cache->converted_obj == op);
        arr = (PyArrayObject *)(cache->arr_or_sequence);
        /* we may need to cast or assert flags (e.g. copy) */
+        if (was_copied_by__array__ == 1 && flags & NPY_ARRAY_ENSURECOPY) {


Suggested change

if (was_copied_by__array__ == 1 && flags & NPY_ARRAY_ENSURECOPY) {

if (was_copied_by__array__ == 1) {

You don't need to test this, it is OK to unset the flag if it isn't set. (to me that is clearer, but maybe it's me)

seberg · 2024-04-11T08:35:54Z

numpy/_core/src/multiarray/ctors.c

@@ -1615,6 +1622,10 @@ PyArray_FromAny_int(PyObject *op, PyArray_Descr *in_descr,
        assert(cache->converted_obj == op);
        arr = (PyArrayObject *)(cache->arr_or_sequence);
        /* we may need to cast or assert flags (e.g. copy) */
+        if (was_copied_by__array__ == 1 && flags & NPY_ARRAY_ENSURECOPY) {
+            flags = flags & ~NPY_ARRAY_ENSURECOPY;
+            flags = flags | NPY_ARRAY_ENSURENOCOPY;


This looks wrong, so please add a test:

np.asarray(array_like, copy=True, order="F")

should just make two copies. You probably want to make sure that users honors the dtype that was passed. That might be nice, but if it is not very clean to test for, I am not sure we should worry much.

Ah right, a copy still needs to be made to convert to order="F". Then I shouldn't add ENSURENOCOPY flag but only remove ENSURECOPY one.

About honoring dtype: I can check if arr in cache has the same dtype as requested dtype variable. I wonder if it's sufficient for comparing dtypes in C-API:

PyArray_TYPE(arr) == dtype->type_num

No, unfortunately checking that is not really trivial (the only way to do it, is the way that the function uses later to see if it has to make a copy).

I just realized, it is even more complicated, for subarray dtypes (maybe those should just never be passed on). But that is probably not really new, although more pronounced by nudging downstream to actually implement dtype.

Maybe the solution is to just unset the dtype when a copy was made, although unfortunately that doesn't help us in the path when no copy was made and dtype is used by the __array__ implementor.

(Eplenation: if you write dtype=(4,)i you get back an array with an additional dimension and dtype=i. But if __array__ does that, then you end up doing another cast!)

Then let's maybe skip checking dtype after __array__ call?
What we currently have is that:

np.array(array_like, dtype=new_dtype, copy=False)

errors when a dtype wasn't honored in __array__ implementation (which is nice I think) and

np.arrray(array_like, dtype=new_dtype)

get's another copy if dtype wasn't honored, same as order="F" case.

Then let's maybe skip checking dtype after array call?
For here, that is perfectly fine to me, it is just a mild performance nudge for downstream after all.

However, maybe we should make a new issue. I am slightly worried that downstream until now always just ignored dtype=, and now we will get new bugs with subarray dtypes.

OTOH, I will also say that, I don't expect anyone to explicitly run into it, so in a sense, it is an unrelated bug fix.

I updated flags logic here and added a test with:

np.asarray(array_like, copy=True, order="F")

ngoldbaum

I think this is OK to merge now.

I don't think this is going to cause much additional churn over adding the copy keyword in the first place, hopefully downstream did the right thing.

mtsokol · 2024-04-19T14:14:54Z

Ok, merging it now.

mtsokol added the 01 - Enhancement label Apr 5, 2024

mtsokol self-assigned this Apr 5, 2024

github-actions bot added the 30 - API label Apr 5, 2024

mtsokol force-pushed the one-copy-__array__ branch from 592ae0a to 4f629bf Compare April 5, 2024 11:10

mtsokol linked an issue Apr 5, 2024 that may be closed by this pull request

ENH: pass copy=True to __array__ #26208

Closed

mtsokol force-pushed the one-copy-__array__ branch 6 times, most recently from f7ff1b7 to d654d93 Compare April 5, 2024 13:52

mtsokol added this to the 2.1.0 release milestone Apr 5, 2024

mtsokol force-pushed the one-copy-__array__ branch from d654d93 to 8c9f073 Compare April 5, 2024 14:31

seberg reviewed Apr 8, 2024

View reviewed changes

mtsokol force-pushed the one-copy-__array__ branch from 8c9f073 to 973ce28 Compare April 8, 2024 18:02

mtsokol force-pushed the one-copy-__array__ branch 2 times, most recently from c1a3d15 to 854340f Compare April 8, 2024 22:10

mtsokol requested a review from seberg April 8, 2024 22:14

mtsokol force-pushed the one-copy-__array__ branch from 854340f to 553c473 Compare April 10, 2024 08:36

ngoldbaum reviewed Apr 10, 2024

View reviewed changes

numpy/_core/tests/test_multiarray.py Outdated Show resolved Hide resolved

seberg reviewed Apr 11, 2024

View reviewed changes

mtsokol added 4 commits April 11, 2024 12:15

API: Enforce one copy for __array__ when copy=True

c8cdda2

Adjust was_copied variable names

ea983c6

Apply review comments

6c7b6f2

linting

3549902

mtsokol force-pushed the one-copy-__array__ branch from e6a3ce5 to 3549902 Compare April 11, 2024 10:16

ngoldbaum approved these changes Apr 17, 2024

View reviewed changes

charris changed the title ~~API: Enforce one copy for __array__ when copy=True~~ API: Enforce one copy for __array__ when copy=True Apr 17, 2024

mtsokol merged commit 1440eee into numpy:main Apr 19, 2024

mtsokol deleted the one-copy-__array__ branch April 19, 2024 14:15

This was referenced Apr 19, 2024

BLD: Accelerate wheels for macOS 14+ scipy/scipy#20510

Merged

TST: Adapt to __array__(copy=True) scipy/scipy#20533

Merged

larsevj mentioned this pull request Aug 23, 2024

Tests fails with numpy 2.1.0 equinor/resdata#1009

Closed

	if (was_copied_by__array__ == 1 && flags & NPY_ARRAY_ENSURECOPY) {
	if (was_copied_by__array__ == 1) {

Uh oh!

API: Enforce one copy for __array__ when copy=True #26215

API: Enforce one copy for __array__ when copy=True #26215

Uh oh!

Conversation

mtsokol commented Apr 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

seberg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtsokol commented Apr 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mtsokol commented Apr 10, 2024

Uh oh!

ngoldbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

charris commented Apr 10, 2024

Uh oh!

ngoldbaum commented Apr 10, 2024

Uh oh!

mtsokol commented Apr 10, 2024

Uh oh!

seberg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtsokol Apr 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngoldbaum left a comment

Choose a reason for hiding this comment

Uh oh!

mtsokol commented Apr 19, 2024

Uh oh!

Uh oh!

API: Enforce one copy for `array` when `copy=True` #26215

API: Enforce one copy for `array` when `copy=True` #26215

mtsokol commented Apr 5, 2024 •

edited

Loading

mtsokol commented Apr 8, 2024 •

edited

Loading

mtsokol Apr 11, 2024 •

edited

Loading