There is similar discussion at #2705.

So across at least two machines running 1.9.1 this bug occurs for 91x91 arrays but not for 90x90 arrays. I'm not sure what significance that holds.

https://gist.github.com/PhilReinhold/5327001ac17f923942ad#file-test_transpose-py

The default buffer size is 8192 items, 90x90 is 8100, 91x91 is 8281. You should actually see that the first wrong value is the third entry of the last row, I.e. the 8193rd entry.

@PhilReinhold This is a design issue, not really a bug that is realistically fixable as far as I understand. At best, some warning could be added, and I'm not sure how difficult that would be. Python allows x += y to mean something different than x = x + y, and numpy uses this fact to do certain things more efficiently, but it can also be confusing or even just bad.

In order to detect this case and issue a warning or whatever, we would need
an implementation of the following operation: given two arrays with base
pointer, strides, and shape, do there exist two index tuples idx1, idx2
such that arr1[idx1] points to the same memory as arr2[idx2].
.
Unfortunately this problem is np complete.
.
We do have a heuristic implementation, np.may_share_memory, which never
claims that two overlapping arrays are non-overlapping, but might make
errors the other way and say that two non-overlapping arrays do overlap.
.
Unfortunately, in order to issue a warning in this case, we need the
opposite function, np.definitely_share_memory, because it's OK to miss a
warning sometimes but not OK to issue warnings on perfectly safe code.
.
If someone contributes a reasonable implementation of
definitely_share_memory then we'll do this; if not, not.
On Jun 19, 2015 1:15 PM, "argriffing" [email protected] wrote:

@PhilReinhold https://github.com/PhilReinhold This is a design issue,
not really a bug that is realistically fixable as far as I understand. At
best, some warning could be added, and I'm not sure how difficult that
would be. Python allows x += y to mean something different than x = x + y,
and numpy uses this fact to do certain things more efficiently, but it can
also be confusing or even just bad.

—
Reply to this email directly or view it on GitHub
#5241 (comment).

Would it be reasonable to back off from x += y to something like x = x + y if the types of x and y are well understood and if something is fishy with memory (e.g. potentially shared memory)? Or would the semantics be different enough to make this not feasible? Or would the loss of efficiency not be acceptable?

Numpy's general philosophy is to provide as much safety as possible without
compromising on speed. Since we can't precisely define "fishy", backing off
to out-of-place automatically like this would inevitably penalize people
who had carefully written safe and efficient code. Plus backing off to
out-of-place is pretty dubious anyway; users expect that if a and b are
views, then a += c should result in changes to b. Raising the error and
having the user write what they mean explicitly ("x = x + y") would be
better than silently backing off.
On Jun 19, 2015 1:49 PM, "argriffing" [email protected] wrote:

Would it be reasonable to back off from x += y to something like x = x + y
if the types of x and y are well understood and if something is fishy
with memory (e.g. potentially shared memory)? Or would the semantics be
different enough to make this not feasible? Or would the loss of efficiency
not be acceptable?

—
Reply to this email directly or view it on GitHub
#5241 (comment).

Should this issue be closed as 'not a bug' or 'wont fix' or something to that effect?

I doubt there's very much code in existence that's both safe and efficient here, given that this depends on Numpy implementation details that we change without notice. For example:

>>> import numpy as np; x = np.arange(10*10).reshape(10,10); x += x.T; print(np.__version__, abs(x-x.T).max())
('1.5.1', 98)
>>> import numpy as np; x = np.arange(10*10).reshape(10,10); x += x.T; print(np.__version__, abs(x-x.T).max())
('1.6.2', 0)

I would say the behavior is undefined. The cases where the traversal order is simple enough to analyze to write correct code for are probably mostly 1D. I do not remember seeing many proposed use cases.

Whether this should be a generic ufunc thing or just for binary operations and assignment, just a warning that things look dangerous or something more is then a more hairy question. The previous suggestion (gh-1683) was to make copies of RHS when it's not clear the operation is safe.

The problem is that even trivial safe cases like 'arr[:, 0] += arr[:, 1]'
can trigger a may_share_memory check, and if we start copying every time
this happens then this will cause unpredictable slowdowns and increased
memory use in situations that are already well defined and work correctly.
Similarly if we start warning every time may_share_memory returns true,
then this will create spurious warnings (which are themselves slow, even if
filtered out) in situations that are already well defined and work
correctly.
.
What we need here is a definitely_share_memory function. (Though this is
hard to make reliable! Even a partially working one could be enough to
start warning though.)
On Jul 27, 2015 15:41, "Pauli Virtanen" [email protected] wrote:

I doubt there's very much code in existence that's both safe and efficient
here, given that this depends on Numpy implementation details that we
change without notice. For example:

import numpy as np; x = np.arange(10_10).reshape(10,10); x += x.T; print(np.version, abs(x-x.T).max())
('1.5.1', 98)
import numpy as np; x = np.arange(10_10).reshape(10,10); x += x.T; print(np.version, abs(x-x.T).max())
('1.6.2', 0)

I would say the behavior is undefined https://xkcd.com/1172/. The cases
where the traversal order is simple enough to analyze to write correct code
for are probably mostly 1D.

Whether this should be a generic ufunc thing or just for binary operations
and assignment, or just a warning that things look dangerous is then a more
hairy question. The previous suggestion (gh-1683
#1683) was to make copies of RHS
when it's not clear the operation is safe.

—
Reply to this email directly or view it on GitHub
#5241 (comment).

'arr[:, 0] += arr[:, 1]'

Remember when gcc reversed the traversal order of memcpy?

Remember when gcc reversed the traversal order of memcpy?

But that wouldn't affect this example in which arr[:, 0] and arr[:, 1] don't actually share memory, right?

duplicate gh-1683