Thanks to visit codestin.com
Credit goes to github.com

Skip to content

ENH: Automatic copy on overlap #6272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
3 of 5 tasks
pv opened this issue Aug 31, 2015 · 11 comments
Closed
3 of 5 tasks

ENH: Automatic copy on overlap #6272

pv opened this issue Aug 31, 2015 · 11 comments
Labels
17 - Task Project Possible project, may require specific skills and long commitment

Comments

@pv
Copy link
Member

pv commented Aug 31, 2015

Tracking issue as a continuation to gh-6166

Some initial code, master...pv:copy-overlap

@charris
Copy link
Member

charris commented Jul 13, 2017

@seberg @pv I was under the impression this was pretty much done for 1.13. Is that the case?

@pv
Copy link
Member Author

pv commented Jul 13, 2017

It's done for (g)ufuncs (and thus also for in-place binops and most of numpy.linalg), and dot. There however may still be some problematic code paths in other operations, such as in indexed assignment. I did not make an exhaustive check on what's left.

@seberg
Copy link
Member

seberg commented Jul 13, 2017

It is true, it might be only smaller things left, had not thought about it much (in general things with an out argument are in danger that are not a ufunc):

  • copyto (should be identical to non-advanced indexing assignments)
  • median/percentile (use _ureduce python function which could possible get the treatment)
  • np.choose seems to have an out argument, so in principel can overlap.
  • np.take can overlap with out
  • The weird indexing likes (which I would avoid personally but...) put, putmask, place.
  • np.around
  • (possibly apply_(along|over)_axis)
  • and probably a few more, looked around a bit but not very systematically

@charris charris changed the title Automatic copy on overlap ENH: Automatic copy on overlap Jul 14, 2017
@ahaldane
Copy link
Member

Reviewing this as part of my 10 random issues, as requested in a 3/20/19 request.

I am 95% sure this is done and just forgotten to be closed, because in 2019 I have seen these overlap checks in the assignment code, including in code involving assignment to slices of an array.

I'm going to go ahead and close this, please reopen if that's wrong.

@matthew-brett
Copy link
Contributor

Is this really safe to close? Has anyone checked the list above?

@mattip
Copy link
Member

mattip commented Mar 23, 2019

I think the only unresolved case is when an array has overlap due to striding tricks. PR #12609 tries to make arrays with 0 strides read-only, in response to issue #2705.

@matthew-brett
Copy link
Contributor

But specifically - has someone gone through that list above and checked whether they are resolved or not? It seems to me that this issue should stay open until someone has done that.

@ahaldane
Copy link
Member

I'll go through it now...

@seberg
Copy link
Member

seberg commented Mar 23, 2019

Indexing assignments do use it, advanced indexing does not. I doubt a lot of the other functions I listed above never got treatment.

@ahaldane
Copy link
Member

ahaldane commented Mar 23, 2019

These seem OK:

  • copyto is OK since it goes through PyArray_AssignArray, which has an overlap check.
  • _ureduce code (eg, median/percentile) looks OK since it is pure python and delegates to ufuncs
  • around is ok because it delegates to ufuncs

Problematic:

  • np.take There is no overlap detection. Demonstration: np.take(x, [1,2,3], out=x[2:5], mode='wrap') -> array([1., 1., 1.]).
  • np.choose no overlap detection. Demonstration: np.choose([0,0,0], [x[:3], x[:3], x[:3]], out=x[1:4], mode='wrap') -> array([1, 1, 1])
  • I didn't carefully check, but I expect put and others in that file are the same.

Note that for both of these, with mode='raise', which is the default, a copy of out is always made to avoid clobbering out if an error occurs during processing, so there will be no risk of overlap. (incidentally, this seems unexpectedly inefficient for most people!).

So if we think those last ones should be fixed, we should keep this open.

@mattip
Copy link
Member

mattip commented Aug 18, 2019

Closing, #13182 covered the last outstanding cases. Please reopen if I misunderstood

@mattip mattip closed this as completed Aug 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
17 - Task Project Possible project, may require specific skills and long commitment
Projects
None yet
Development

No branches or pull requests

6 participants