Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Consider reverting the renaming of compiled c test modules (part of the pytest move)? #10845

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
lesteve opened this issue Apr 4, 2018 · 20 comments · Fixed by #10850
Closed

Comments

@lesteve
Copy link
Contributor

lesteve commented Apr 4, 2018

The renaming of compiled c test modules in #10790 broke the scikit-learn tests because we were using numpy.core.umath_tests.inner1d. There was no good reason to use inner1d and we now used standard numpy constructs in scikit-learn master. The problem is that it will affect the latest scikit-learn (0.19.1) as soon as numpy 1.15 is released. Given the current rate of release, it is rather likely that numpy 1.15 will be released before the next scikit-learn release (0.20).

It feels like other projects could be using some of the renamed modules, so maybe the safest approach would be to keep the old modules around for a deprecation cycle with a DeprecationWarning saying that this modules are for numpy internal use only. Would you be willing to consider this option?

I first commented in #10790 (comment) but @charris suggested I opened a new issue, so here it is!

@rth
Copy link
Contributor

rth commented Apr 4, 2018

The problem is that it will affect the latest scikit-learn (0.19.1) as soon as numpy 1.15 is released.

By affect, @lesteve means that it will raise an error if one imports anything from sklearn.ensemble (cf. scikit-learn/scikit-learn#10880 (comment) ) which will break existing code for a number of scikit-learn users...

@pv
Copy link
Member

pv commented Apr 4, 2018 via email

@charris charris added this to the 1.15.0 release milestone Apr 4, 2018
@charris
Copy link
Member

charris commented Apr 4, 2018

I'll put a shim in for this as suggested by @pv, also change the warnings in the numpy.testing shims to DeprecationWarning. For the other name changes, I will hold off on shims until there is a reported problem. An error gets noticed ...

@lesteve
Copy link
Contributor Author

lesteve commented Apr 4, 2018

This seems fine as far as scikit-learn is concerned, thanks a lot for tackling this!

@mhvk
Copy link
Contributor

mhvk commented Apr 4, 2018

Hmm, I've actually been using those routines as the only functioning examples of gufuncs, and thus useful as test cases. Is it just a sense of "this is private, should have an _" that drives the change, or is there more to it?

@charris
Copy link
Member

charris commented Apr 4, 2018

Is it just a sense of "this is private, should have an _" that drives the change

Yes. I note that many of the linalg functions are using gufuncs, what exactly do you need for test cases?

@mhvk
Copy link
Contributor

mhvk commented Apr 4, 2018

Functions that are simple and can be easily used to check __array_ufunc__ works... But I don't have a strong objection to making it _umath_tests; I just don't see a strong need, since most people should never look under core anyway (that name already makes obvious that whatever is important will be imported a level up).

@charris
Copy link
Member

charris commented Apr 4, 2018

since most people should never look under core anyway

Just so. And if it was obvious we wouldn't be having this problem :) Where are you using them for test cases? NumPy using them is not a problem.

@mhvk
Copy link
Contributor

mhvk commented Apr 4, 2018

I'm not using them on anything public - my comment was mainly to support @lesteve's note that there may well be more code that expects to find these functions, and one should wonder whether the arguably minor costs are worth what seems a very minor benefit of the name change. (But since it is all about very minor things, and it doesn't really affect me, just decide what you think best!)

@charris
Copy link
Member

charris commented Apr 4, 2018

It's not a minor cost. If we change or rearrange numpy internals and people complain, that is a major cost. And we will be doing a lot of rearrangement going forward. If you have tests for __array_ufunc__ shouldn't they be in NumPy? Or are they already.

@mhvk
Copy link
Contributor

mhvk commented Apr 4, 2018

Yes, there are tests for __array_ufunc__ that use these routines in numpy (but as you say, those are not a problem, we can just adjust them). I don't have anything for which the name change is an issue myself; I really am just quite aware of their existence as simple examples of gufunc.

Note that I am probably missing the big picture here about the larger re-arrangements that are planned.

@charris
Copy link
Member

charris commented Apr 4, 2018

Ah, OK. Those tests were already adjusted. We will be merging the umath and multiarray modules. In general, I think we are heading to fewer modules in numpy.core.

@mhvk
Copy link
Contributor

mhvk commented Apr 4, 2018

Yes, of course! That overall plan definitely sounds good!

@charris
Copy link
Member

charris commented Apr 4, 2018

See #10850.

charris added a commit to charris/numpy that referenced this issue Apr 7, 2018
The following modules have been moved or renamed and should not be
imported. This adds shim modules for the old names that issue a
DeprecationWarning on import.

* numpy/core/umath_tests.py
* numpy/testing/decorators.py
* numpy/testing/noseclasses.py
* numpy/testing/nosetester.py
* numpy/testing/utils.py

Closes numpy#10845.
@hamogu
Copy link

hamogu commented Aug 31, 2018

I want to note that inner1d from numpy.core.umath_tests outperforms other, non-private numpy functions by a factor of 2 for row-wise dot products:
https://stackoverflow.com/questions/15616742/vectorized-way-of-calculating-row-wise-dot-product-two-matrices-with-scipy#answer-39657770

I'm working with a numpy-based ray-tracing code where row-wise dot products are by far the most common operation that I do and I've been starting to use numpy.core.umath_tests.inner1d for that reason; taking the risk that you move it. There are other related Stackoverflow questions (e.g. https://stackoverflow.com/questions/6229519/numpy-column-wise-dot-product or https://stackoverflow.com/questions/48066890/using-numpy-einsum-to-compute-inner-product-of-column-vectors-of-a-matrix) indicating some interest in a function like this.

Should I open a feature request to make inner1d part of the public API?

@charris
Copy link
Member

charris commented Aug 31, 2018

I think we will be adding several new ufuncs for these types of stacked arrays/vectors in the future, so I would prefer to wait on that.

@hamogu
Copy link

hamogu commented Aug 31, 2018

@charris : Thanks for the fast answer. Looking forward to seeing them!

@a-recknagel
Copy link

a-recknagel commented Sep 4, 2018

@charris Is there a specific reason why you gave the warnings a stacklevel of 2? I don't see it turning up in the discussion, and since it isn't called in a wrapper the need isn't obvious to me.

I noticed it because the stackframe-nesting leads to weird side-effects like pytest not being able to suppress them because it only checks the first layer.

Edit: Or should I open an issue for this?

@charris
Copy link
Member

charris commented Sep 4, 2018

We try to make the error messages come from the point of call. How are you suppressing the warnings in pytest.

You might open an issue for discussion.

@a-recknagel
Copy link

a-recknagel commented Sep 5, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants