API: Cleaning `numpy/init.py` and main namespace - Part 1 [NEP 52] #24316

mtsokol · 2023-08-02T14:56:16Z

Here I share a draft PR connected to issue #24306. It mostly covers restructuring of numpy/__init__.py file.

In a nutshell:

Every item of the main namespace is imported explicitly from predefined API list (the list is still being discussed in the related issue) rather than implicitly with * import.
The file _main_namespace_definition.py is meant to be a contract of the main namespace and is used for defining __dir__ and __all__ attributes of NumPy's top namespace. Therefore it is versioned and can't be altered without modifying file in question.
Removing from .core import * and from .lib import * uncovered some cyclic dependencies in the codebase (for now I explicitly imported these names that are used internally with np.* causing a cycle), but ideally there should be none of them.
I refactored some parts of __init__.py file that I thought were obsolete.
I think it's easier to review modified __init__.py as continuous file, rather than a diff.

Please share your feedback!

rgommers · 2023-08-02T20:01:07Z

Thanks @mtsokol, this is a useful thing to tackle now. I'm thinking we may want to identify the parts tht can be merged straight away and do this in a few different PRs; the _main_namespace_definition.py seems like it'll stay WIP for a while, while some other parts are quite straightforward. I'm thinking first PR the obvious cleanups (I can add review comments on which ones are mergeable now), and a second one only doing import * removals. WDYT?

mtsokol · 2023-08-02T20:12:01Z

@rgommers, works for me! Please comment these items - I will work on it tomorrow. Then I guess first PR will be about "cleanup __init__.py", by removing outdated items and of course import *. Then a separate PR will be to introduce this main namespace contract.

rgommers

@mtsokol I added the comments regarding what can be merged in a first PR. That should make the diff here a lot smaller.

Also note that if you push updates to this PR, it's probably preferable to add [skip ci] in the commit message - no need for a full battery of CI here yet.

numpy/__init__.py

mtsokol · 2023-08-03T12:05:13Z

@mtsokol I added the comments regarding what can be merged in a first PR. That should make the diff here a lot smaller.

Also note that if you push updates to this PR, it's probably preferable to add [skip ci] in the commit message - no need for a full battery of CI here yet.

@rgommers it works for me! Then this PR will be for the first batch of changes (general cleaning of numpy/__init__.py and removing NumPy's warnings and exceptions from the main namespace).

The second, a separate PR, will cover solving cyclic dependencies and getting rid of from ... import *, then the third one will introduce a separate file/contract for explicit definition of the main namespace (defining globals(), __all__ and __dir__ this way).

I'm running CI here because I prepared first batch of changes (also, I'm working on reflecting them in other libraries).

mtsokol · 2023-08-03T12:38:52Z

@rgommers Looks that there's still a RankWarning class that needs to be removed from top-level __init__.pyi. It's originally from numpy.polynomial.polyutils.py, in my opinion it's domain specific to polynomials, so it doesn't need to be moved to numpy.exceptions. WDYT?

rgommers · 2023-08-03T13:33:55Z

@rgommers Looks that there's still a RankWarning class that needs to be removed from top-level __init__.pyi. It's originally from numpy.polynomial.polyutils.py, in my opinion it's domain specific to polynomials, so it doesn't need to be moved to numpy.exceptions. WDYT?

That's not completely clear cut (right now it goes with np.polyfit), so I'd not touch it here and add it to your "tentative" list.

mtsokol · 2023-08-03T15:04:09Z

I think it's ready for a review: In files where an exception/warning was used only once I used np.exceptions.<>. In cases where it was used multiple times I added an explicit import.

rgommers

This looks great, thanks Mateusz! And thank you for the due diligence and fixing things in Matplotlib, Pandas, SciPy, scikit-learn and JAX.

The list of differences between the np.__dir__() output on this PR vs. 1.25.0 is:

{'ERR_CALL',
 'ERR_DEFAULT',
 'ERR_IGNORE',
 'ERR_LOG',
 'ERR_PRINT',
 'ERR_RAISE',
 'ERR_WARN',
 'SHIFT_DIVIDEBYZERO',
 'SHIFT_INVALID',
 'SHIFT_OVERFLOW',
 'SHIFT_UNDERFLOW',
 '__deprecated_attrs__',
 '__expired_functions__',
 '_builtins',
 '_financial_names',
 '_using_numpy2_behavior',
 'cast',
 'compat',
 'fastCopyAndTranspose',
 'geterrobj',
 'kernel_version',
 'lookfor',
 'numarray',
 'oldnumeric',
 'set_numeric_ops',
 'seterrobj',
 'source'}

all those things have indeed been removed, so this looks good.

There's nothing in here that should be controversial, so let's get it in to keep the ball rolling.

rgommers · 2023-08-07T19:38:16Z

numpy/__init__.py

-    # but do not use them, we define them here for backward compatibility.
-    oldnumeric = 'removed'
-    numarray = 'removed'
-
    def __getattr__(attr):


For a next PR: copying the pattern from scipy/__init__.py for __getattr__ to import all submodules in a lazy way rather than only numpy.testing would be useful.

Sure! And as we discussed, this will help fixing cyclic dependencies.

seberg · 2023-08-08T07:46:48Z

Just a note, the effective change here was that previously ComplexWarning and some other errors were available as np.ComplexWarning but hidden because we wanted to move to np.exceptions.ComplexWarning. This finalizes the move without a deprecation.
I can see that mostly being relevant for larger libraries who we can expect to deal with it, but if anyone thinks that a wider range of users have such code, I would be fine with keeping the "hidden" status also for a bit longer.

mtsokol · 2023-08-08T08:27:32Z

Just a note, the effective change here was that previously ComplexWarning and some other errors were available as np.ComplexWarning but hidden because we wanted to move to np.exceptions.ComplexWarning. This finalizes the move without a deprecation. I can see that mostly being relevant for larger libraries who we can expect to deal with it, but if anyone thinks that a wider range of users have such code, I would be fine with keeping the "hidden" status also for a bit longer.

I can add a custom message about these warnings/exceptions when accessing them from the main namespace (same as __expired_functions__ worked in __init__.py).

rgommers · 2023-08-08T08:40:55Z

I can add a custom message about these warnings/exceptions when accessing them from the main namespace (same as __expired_functions__ worked in __init__.py).

I'd prefer not to do that for now - at least not until/unless we start seeing a real need. The change is trivial and should be easy to find in case one runs into it. If we are going to add messages for all changes, we will again end up with hundreds of lines of cruft in __init__.py that are going to hang around there for years.

We already planned to have a single doc page with all these changes for 2.0; no need to do double work here. Everyone who uses nightlies can easily deal with this.

seberg · 2023-08-08T08:44:00Z

Right agreed. My concern is currently only about the sum of changes being overwhelming. For some things we have good reasons to do so because they are things that nobody understands or every dev understands that their logic is flawed. For these, they are a bit fuzzy to me: it is basically a file with "legacy aliases" that is just a long list we would keep long enough that users can adopt it without a try/except or if numpy_version >.

The branching is the real reason here, as we have said many times asking users to change things twice isn't great. I don't think this matters for larger libraries, they are used to it, but it does matter for scripts/small libraries.

seberg · 2023-08-08T08:46:42Z

In other words, the reason I am fine with it unless someone disagrees, is that I think for the users that we should care about (those who are not used to adding such branching), my guess is that there are very few who will notice the changes.

rgommers · 2023-08-08T08:54:22Z

Exactly, I agree with the "no branching for the average user" rule. These are examples I think that are well below the line of usage frequency. Things like widely-used aliases (e.g., absolute as alias of abs) are used enough that we should keep it as a hidden aliases. And the line is somewhere in the middle between those.

API: Start main namespace overhaul

b1d9c33

mtsokol marked this pull request as draft August 2, 2023 15:46

rgommers added the 62 - Python API Changes or additions to the Python API. Mailing list should usually be notified. label Aug 2, 2023

API: Consider sized aliases separately

1e1b754

mtsokol force-pushed the overhaul-of-main-namespace branch from b970255 to 1e1b754 Compare August 3, 2023 09:30

rgommers reviewed Aug 3, 2023

View reviewed changes

API: Start cleanup of numpy/__init__.py

c55bee6

mtsokol force-pushed the overhaul-of-main-namespace branch from bab34f5 to c55bee6 Compare August 3, 2023 12:08

mtsokol marked this pull request as ready for review August 3, 2023 12:10

API: Fix RankWarning typing test

775b5dd

mtsokol force-pushed the overhaul-of-main-namespace branch from 53b8a39 to 775b5dd Compare August 3, 2023 14:18

mtsokol changed the title ~~[WIP] API: Overhaul of NumPy main namespace [NEP 52]~~ API: Cleaning numpy/__init__.py and main namespace - Part 1 [NEP 52] Aug 3, 2023

mtsokol requested a review from rgommers August 3, 2023 15:04

rgommers approved these changes Aug 7, 2023

View reviewed changes

rgommers merged commit c8e2343 into numpy:main Aug 7, 2023

rgommers added this to the 2.0.0 release milestone Aug 7, 2023

rgommers added the 03 - Maintenance label Aug 7, 2023

mtsokol deleted the overhaul-of-main-namespace branch August 7, 2023 20:20

mtsokol mentioned this pull request Aug 8, 2023

Removed numpy exceptions imports from the main namespace [NEP 52 - Part 1] cupy/cupy#7796

Closed

eendebakpt mentioned this pull request Aug 14, 2023

[BUG] Return value of use_hugepage in hugepage_setup #24412

Merged

mtsokol mentioned this pull request Aug 23, 2023

DOC: Add missing changelogs for NEP 52 PRs #24510

Merged

ngoldbaum added the Numpy 2.0 API Changes label Aug 24, 2023

Uh oh!

API: Cleaning numpy/__init__.py and main namespace - Part 1 [NEP 52] #24316

API: Cleaning numpy/__init__.py and main namespace - Part 1 [NEP 52] #24316

Uh oh!

Conversation

mtsokol commented Aug 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgommers commented Aug 2, 2023

Uh oh!

mtsokol commented Aug 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mtsokol commented Aug 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mtsokol commented Aug 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgommers commented Aug 3, 2023

Uh oh!

mtsokol commented Aug 3, 2023

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

rgommers Aug 7, 2023

Choose a reason for hiding this comment

Uh oh!

mtsokol Aug 7, 2023

Choose a reason for hiding this comment

Uh oh!

seberg commented Aug 8, 2023

Uh oh!

mtsokol commented Aug 8, 2023

Uh oh!

rgommers commented Aug 8, 2023

Uh oh!

seberg commented Aug 8, 2023

Uh oh!

seberg commented Aug 8, 2023

Uh oh!

rgommers commented Aug 8, 2023

Uh oh!

Uh oh!

API: Cleaning `numpy/init.py` and main namespace - Part 1 [NEP 52] #24316

API: Cleaning `numpy/init.py` and main namespace - Part 1 [NEP 52] #24316

mtsokol commented Aug 2, 2023 •

edited

Loading

mtsokol commented Aug 2, 2023 •

edited

Loading

mtsokol commented Aug 3, 2023 •

edited

Loading

mtsokol commented Aug 3, 2023 •

edited

Loading