API: Rename `numpy/core` to `numpy/_core` [NEP 52] #24634

mtsokol · 2023-09-04T14:16:04Z

This PR introduces a breaking change of renaming numpy/core module to numpy/_core. The first commit renames the module, where following ones adjust the codebase to this change.

Some stages will fail for now (e.g. building docs, as matplotlib needs to have some changes introduced), but I checked all 2500 occurrences of core in the code 😄 and this PR should cover all of the relevant ones.

rgommers

Thanks @mtsokol! Wow, this turned out to be a large PR. A few initial comments. Let's see what is needed in downstream PRs to stop using numpy.core completely. I believe we have to put numpy/core/__init__.py back with a deprecation warning emitted when accessed, and a __getattr__ to forward imports to the relevant numpy._core items. Otherwise this will probably be the most disruptive change in the release.

numpy/tests/test_public_api.py

tools/refguide_check.py

numpy/lib/tests/test_format.py

ksunden · 2023-09-04T16:50:59Z

(e.g. building docs, as matplotlib needs to have some changes introduced)

It's not actually obvious to me why this is failing, we don't import np.core (directly) anywhere.

The failure is on import of a C-extension module, which does depend on the Numpy C API via:

#include <numpy/ndarrayobject.h>

and

#include "numpy/arrayobject.h"

None of our C source files refer to "core" directly, either.

So quite possibly it is "just" a matter of recompiling... but that is itself a potential problem if this affects the ability to have binaries compatible with different numpy versions. (keeping existing binaries forward-compatible would be ideal, if possible)

I do suspect that PyBind11 will have problems that need to be addressed (which may be prerequisites for even testing what, if anything, needs to be done for mpl specifically). A quick scan of the source there shows that they do refer to numpy.core in at least a handful of places. The particular c-extension that is failing is not a pybind11 module, but we do have some which are (and are slowly working to increase usage of pybind11).

ksunden · 2023-09-04T17:06:54Z

Pandas also appears to be affected (there are some tests which refer to np.core directly, but nothing in production code), but also from the doc build, a pandas Cython extension module is broken at cnp.import_array() (cnp just being the cimport numpy as cnp)

rgommers · 2023-09-05T13:04:07Z

So quite possibly it is "just" a matter of recompiling... but that is itself a potential problem if this affects the ability to have binaries compatible with different numpy versions. (keeping existing binaries forward-compatible would be ideal, if possible)

I think that that should be possible for this PR, and that's probably the number one prio to sort out here. However, I do want to point out that the plan is for NumPy 2.0 to break ABI compatibility - so keeping existing binaries forward-compatible during this release cycle is highly unlikely to happen. See the guidance in gh-24300.

ksunden · 2023-09-05T19:36:42Z

Hmmm... well that raises some infrastructure/CI questions during the transition... if matplotlib (and pandas) is required for numpy docs, but matplotlib needs to be rebuilt due to ABI changes. Perhaps as backwards incompatible changes roll in, you need to install from source to avoid a chicken/egg problem. (Though if/when changes are needed to pybind11/cython, etc, we may not escape the problem entirely)

Perhaps the way to do it for mpl is to pin to <2.0 on the release branches, but leave unpinned on main, so that makes it easier for devs/numpy CI to install with 2.0-compiled versions without having to edit the dependency metadata.

rgommers · 2023-09-05T19:51:46Z

but matplotlib needs to be rebuilt due to ABI changes. Perhaps as backwards incompatible changes roll in,

We won't be breaking things all the time - C API/ABI changes are being prepared behind a switch.

Perhaps the way to do it for mpl is to pin to <2.0 on the release branches, but leave unpinned on main, so that makes it easier for devs/numpy CI to install with 2.0-compiled versions without having to edit the dependency metadata.

Yes, that sounds exactly right. I just checked the latest Matplotlib release (3.7.2), and it uses numpy>=1.20 in its setup.py. That is going to break - you should start adding <2.0. I've tried to communicate this in as many places as possible (numpy mailing list, 2.0 tracking issue, https://numpy.org/devdocs/dev/depending_on_numpy.html, other issues and meetings), but it feels like no one has seen it. So it's time to get more proactive and find non-numpy venues (blogs/podcast/etc.) to flag this (as @mattip suggested recently).

mtsokol · 2023-09-07T09:31:11Z

(e.g. building docs, as matplotlib needs to have some changes introduced)

It's not actually obvious to me why this is failing, we don't import np.core (directly) anywhere. ...

@ksunden I managed to solve the issue - it was due to the fact that matplotlib used in docs is compiled against numpy with numpy.core which imports from core as a part of numpy C-API initialization. I added numpy.core shims as a proxy to numpy._core for a full backward compatibility.

rgommers · 2023-09-07T12:30:00Z

This LGTM:

>>> np.core.multiarray.concatenate
<ipython-input-4-423c639bb0c0>:1: DeprecationWarning: `numpy.core` has been made officially private and renamed to `numpy._core`. Accessing `_core` directly is discouraged, you should use a public module instead that exports the attribute in question. If you still would like to access an attribute from it, please use `np._core.multiarray`.
  np.core.multiarray.concatenate
<function concatenate at 0x7f5ecca17630>

>>> np.core._multiarray_umath.arange
<ipython-input-5-10613df9b3ac>:1: DeprecationWarning: `numpy.core` has been made officially private and renamed to `numpy._core`. Accessing `_core` directly is discouraged, you should use a public module instead that exports the attribute in question. If you still would like to access an attribute from it, please use `np._core._multiarray_umath`.
  np.core._multiarray_umath.arange
<built-in function arange>

Everything I tried still works, with a deprecation message appearing. The thing I'm wondering about too is if the reverse is going to keep working. If the one problem here was:

PyObject *numpy =  PyImport_ImportModule("numpy.core._multiarray_umath");

and now we rebuild MPL against a new numpy version where that extension is in _core, can the C API shim or numpy2_compat package from https://numpy.org/neps/nep-0053-c-abi-evolution.html fix that? If not, we may have special-case that to ensure that that particular import works without a deprecation message - maybe continue baking the core names into the extension modules (that doesn't matter much in the end after all, as long as we get the deprecation messages the way we expect them to appear).

mtsokol · 2023-09-07T13:28:28Z

Everything I tried still works, with a deprecation message appearing. The thing I'm wondering about too is if the reverse is going to keep working. If the one problem here was:
PyObject *numpy =  PyImport_ImportModule("numpy.core._multiarray_umath");
and now we rebuild MPL against a new numpy version where that extension is in _core, can the C API shim or numpy2_compat package from https://numpy.org/neps/nep-0053-c-abi-evolution.html fix that?

@rgommers I don't think there's anything that needs to be fixed. If a C file imports numpy.core._multiarray_umath from a NumPy with _core, it will work as we have core shim in Python, I checked it myself (Python's deprecation warning is not displayed during C execution).
Did I get your question right?

rgommers · 2023-09-07T13:32:46Z

Did I get your question right?

Not quite, I meant the reverse. If we build Matplotlib against NumPy 2.0, and hence the C code (after this PR) uses numpy._core._multiarray_umath, that won't work against NumPy 1.25.0 because there's no numpy._core there. So my question is how we make that work.

Another thought: it could be done through try-except logic in the C code: if numpy._core doesn't exist, then fall back to numpy.core. Or import numpy first, check the version logic, and then decide if we need core or _core.

mtsokol · 2023-09-07T13:37:24Z

Did I get your question right?

Not quite, I meant the reverse. If we build Matplotlib against NumPy 2.0, and hence the C code (after this PR) uses numpy._core._multiarray_umath, that won't work against NumPy 1.25.0 because there's no numpy._core there. So my question is how we make that work.

Another thought: it could be done through try-except logic in the C code: if numpy._core doesn't exist, then fall back to numpy.core. Or import numpy first, check the version logic, and then decide if we need core or _core.

Ah, now it's clear: Then it can be done either by trying to import _core/core or checking numpy version directly in import_array() in the generated header.

seberg · 2023-09-07T14:25:02Z

Changing from where we import based on NumPy is not a problem. It would be good if the old place keeps working modulo a possible deprecation warning. That just makes our life easier right now where we are not actually forcing recompilation (but just hoping that the ABI changes are small enough that downstream is fine).

ngoldbaum · 2023-10-09T22:41:28Z

numpy/core/multiarray.py


-"""
+_raise_warning("*")


The warning I get from this is not great:

DeprecationWarning: `numpy.core` has been made officially private and renamed to `numpy._core`. Accessing `_core` directly is discouraged, as members of `_core` can change without any warning at any time. You should use a public module instead that exports the attribute in question. If you still would like to access an attribute from it, please use `np._core.*`.

Note that the suggestion is wrong (it should point to np.core.multiarray, not np.core). But also it's not giving me the name of the symbol causing this issue, which it could.

Can we replace this with a module-level getattr please? Also customize the warning for submodules of np.core.

Based on our discussions, file-level warnings have been removed, so file-wise imports, such as:

from numpy.core.multiarray import _reconstruct

raise no warning (allowing undisturbed loading of 1.x pickle files).

The warning is still present when accessing via np.core.*. Therefore:

import numpy as np np.core.multiarray

gives:

<ipython-input-1-725e8a67566f>:1: DeprecationWarning: `numpy.core` has been made officially private and renamed to `numpy._core`. Accessing `_core` directly is discouraged, as members of `_core` can change without any warning at any time. You should use a public module instead that exports the attribute in question. If you still would like to access an attribute from it, please use `np._core.multiarray`.

ngoldbaum · 2023-10-09T22:43:30Z

numpy/core/_multiarray_umath.py

+from numpy._core import _multiarray_umath
+from ._utils import _raise_warning
+
+_raise_warning("*")


Removed - explained in the previous comment.

mtsokol · 2023-10-12T14:32:42Z

I removed NumpyUnpickler and refined numpy.core stubs (numpy._core will be backported to 1.26 in #24906).

ngoldbaum

Just a few more comments to tweak some of the errors and to catch some stuff that might break existing fixes for really old pickle files.

doc/release/upcoming_changes/24634.change.rst

numpy/_core/tests/test_regression.py

numpy/_core/__init__.py

numpy/_core/_internal.py

ngoldbaum · 2023-10-12T18:39:43Z

numpy/core/arrayprint.py

For this and all of the other submodules in np.core (even the private ones!), can you add module-level __getattr__ implementations saying that accessing things via np.core is deprecated. Along with a nice message explaining exactly what was accessed and what the migration is.

Just as an example, right now you can do this without ever seeing a warning:

In [1]: import numpy.core.multiarray In [2]: numpy.core.multiarray.ndarray Out[2]: numpy.ndarray

Whereas I would expect to see a deprecation warning like

numpy.core has been made officially private and renamed to numpy._core. Accessing _core directly is discouraged, as members of _core can change without any warning at any time. You should use a public module instead that exports the attribute in question. If you still would like to access this attribute without a warning despite the lack of backward compatibility guarantees, please use numpy._core.multiarray.ndarray.

I'm not sure about this change - when I add a custom __getattr__ for numpy.core.multiarray submodule that shows a warning, then:

from numpy.core.multiarray import _reconstruct

Also gives a warning, as _reconstruct is accessed with getattr(numpy.core.multiarray, "_reconstruct") internally.

This will cause pickles to also raise warnings that we want to avoid. I think the only way to have warnings is when user does np.core.* which triggers core's __getattr__, so as it's implemented right now. Do I understand it correctly?

That sounds mostly right. When imported explicitly, a submodule import will (normally?) side-step the parents module __getattr__. But that I suspect that is the only special path you can (ab)use here.
Once import of the submodule finished it has the same effect as __getattr__() having being called: the submodule is added to the module __dict__() and __getattr__() will never be called again (even by attribute access as is the case above).

But... if you like, I suspect you can implement a warning __getattr__() for all symbols that are imported from elsewhere. Note that multiarray._reconstruct is special there because its __module__ is explicitly replaced to be np.core.multiarray.

For some of those, they will not have a __module__, but in that case the submodule presumably is listed earlier in sys.modules so pickle will have used the submodule.
(I am not 100% sure this cannot ever be broken, but I will bet it is OK in practice.)

Do I understand it correctly?

Hmm, I thought I had a version where a module-level __getattr__ would work a few days ago while I was messing around with his, but now I can't get that to work. I'm going to try a little more to see where I'm not understanding things.

numpy/core/_utils.py

ngoldbaum · 2023-10-16T16:47:36Z

OK I think this is really ready and I'd like to merge it tomorrow. @mtsokol I think you mentioned you'd like to do some history rewriting before I merge?

I think keeping the mechanicalcore to _core renaming in its own commit and subsequent things we did to create the stub namespace in separate commits makes the most sense. I'd like to avoid squashing this into one commit. This will hopefully make git archeology in the future a little bit easier.

MAINT: Update git submodules, fix test_format

Match x86-simd-sort with the main branch Add remaining shims and a test Adjust docstrings and imports for isalpha Match svml version with the main branch Apply review changes Rework np.core stubs to generate warnings more often Co-authored-by: Nathan Goldbaum <[email protected]>

mtsokol · 2023-10-17T09:37:21Z

OK I think this is really ready and I'd like to merge it tomorrow. @mtsokol I think you mentioned you'd like to do some history rewriting before I merge?

@ngoldbaum Sure! I squashed it into five commits:

Rename numpy.core module to numpy.core
Adjust numpy.core paths in the codebase to _core
numpy.core shims
The crucial C-level change with imports
The rest of changes, including your imports logic

ngoldbaum · 2023-10-17T14:41:13Z

Thanks for pushing this through @mtsokol!

If you're reading this because you are seeing new warnings about deprecations and are unsure what to do, read on.

The numpy.core namespace was always intended to contain private implementation details, but this was never clearly marked. As part of the cleanups scheduled for Numpy 2.0, this PR renamed np.core to np._core and made accessing items in np.core generate deprecation warnings.

The most common usage of numpy.core we've found is to access functionality in NumPy that is already publicly available elsewhere. To summarize the NumPy 2.0 migration guide, the most likely fix for the warnings you are seeings is that you will need to import from the main numpy namespace instead of numpy.core.

If it turns out that you are importing something from numpy.core that is not publicly available, then it is likely you are making use of private numpy internals. If you would like to continue doing that despite the lack of backward compatibility guarantees from NumPy, you can import from numpy._core without generating any warnings.

If you feel that you are using something that should be public but is not, please open an issue describing your need.

mtsokol self-assigned this Sep 4, 2023

github-actions bot added the 30 - API label Sep 4, 2023

mtsokol added the Numpy 2.0 API Changes label Sep 4, 2023

mtsokol added this to the 2.0.0 release milestone Sep 4, 2023

BvB93 mentioned this pull request Sep 4, 2023

ENH: Refactor the typing "reveal" tests using typing.assert_type #24635

Merged

mtsokol force-pushed the rename-numpy-core branch from a55293f to 59c8309 Compare September 4, 2023 15:03

rgommers reviewed Sep 4, 2023

View reviewed changes

numpy/tests/test_public_api.py Outdated Show resolved Hide resolved

tools/refguide_check.py Show resolved Hide resolved

tools/refguide_check.py Outdated Show resolved Hide resolved

numpy/lib/tests/test_format.py Show resolved Hide resolved

mtsokol mentioned this pull request Sep 5, 2023

MAINT: Import fromarrays from rec scipy/scipy#19183

Merged

mtsokol force-pushed the rename-numpy-core branch 2 times, most recently from 24b4b19 to eab5a88 Compare September 7, 2023 07:59

mtsokol mentioned this pull request Sep 8, 2023

MAINT: Check numpy version in C-API mtsokol/numpy#1

Merged

mtsokol force-pushed the rename-numpy-core branch from cdebcf3 to d4a334c Compare September 11, 2023 19:12

mtsokol requested a review from rgommers September 12, 2023 07:09

ksunden mentioned this pull request Sep 15, 2023

[MNT]: Numpy 2.0 support strategy matplotlib/matplotlib#26778

Closed

mtsokol force-pushed the rename-numpy-core branch from 95b767d to 031e551 Compare September 17, 2023 18:52

mtsokol mentioned this pull request Sep 17, 2023

Question regarding renaming core to _core intel/x86-simd-sort#78

Closed

mtsokol force-pushed the rename-numpy-core branch 2 times, most recently from 4ee7e57 to a7e8659 Compare September 20, 2023 19:18

mtsokol mentioned this pull request Sep 21, 2023

MAINT: Add warning filter for numpy.core rename scipy/scipy#19275

Merged

mtsokol force-pushed the rename-numpy-core branch from 9bb283c to 179f441 Compare October 9, 2023 07:42

ngoldbaum reviewed Oct 9, 2023

View reviewed changes

mtsokol force-pushed the rename-numpy-core branch from 179f441 to 897e7f6 Compare October 12, 2023 11:36

ngoldbaum reviewed Oct 12, 2023

View reviewed changes

mtsokol force-pushed the rename-numpy-core branch 6 times, most recently from 218b64f to 7ad9288 Compare October 16, 2023 14:25

mtsokol and others added 5 commits October 17, 2023 10:53

MAINT: Rename numpy.core module to numpy._core

923f219

MAINT: Update numpy.core paths to numpy._core

47b70cb

MAINT: Update git submodules, fix test_format

MAINT: Expand numpy.core shims

4a7191c

MAINT: Check numpy version in C-API

72702be

mtsokol force-pushed the rename-numpy-core branch from 7ad9288 to 7d4f488 Compare October 17, 2023 09:02

ngoldbaum merged commit f8e5761 into numpy:main Oct 17, 2023

mtsokol deleted the rename-numpy-core branch October 17, 2023 14:42

mroeschke mentioned this pull request Oct 17, 2023

TST: Remove np.core usage pandas-dev/pandas#55565

Merged

neutrinoceros mentioned this pull request Oct 18, 2023

BUG: fix future incompatibility with numpy 2.0 (numpy.core isn't public API) yt-project/unyt#455

Merged

neutrinoceros mentioned this pull request Apr 24, 2024

ENH: avoid deprecation warnings from numpy 2 h5py/h5py#2416

Merged

mattip mentioned this pull request Jul 2, 2024

DOC, CI: Docs build is failing on CircleCI #26820

Closed

cplusv2023 mentioned this pull request Dec 10, 2024

[FEA] Numpy 2.0.0+ include path ahay/src#313

Closed

This was referenced Apr 23, 2025

Workaround for numpy.core DeprecationWarning tomMoral/joblib#2

Closed

Workaround for numpy.core DeprecationWarning joblib/joblib#1714

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API: Rename `numpy/core` to `numpy/_core` [NEP 52] #24634

API: Rename `numpy/core` to `numpy/_core` [NEP 52] #24634

mtsokol commented Sep 4, 2023

rgommers left a comment

ksunden commented Sep 4, 2023 •

edited

Loading

ksunden commented Sep 4, 2023

rgommers commented Sep 5, 2023

ksunden commented Sep 5, 2023

rgommers commented Sep 5, 2023

mtsokol commented Sep 7, 2023 •

edited

Loading

rgommers commented Sep 7, 2023

mtsokol commented Sep 7, 2023 •

edited

Loading

rgommers commented Sep 7, 2023

mtsokol commented Sep 7, 2023 •

edited

Loading

seberg commented Sep 7, 2023

ngoldbaum Oct 9, 2023

mtsokol Oct 12, 2023

ngoldbaum Oct 9, 2023

mtsokol Oct 12, 2023

mtsokol commented Oct 12, 2023

ngoldbaum left a comment

ngoldbaum Oct 12, 2023 •

edited

Loading

mtsokol Oct 13, 2023

seberg Oct 13, 2023

ngoldbaum Oct 13, 2023

ngoldbaum commented Oct 16, 2023 •

edited

Loading

mtsokol commented Oct 17, 2023

ngoldbaum commented Oct 17, 2023


		"""
		_raise_warning("*")

API: Rename numpy/core to numpy/_core [NEP 52] #24634

API: Rename numpy/core to numpy/_core [NEP 52] #24634

Conversation

mtsokol commented Sep 4, 2023

rgommers left a comment

Choose a reason for hiding this comment

ksunden commented Sep 4, 2023 • edited Loading

ksunden commented Sep 4, 2023

rgommers commented Sep 5, 2023

ksunden commented Sep 5, 2023

rgommers commented Sep 5, 2023

mtsokol commented Sep 7, 2023 • edited Loading

rgommers commented Sep 7, 2023

mtsokol commented Sep 7, 2023 • edited Loading

rgommers commented Sep 7, 2023

mtsokol commented Sep 7, 2023 • edited Loading

seberg commented Sep 7, 2023

ngoldbaum Oct 9, 2023

Choose a reason for hiding this comment

mtsokol Oct 12, 2023

Choose a reason for hiding this comment

ngoldbaum Oct 9, 2023

Choose a reason for hiding this comment

mtsokol Oct 12, 2023

Choose a reason for hiding this comment

mtsokol commented Oct 12, 2023

ngoldbaum left a comment

Choose a reason for hiding this comment

ngoldbaum Oct 12, 2023 • edited Loading

Choose a reason for hiding this comment

mtsokol Oct 13, 2023

Choose a reason for hiding this comment

seberg Oct 13, 2023

Choose a reason for hiding this comment

ngoldbaum Oct 13, 2023

Choose a reason for hiding this comment

ngoldbaum commented Oct 16, 2023 • edited Loading

mtsokol commented Oct 17, 2023

ngoldbaum commented Oct 17, 2023

API: Rename `numpy/core` to `numpy/_core` [NEP 52] #24634

API: Rename `numpy/core` to `numpy/_core` [NEP 52] #24634

ksunden commented Sep 4, 2023 •

edited

Loading

mtsokol commented Sep 7, 2023 •

edited

Loading

mtsokol commented Sep 7, 2023 •

edited

Loading

mtsokol commented Sep 7, 2023 •

edited

Loading

ngoldbaum Oct 12, 2023 •

edited

Loading

ngoldbaum commented Oct 16, 2023 •

edited

Loading