API: Cleaning `numpy/init.py` and main namespace - Part 5 [NEP 52] #24587

mtsokol · 2023-08-30T12:32:58Z

Relevant issue: #24306

Scope of the PR:

Remove compare_chararrays as it comes from a legacy module. It's still available as np.char.compare_chararrays.
~~Remove~~ Deprecate chararray as it comes from a legacy module. It's still available as np.char.chararray.
Remove format_parser as it's the only function from rec present in the main namespace. It's still available as np.rec.format_parser.
~~Remove recarray. It's still available as np.rec.recarray.~~

ngoldbaum · 2023-08-30T14:45:55Z

It looks like pandas needs to be updated for the doc build to pass.

It also looks like the repr for recarray needs to be updated to reflect its new canonical location:

File "/tmp/tmpraxm5dc8/doc/source/user/basics.rec.rst", line 699, in doc/source/user/basics.rec.rst
Failed example:
    type(recordarr.bar)
Expected:
    <class 'numpy.recarray'>
Got:
    <class 'numpy.core.records.recarray'>

* based on numpy/numpy#24587: * replace the single usage of `np.recarray` I found in our codebase--there are references to third-party libraries providing us with `recarray` that I haven't touched * similar PRs have been merged to `pandas` and `scipy` [skip cirrus]

numpy/char/__init__.pyi

rgommers · 2023-09-06T19:03:48Z

Can we actually do this? It looks to me like np.recarray and np.chararray have too much usage (~9,600 and ~2,500 hits in code search for me). So this may be more disruption than it's worth.

mtsokol · 2023-09-07T06:27:35Z

Can we actually do this? It looks to me like np.recarray and np.chararray have too much usage (~9,600 and ~2,500 hits in code search for me). So this may be more disruption than it's worth.

@rgommers These are used but the change to move to np.rec.recarray and np.char.chararray is straightforward and backward compatible. But I agree it will cause some disruption.

I think we have three options:

Keep the PR in the current form: np.recarray and np.chararray are moved to np.rec and np.char.
Revert changes and both names are available from np.* namespace and np.rec.* and np.char.* namespaces as they are right now in main (Drawback: this will break "each function in exactly one place" rule that we're trying to enforce).
Keep np.recarray and np.chararray but remove them from their np.rec.* and np.char.* namespaces (Does it make sense?) - I see that np.rec.recarray and np.char.chararray are way less popular (a hundred hits in search compared to thousands you mentioned).

What do you think about these options? Which should I go with?

rgommers · 2023-09-07T12:10:43Z

(Drawback: this will break "each function in exactly one place" rule that we're trying to enforce).

Yes, that'd be a little unfortunate. However, I'm not sure it's really that black-and-white. For all functions/objects we should have one documented/preferred location, but for some other popular functions we've also decided to leave a compatibility shim in place.

For chararray I think @ngoldbaum had a preference to move that away from the main namespace to make space for the better new functionality (NEP 55)? Not sure though. Let's see what he thinks about that one.

For recarray we don't really have a replacement, so I think it's okay to leave it in the main namespace as the preferred location. I'm okay keeping both np.recarray and np.rec.recarray, since it's a single object only, so we could consider this one as a case where practicality beats purity.

ngoldbaum · 2023-09-11T15:48:48Z

For chararray I think @ngoldbaum had a preference to move that away from the main namespace to make space for the better new functionality (NEP 55)? Not sure though. Let's see what he thinks about that one.

If taking it out of the main namespace would be too disruptive it should stay, practicality beats purity. That said, a user-visible deprecation warning in NumPy 2.0 with a plan to remove it in a few versions also makes sense I think.

mtsokol · 2023-09-12T10:22:28Z

Hi @rgommers @ngoldbaum,

I updated this PR according to your feedback:

np.recarray is available again (also np.rec.recarray can be used),
np.chararray is available again but accessing it yields a DeprecationWarning.

ngoldbaum

While we may not be deprecating np.recarray, I think it would still make sense to change its repr, in this PR I have:

In [3]: np.recarray
Out[3]: numpy.recarray

Whereas I would expect it to be an alias to numpy.rec.recarray now rather than the other way around.

ngoldbaum · 2023-09-12T15:32:05Z

doc/release/upcoming_changes/24587.python_removal.rst

+  Use ``np.char.compare_chararrays`` instead.
+
+* ``np.chararray`` has been deprecated and it will be removed in the future.
+  Use ``np.char.chararray`` instead.


I think I would rephrase this slightly as "The charrarray in the main namespace has been deprecated. It can be imported without a deprecation warning from np.char.chararray for now, but we are planning to fully deprecate and remove chararray in the future.", since I think the intention is to fully deprecate it and remove it, we just haven't actually done the work yet, so can't outright remove it now. I proposed doing those deprecations before NumPy 2.0 as part of NEP 55, but that hasn't fully gone through yet. If that happens before NumPy 2.0, I will update this release note.

Sure! I updated this change note.

ngoldbaum · 2023-09-12T15:37:50Z

numpy/__init__.py

+            warnings.warn(
+                "`np.chararray` is deprecated and will be removed from "
+                "the main namespace in the future. Use `np.char.chararray` "
+                "instead.", DeprecationWarning, stacklevel=2)


For the migration, I would say "Use an array with a string or bytes dtype instead," while it will still be available as np.char.charray, that will likely also be deprecated before NumPy 2.0 is released.

Sure! Updated.

mtsokol · 2023-09-13T11:10:38Z

While we may not be deprecating np.recarray, I think it would still make sense to change its repr, in this PR I have:
In [3]: np.recarray
Out[3]: numpy.recarray
Whereas I would expect it to be an alias to numpy.rec.recarray now rather than the other way around.

@ngoldbaum I decided to keep repr as numpy.recarray - in my understanding all items present in the main namespace should have numpy.* repr. There is a separate test: test_numpy_namespace with check_dir function, that checks if main namespace functions have correct (numpy.*) __module__ attribute.

Should I change recarray repr and add to the allowlist in the test?

ngoldbaum · 2023-09-13T14:36:10Z

I personally think changing the canonical location makes sense and is in the spirit of the other changes we're making for the API migration. Feel free to wait for others to weigh in.

… from main namespace

Co-authored-by: Bas van Beek <[email protected]>

mtsokol · 2023-09-17T18:18:13Z

I personally think changing the canonical location makes sense and is in the spirit of the other changes we're making for the API migration. Feel free to wait for others to weigh in.

Makes sense to me - I updated the repr to numpy.rec.recarray.

ngoldbaum · 2023-09-18T16:18:33Z

Thanks so much for your diligence on pushing forward refactoring the main namespace @mtsokol!

mtsokol self-assigned this Aug 30, 2023

github-actions bot added the 30 - API label Aug 30, 2023

mtsokol added the Numpy 2.0 API Changes label Aug 30, 2023

mtsokol force-pushed the overhaul-of-main-namespace-part-5 branch 2 times, most recently from 7dad0c1 to b229e45 Compare September 1, 2023 11:07

tylerjereddy mentioned this pull request Sep 3, 2023

MAINT: NumPy recarray shim MDAnalysis/mdanalysis#4275

Merged

BvB93 mentioned this pull request Sep 4, 2023

ENH: Refactor the typing "reveal" tests using typing.assert_type #24635

Merged

mtsokol force-pushed the overhaul-of-main-namespace-part-5 branch from b229e45 to a622a12 Compare September 6, 2023 09:17

BvB93 reviewed Sep 6, 2023

View reviewed changes

numpy/char/__init__.pyi Outdated Show resolved Hide resolved

mtsokol force-pushed the overhaul-of-main-namespace-part-5 branch 3 times, most recently from bbf20e2 to 668edfa Compare September 12, 2023 10:15

ngoldbaum reviewed Sep 12, 2023

View reviewed changes

ngoldbaum added the triage review Issue/PR to be discussed at the next triage meeting label Sep 12, 2023

mtsokol added 3 commits September 17, 2023 19:35

API: Remove chararray, compare_chararrays, recarray and format_parser…

2391866

… from main namespace

Add pyi class definitions

2a7aebd

Add changelog file

6d3051e

mtsokol and others added 6 commits September 17, 2023 19:35

MAINT: Create rec and char submodules

6643c0e

Update numpy/setup.py

4727fb1

MAINT: Adjust typing tests

1aef35a

Update numpy/char/__init__.pyi

5752d92

Co-authored-by: Bas van Beek <[email protected]>

Restore np.recarray and deprecate np.chararray

cee5097

Apply review comments

eb6a10a

mtsokol force-pushed the overhaul-of-main-namespace-part-5 branch from 927f062 to eb6a10a Compare September 17, 2023 17:47

mtsokol added 2 commits September 17, 2023 20:02

Apply review comments

5312b6e

Fix rst doc

2752b04

ngoldbaum merged commit 1008227 into numpy:main Sep 18, 2023

mtsokol deleted the overhaul-of-main-namespace-part-5 branch September 18, 2023 16:19

pllim mentioned this pull request Sep 21, 2023

TST: Fix devdeps astropy/astropy#15363

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API: Cleaning `numpy/init.py` and main namespace - Part 5 [NEP 52] #24587

API: Cleaning `numpy/init.py` and main namespace - Part 5 [NEP 52] #24587

mtsokol commented Aug 30, 2023 •

edited

Loading

ngoldbaum commented Aug 30, 2023

rgommers commented Sep 6, 2023

mtsokol commented Sep 7, 2023

rgommers commented Sep 7, 2023

ngoldbaum commented Sep 11, 2023 •

edited

Loading

mtsokol commented Sep 12, 2023

ngoldbaum left a comment

ngoldbaum Sep 12, 2023

mtsokol Sep 13, 2023

ngoldbaum Sep 12, 2023

mtsokol Sep 13, 2023

mtsokol commented Sep 13, 2023

ngoldbaum commented Sep 13, 2023

mtsokol commented Sep 17, 2023

ngoldbaum commented Sep 18, 2023

API: Cleaning numpy/__init__.py and main namespace - Part 5 [NEP 52] #24587

API: Cleaning numpy/__init__.py and main namespace - Part 5 [NEP 52] #24587

Conversation

mtsokol commented Aug 30, 2023 • edited Loading

ngoldbaum commented Aug 30, 2023

rgommers commented Sep 6, 2023

mtsokol commented Sep 7, 2023

rgommers commented Sep 7, 2023

ngoldbaum commented Sep 11, 2023 • edited Loading

mtsokol commented Sep 12, 2023

ngoldbaum left a comment

Choose a reason for hiding this comment

ngoldbaum Sep 12, 2023

Choose a reason for hiding this comment

mtsokol Sep 13, 2023

Choose a reason for hiding this comment

ngoldbaum Sep 12, 2023

Choose a reason for hiding this comment

mtsokol Sep 13, 2023

Choose a reason for hiding this comment

mtsokol commented Sep 13, 2023

ngoldbaum commented Sep 13, 2023

mtsokol commented Sep 17, 2023

ngoldbaum commented Sep 18, 2023

API: Cleaning `numpy/init.py` and main namespace - Part 5 [NEP 52] #24587

API: Cleaning `numpy/init.py` and main namespace - Part 5 [NEP 52] #24587

mtsokol commented Aug 30, 2023 •

edited

Loading

ngoldbaum commented Sep 11, 2023 •

edited

Loading