Thanks to visit codestin.com
Credit goes to github.com

Upgrade Faiss OSS side to numpy2 #4523

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Closed

mnorris11 wants to merge 1 commit into facebookresearch:main from mnorris11:export-D79900113

mnorris11 commented Aug 8, 2025 •

edited

Loading

Differential Revision: D79900113

This upgrades OSS Faiss to numpy 2.3.3. I tried to fix issues based on https://numpy.org/devdocs/numpy_2_0_migration_guide.html.

Overall, numpy2 causes some test issues on various platforms/optimization levels.

Test changes that are OK:

test_contrib.py: test_float: changing to use item(): this is just the new way of getting the value, like recommended in the migration guide.

Test changes that indicate some problem:

test_contrib.py: test_ivf_train_2level: This fails on many platforms (avx2, conda installs). I could not find what the issue was. I had to increase the bounds slightly.
test_index_accuracy.py: test_OIVFPQ: This fails just on AVX512. I had to increase the iterations from 10 to 12.
test_local_search_quantizer.py and test_standalone_codec.py: Some of these break on Windows. Others needed lighter tolerances.

Debugging attempts: Especially for test_ivf_train_2level, I tried to unsuccessfully track down what might have changed. I suspected the type promotion change from numpy1 to 2.

- np.float32(3) + 3. now returns a float32 when it previously returned a float64.
- np.array([3], dtype=np.float32) + np.float64(3) will now return a float64 array. (The higher precision of the scalar is not ignored.)

I changed all datasets to be explicitly float64s, and checked for any constants in the code path. No luck.

meta-cla bot added the CLA Signed label

Contributor

facebook-github-bot commented Aug 8, 2025

This pull request was exported from Phabricator. Differential Revision: D79900113

facebook-github-bot added the fb-exported label

mnorris11 added the ignore label

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request


          Check errors in numpy >= 2 (facebookresearch#4523)

c73e917

Summary:

Rollback Plan:

Differential Revision: D79900113

mnorris11 force-pushed the export-D79900113 branch from f86a838 to c73e917 Compare

August 19, 2025 23:31

Contributor

facebook-github-bot commented Aug 19, 2025

This pull request was exported from Phabricator. Differential Revision: D79900113

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request


          Check errors in numpy >= 2 (facebookresearch#4523)

a012c95

Summary:

Rollback Plan:

Differential Revision: D79900113

mnorris11 force-pushed the export-D79900113 branch from c73e917 to a012c95 Compare

August 20, 2025 02:05

Contributor

facebook-github-bot commented Aug 20, 2025

This pull request was exported from Phabricator. Differential Revision: D79900113

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request


          Check errors in numpy >= 2 (facebookresearch#4523)

4e82dc8

Summary:

Rollback Plan:

Differential Revision: D79900113

mnorris11 force-pushed the export-D79900113 branch from a012c95 to 4e82dc8 Compare

August 20, 2025 15:41

Contributor

facebook-github-bot commented Aug 20, 2025

This pull request was exported from Phabricator. Differential Revision: D79900113

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request


          Check errors in numpy >= 2 (facebookresearch#4523)

a8d5ae6

Summary:

Rollback Plan:

Differential Revision: D79900113

mnorris11 force-pushed the export-D79900113 branch from 4e82dc8 to a8d5ae6 Compare

August 20, 2025 18:10

Contributor

facebook-github-bot commented Aug 20, 2025

This pull request was exported from Phabricator. Differential Revision: D79900113

mnorris11 force-pushed the export-D79900113 branch from a8d5ae6 to d5a2a0f Compare

August 20, 2025 19:55

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request


          Check errors in numpy >= 2 (facebookresearch#4523)

d5a2a0f

Summary:

Rollback Plan:

Differential Revision: D79900113

Contributor

facebook-github-bot commented Aug 20, 2025

This pull request was exported from Phabricator. Differential Revision: D79900113

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request


          Check errors in numpy >= 2 (facebookresearch#4523)

36ce062

Summary:

Rollback Plan:

Differential Revision: D79900113

mnorris11 force-pushed the export-D79900113 branch from d5a2a0f to 36ce062 Compare

August 20, 2025 22:35

Contributor

facebook-github-bot commented Aug 20, 2025

This pull request was exported from Phabricator. Differential Revision: D79900113

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request


          Check errors in numpy >= 2 (facebookresearch#4523)

e2034e1

Summary:

Rollback Plan:

Differential Revision: D79900113

mnorris11 force-pushed the export-D79900113 branch from 36ce062 to e2034e1 Compare

August 20, 2025 23:51

Contributor

facebook-github-bot commented Aug 20, 2025

This pull request was exported from Phabricator. Differential Revision: D79900113

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request


          Check errors in numpy >= 2 (facebookresearch#4523)

240591c

Summary:

Rollback Plan:

Differential Revision: D79900113

mnorris11 force-pushed the export-D79900113 branch from e2034e1 to 240591c Compare

August 21, 2025 02:46

Contributor

facebook-github-bot commented Aug 21, 2025

This pull request was exported from Phabricator. Differential Revision: D79900113

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request


          Check errors in numpy >= 2 (facebookresearch#4523)

d424a18

Summary:

Rollback Plan:

Differential Revision: D79900113

mnorris11 force-pushed the export-D79900113 branch from 240591c to d424a18 Compare

August 21, 2025 15:40

mnorris11 force-pushed the export-D79900113 branch from bb0d0af to 4280ec5 Compare

October 1, 2025 03:54

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request


          Check errors in numpy >= 2 (facebookresearch#4523)

b5ac665

Summary: Pull Request resolved: facebookresearch#4523

Differential Revision: D79900113

mnorris11 force-pushed the export-D79900113 branch from 4280ec5 to b5ac665 Compare

October 1, 2025 15:37

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request


          Check errors in numpy >= 2 (facebookresearch#4523)

117f461

Summary: Pull Request resolved: facebookresearch#4523

Differential Revision: D79900113

mnorris11 force-pushed the export-D79900113 branch 2 times, most recently from 117f461 to 5d35899 Compare

October 1, 2025 16:38

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request


          Upgrade Faiss OSS side to numpy2 (facebookresearch#4523)

5d35899

Summary:

This upgrades OSS Faiss to numpy 2.3.3. I tried to fix issues based on https://numpy.org/devdocs/numpy_2_0_migration_guide.html.

There are some issues:
-

Differential Revision: D79900113

mnorris11 changed the title ~~Check errors in numpy >= 2~~ Upgrade Faiss OSS side to numpy2

mnorris11 mentioned this pull request

Faiss compatibility with NumPy 2 (ImportError: numpy.core.multiarray failed to import) #3526

Closed

4 tasks

mnorris11 marked this pull request as ready for review

October 1, 2025 18:02

jakirkham reviewed

View reviewed changes

jakirkham left a comment •

edited

Loading

Thanks Michael! 🙏

Had a few suggestions on how to build with NumPy 2, which will make the packages more flexible for users installing them

Also these conda-forge NumPy docs may provide additional context

conda/faiss-gpu-cuvs/meta.yaml Outdated

    
                      - _openmp_mutex =4.5=2_kmp_llvm  # [x86_64]

                      - python {{ python }}

                      - numpy >=1.19,<2

                      - numpy =2.3.3

jakirkham Oct 1, 2025

This can be relaxed to just ensure NumPy 2 is picked up

Suggested change

      
                    - numpy =2.3.3
          
                    - numpy >=2,<3.0a0

Though if you need a particular NumPy version, the lower bound could be updated

Suggested change

      
                    - numpy =2.3.3
          
                    - numpy >=2.3,<3.0a0

conda/faiss-gpu-cuvs/meta.yaml Outdated

    
                      - _openmp_mutex =4.5=2_kmp_llvm  # [x86_64]

                      - python {{ python }}

                      - numpy >=1.19,<2

                      - numpy =2.3.3

jakirkham Oct 1, 2025

The numpy package already specifies run_exports that constrains the numpy version used at runtime (both in conda-forge and in defaults)

So this can just be either...

Suggested change

      
                    - numpy =2.3.3
          
                    - numpy

...or dropped entirely...

Suggested change

- numpy =2.3.3

conda/faiss-gpu-cuvs/meta.yaml Outdated

    
                  test:

                    requires:

                      - numpy >=1.19,<2

                      - numpy =2.3.3

jakirkham Oct 1, 2025

If you want to constrain to NumPy 2 in tests, this could be done like so...

Suggested change

      
                    - numpy =2.3.3
          
                    - numpy >=2,<3.0a0

If you need the lower bound, then you could add it like so

Suggested change

      
                    - numpy =2.3.3
          
                    - numpy >=2.3,<3.0a0

conda/faiss-gpu/meta.yaml Outdated

    
                      - mkl =2023.0  # [x86_64]

                      - python {{ python }}

                      - numpy >=1.19,<2

                      - numpy =2.3.3

jakirkham Oct 1, 2025

Would recommend the same changes in this and other files

jakirkham commented Oct 1, 2025

The NumPy changes also look reasonable to my eyes. Though cc-ing @seberg in case I've missed anything

Also NumPy has a migration guide for NumPy 2. Additionally ruff has builtin support for migrating Python code to be NumPy 2 compliant, which can help automatically update code

Author

mnorris11 commented Oct 1, 2025 •

edited

Loading

Thanks Michael! 🙏

Had a few suggestions on how to build with NumPy 2, which will make the packages more flexible for users installing them

Also these conda-forge NumPy docs may provide additional context

Thanks @jakirkham for the review!
The flexible versions you suggest sound the best to me. This is the most user friendly.
We have experienced frequent nightly failures in the past when conda decides to pick up a different version (i.e. 2.1 instead of 2.3 etc), so by default I have been pinning exact versions. (I will change it to go with the flexible pinning for now, but we may have to go back to an exact version later.)

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request


          Upgrade Faiss OSS side to numpy2 (facebookresearch#4523)

0d88108

Summary:

This upgrades OSS Faiss to numpy 2.3.3. I tried to fix issues based on https://numpy.org/devdocs/numpy_2_0_migration_guide.html.

Overall, numpy2 causes some test issues on various platforms/optimization levels.

Test changes that are OK:
- test_contrib.py: test_float: changing to use item(): this is just the new way of getting the value, like recommended in the migration guide.

Test changes that indicate some problem:
- test_contrib.py: test_ivf_train_2level: This fails on many platforms (avx2, conda installs). I could not find what the issue was. I had to increase the bounds slightly.
- test_index_accuracy.py: test_OIVFPQ: This fails just on AVX512. I had to increase the iterations from 10 to 12.
- test_local_search_quantizer.py and test_standalone_codec.py: Some of these break on Windows. Others needed lighter tolerances.

Debugging attempts: Especially for test_ivf_train_2level, I tried to unsuccessfully track down what might have changed. I suspected the type promotion change from numpy1 to 2.
```
- np.float32(3) + 3. now returns a float32 when it previously returned a float64.
- np.array([3], dtype=np.float32) + np.float64(3) will now return a float64 array. (The higher precision of the scalar is not ignored.)
```
I changed all datasets to be explicitly float64s, and checked for any constants in the code path. No luck.

Devmate was less than useless, it was actively a waste of time (as usual...)

Differential Revision: D79900113

mnorris11 force-pushed the export-D79900113 branch from 5d35899 to 0d88108 Compare

October 1, 2025 21:00

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request


          Upgrade Faiss OSS side to numpy2 (facebookresearch#4523)

5ee5802

Summary:

This upgrades OSS Faiss to numpy 2.3.3. I tried to fix issues based on https://numpy.org/devdocs/numpy_2_0_migration_guide.html.

Overall, numpy2 causes some test issues on various platforms/optimization levels.

Test changes that are OK:
- test_contrib.py: test_float: changing to use item(): this is just the new way of getting the value, like recommended in the migration guide.

Test changes that indicate some problem:
- test_contrib.py: test_ivf_train_2level: This fails on many platforms (avx2, conda installs). I could not find what the issue was. I had to increase the bounds slightly.
- test_index_accuracy.py: test_OIVFPQ: This fails just on AVX512. I had to increase the iterations from 10 to 12.
- test_local_search_quantizer.py and test_standalone_codec.py: Some of these break on Windows. Others needed lighter tolerances.

Debugging attempts: Especially for test_ivf_train_2level, I tried to unsuccessfully track down what might have changed. I suspected the type promotion change from numpy1 to 2.
```
- np.float32(3) + 3. now returns a float32 when it previously returned a float64.
- np.array([3], dtype=np.float32) + np.float64(3) will now return a float64 array. (The higher precision of the scalar is not ignored.)
```
I changed all datasets to be explicitly float64s, and checked for any constants in the code path. No luck.

Devmate was less than useless, it was actively a waste of time (as usual...)

Differential Revision: D79900113

mnorris11 force-pushed the export-D79900113 branch from 05a9e2f to 5ee5802 Compare

October 2, 2025 02:57

meta-codesync bot commented Oct 2, 2025

@mnorris11 has exported this pull request. If you are a Meta employee, you can view the originating Diff in D79900113.

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request


          Upgrade Faiss OSS side to numpy2 (facebookresearch#4523)

ef1b1a7

Summary:

This upgrades OSS Faiss to numpy 2.3.3. I tried to fix issues based on https://numpy.org/devdocs/numpy_2_0_migration_guide.html.

Overall, numpy2 causes some test issues on various platforms/optimization levels.

Test changes that are OK:
- test_contrib.py: test_float: changing to use item(): this is just the new way of getting the value, like recommended in the migration guide.

Test changes that indicate some problem:
- test_contrib.py: test_ivf_train_2level: This fails on many platforms (avx2, conda installs). I could not find what the issue was. I had to increase the bounds slightly.
- test_index_accuracy.py: test_OIVFPQ: This fails just on AVX512. I had to increase the iterations from 10 to 12.
- test_local_search_quantizer.py and test_standalone_codec.py: Some of these break on Windows. Others needed lighter tolerances.

Debugging attempts: Especially for test_ivf_train_2level, I tried to unsuccessfully track down what might have changed. I suspected the type promotion change from numpy1 to 2.
```
- np.float32(3) + 3. now returns a float32 when it previously returned a float64.
- np.array([3], dtype=np.float32) + np.float64(3) will now return a float64 array. (The higher precision of the scalar is not ignored.)
```
I changed all datasets to be explicitly float64s, and checked for any constants in the code path. No luck.

Devmate was less than useless, it was actively a waste of time (as usual...)

Differential Revision: D79900113

mnorris11 force-pushed the export-D79900113 branch from 5ee5802 to ef1b1a7 Compare

October 2, 2025 03:10


          Upgrade Faiss OSS side to numpy2 (facebookresearch#4523)

afda341

Summary:

This upgrades OSS Faiss to numpy 2.3.3. I tried to fix issues based on https://numpy.org/devdocs/numpy_2_0_migration_guide.html.

Overall, numpy2 causes some test issues on various platforms/optimization levels.

Test changes that are OK:
- test_contrib.py: test_float: changing to use item(): this is just the new way of getting the value, like recommended in the migration guide.

Test changes that indicate some problem:
- test_contrib.py: test_ivf_train_2level: This fails on many platforms (avx2, conda installs). I could not find what the issue was. I had to increase the bounds slightly.
- test_index_accuracy.py: test_OIVFPQ: This fails just on AVX512. I had to increase the iterations from 10 to 12.
- test_local_search_quantizer.py and test_standalone_codec.py: Some of these break on Windows. Others needed lighter tolerances.

Debugging attempts: Especially for test_ivf_train_2level, I tried to unsuccessfully track down what might have changed. I suspected the type promotion change from numpy1 to 2.
```
- np.float32(3) + 3. now returns a float32 when it previously returned a float64.
- np.array([3], dtype=np.float32) + np.float64(3) will now return a float64 array. (The higher precision of the scalar is not ignored.)
```
I changed all datasets to be explicitly float64s, and checked for any constants in the code path. No luck.

Devmate was less than useless, it was actively a waste of time (as usual...)

Differential Revision: D79900113

mnorris11 force-pushed the export-D79900113 branch from ef1b1a7 to afda341 Compare

October 2, 2025 03:50

seberg commented Oct 2, 2025

Looks fine, cannot say quickly why the item calls are needed, but probably OK. Tolerance changes may mean that some internal computation previously went to float64 and now stay at float32, I guess. But that is presumably also fine.

meta-codesync bot closed this in

4fab13c

facebook-github-bot added the Merged label

Contributor

facebook-github-bot commented Oct 2, 2025

This pull request has been merged in 4fab13c.

jakirkham commented Oct 2, 2025

Thanks @jakirkham for the review!
The flexible versions you suggest sound the best to me. This is the most user friendly.
We have experienced frequent nightly failures in the past when conda decides to pick up a different version (i.e. 2.1 instead of 2.3 etc), so by default I have been pinning exact versions. (I will change it to go with the flexible pinning for now, but we may have to go back to an exact version later.)

Happy to help 🙂

Thanks Michael! 🙏

If you run into issues, please let me know. Would be interested in taking a look at those when they come up

jakirkham reviewed

View reviewed changes

.github/actions/build_cmake/action.yml

    
                      # Ensure starting packages are from conda-forge.

                      conda list --show-channel-urls

                      conda update -y -q conda

                      conda install -y -q "conda<=25.07"

jakirkham Oct 2, 2025

Guessing this is due to an issue with conda version 25.09, do you have more details on the issue seen?

Author

mnorris11 Oct 2, 2025

Yes, seems like it just surfaced. Prior runs used 25.07 so I just pinned there.
https://github.com/facebookresearch/faiss/actions/runs/18175470029/job/51741940955
The last logs are:

Error while loading conda entry point: conda-libmamba-solver (cannot import name 'Spinner' from 'conda.common.io' (/home/runner/miniconda3/lib/python3.12/site-packages/conda/common/io.py))

CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic

Error: Process completed with exit code 1.

jakirkham Oct 2, 2025

Thanks Michael! 🙏

Think I can see what is going on. Have filed an upstream issue: conda/conda#15287

Also to help with debugging future Conda or conda-forge issues, have suggested a few commands we can add to CI in issue: #4599

This would help in providing additional detail when debugging. Additionally this would help in upstream bug reports

jakirkham mentioned this pull request

conda version 25.9.0 has too lax of a constraint on conda-libmamba-solver conda/conda#15287

Open

2 tasks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed fb-exported ignore Merged meta-exported