Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Upgrade Faiss OSS side to numpy2 #4523

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

Upgrade Faiss OSS side to numpy2 #4523

wants to merge 1 commit into from
+65 −58

Conversation

mnorris11
Copy link

@mnorris11 mnorris11 commented Aug 8, 2025 •
edited
Loading

Differential Revision: D79900113

This upgrades OSS Faiss to numpy 2.3.3. I tried to fix issues based on https://numpy.org/devdocs/numpy_2_0_migration_guide.html.

Overall, numpy2 causes some test issues on various platforms/optimization levels.

Test changes that are OK:

  • test_contrib.py: test_float: changing to use item(): this is just the new way of getting the value, like recommended in the migration guide.

Test changes that indicate some problem:

  • test_contrib.py: test_ivf_train_2level: This fails on many platforms (avx2, conda installs). I could not find what the issue was. I had to increase the bounds slightly.
  • test_index_accuracy.py: test_OIVFPQ: This fails just on AVX512. I had to increase the iterations from 10 to 12.
  • test_local_search_quantizer.py and test_standalone_codec.py: Some of these break on Windows. Others needed lighter tolerances.

Debugging attempts: Especially for test_ivf_train_2level, I tried to unsuccessfully track down what might have changed. I suspected the type promotion change from numpy1 to 2.

- np.float32(3) + 3. now returns a float32 when it previously returned a float64.
- np.array([3], dtype=np.float32) + np.float64(3) will now return a float64 array. (The higher precision of the scalar is not ignored.)

I changed all datasets to be explicitly float64s, and checked for any constants in the code path. No luck.

@meta-cla meta-cla bot added the CLA Signed label Aug 8, 2025
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D79900113

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request Aug 19, 2025
@facebook-github-bot
Check errors in numpy >= 2 (facebookresearch#4523)
c73e917 
Summary:

Rollback Plan:

Differential Revision: D79900113
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D79900113

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request Aug 20, 2025
@facebook-github-bot
Check errors in numpy >= 2 (facebookresearch#4523)
a012c95 
Summary:

Rollback Plan:

Differential Revision: D79900113
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D79900113

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request Aug 20, 2025
@facebook-github-bot
Check errors in numpy >= 2 (facebookresearch#4523)
4e82dc8 
Summary:

Rollback Plan:

Differential Revision: D79900113
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D79900113

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request Aug 20, 2025
@facebook-github-bot
Check errors in numpy >= 2 (facebookresearch#4523)
a8d5ae6 
Summary:

Rollback Plan:

Differential Revision: D79900113
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D79900113

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request Aug 20, 2025
@facebook-github-bot
Check errors in numpy >= 2 (facebookresearch#4523)
d5a2a0f 
Summary:

Rollback Plan:

Differential Revision: D79900113
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D79900113

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request Aug 20, 2025
@facebook-github-bot
Check errors in numpy >= 2 (facebookresearch#4523)
36ce062 
Summary:

Rollback Plan:

Differential Revision: D79900113
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D79900113

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request Aug 20, 2025
@facebook-github-bot
Check errors in numpy >= 2 (facebookresearch#4523)
e2034e1 
Summary:

Rollback Plan:

Differential Revision: D79900113
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D79900113

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request Aug 21, 2025
@facebook-github-bot
Check errors in numpy >= 2 (facebookresearch#4523)
240591c 
Summary:

Rollback Plan:

Differential Revision: D79900113
@facebook-github-bot
Copy link
Contributor

This pull request was exported from Phabricator. Differential Revision: D79900113

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request Aug 21, 2025
@facebook-github-bot
Check errors in numpy >= 2 (facebookresearch#4523)
d424a18 
Summary:

Rollback Plan:

Differential Revision: D79900113
mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request Oct 1, 2025
@facebook-github-bot
Check errors in numpy >= 2 (facebookresearch#4523)
b5ac665 
Summary: Pull Request resolved: facebookresearch#4523

Differential Revision: D79900113
mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request Oct 1, 2025
@facebook-github-bot
Check errors in numpy >= 2 (facebookresearch#4523)
117f461 
Summary: Pull Request resolved: facebookresearch#4523

Differential Revision: D79900113
@mnorris11 mnorris11 force-pushed the export-D79900113 branch 2 times, most recently from 117f461 to 5d35899 Compare October 1, 2025 16:38
mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request Oct 1, 2025
@facebook-github-bot
Upgrade Faiss OSS side to numpy2 (facebookresearch#4523)
5d35899 
Summary:

This upgrades OSS Faiss to numpy 2.3.3. I tried to fix issues based on https://numpy.org/devdocs/numpy_2_0_migration_guide.html.

There are some issues:
-

Differential Revision: D79900113
@mnorris11 mnorris11 changed the title Check errors in numpy >= 2 Upgrade Faiss OSS side to numpy2 Oct 1, 2025
@mnorris11 mnorris11 mentioned this pull request Oct 1, 2025
Closed
4 tasks
@mnorris11 mnorris11 marked this pull request as ready for review October 1, 2025 18:02
jakirkham
jakirkham reviewed Oct 1, 2025
View reviewed changes
Copy link

@jakirkham jakirkham left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Michael! 🙏

Had a few suggestions on how to build with NumPy 2, which will make the packages more flexible for users installing them

Also these conda-forge NumPy docs may provide additional context

conda/faiss-gpu-cuvs/meta.yaml Outdated
- _openmp_mutex =4.5=2_kmp_llvm # [x86_64]
- python {{ python }}
- numpy >=1.19,<2
- numpy =2.3.3
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be relaxed to just ensure NumPy 2 is picked up

Suggested change
- numpy =2.3.3
- numpy >=2,<3.0a0

Though if you need a particular NumPy version, the lower bound could be updated

Suggested change
- numpy =2.3.3
- numpy >=2.3,<3.0a0

conda/faiss-gpu-cuvs/meta.yaml Outdated
- _openmp_mutex =4.5=2_kmp_llvm # [x86_64]
- python {{ python }}
- numpy >=1.19,<2
- numpy =2.3.3
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The numpy package already specifies run_exports that constrains the numpy version used at runtime (both in conda-forge and in defaults)

So this can just be either...

Suggested change
- numpy =2.3.3
- numpy

...or dropped entirely...

Suggested change
- numpy =2.3.3

conda/faiss-gpu-cuvs/meta.yaml Outdated
test:
requires:
- numpy >=1.19,<2
- numpy =2.3.3
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you want to constrain to NumPy 2 in tests, this could be done like so...

Suggested change
- numpy =2.3.3
- numpy >=2,<3.0a0

If you need the lower bound, then you could add it like so

Suggested change
- numpy =2.3.3
- numpy >=2.3,<3.0a0

conda/faiss-gpu/meta.yaml Outdated
- mkl =2023.0 # [x86_64]
- python {{ python }}
- numpy >=1.19,<2
- numpy =2.3.3
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would recommend the same changes in this and other files

@jakirkham
Copy link

The NumPy changes also look reasonable to my eyes. Though cc-ing @seberg in case I've missed anything

Also NumPy has a migration guide for NumPy 2. Additionally ruff has builtin support for migrating Python code to be NumPy 2 compliant, which can help automatically update code

@mnorris11
Copy link
Author

mnorris11 commented Oct 1, 2025 •
edited
Loading

Thanks Michael! 🙏

Had a few suggestions on how to build with NumPy 2, which will make the packages more flexible for users installing them

Also these conda-forge NumPy docs may provide additional context

Thanks @jakirkham for the review!
The flexible versions you suggest sound the best to me. This is the most user friendly.
We have experienced frequent nightly failures in the past when conda decides to pick up a different version (i.e. 2.1 instead of 2.3 etc), so by default I have been pinning exact versions. (I will change it to go with the flexible pinning for now, but we may have to go back to an exact version later.)

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request Oct 1, 2025
@facebook-github-bot
Upgrade Faiss OSS side to numpy2 (facebookresearch#4523)
0d88108 
Summary:

This upgrades OSS Faiss to numpy 2.3.3. I tried to fix issues based on https://numpy.org/devdocs/numpy_2_0_migration_guide.html.

Overall, numpy2 causes some test issues on various platforms/optimization levels.

Test changes that are OK:
- test_contrib.py: test_float: changing to use item(): this is just the new way of getting the value, like recommended in the migration guide.

Test changes that indicate some problem:
- test_contrib.py: test_ivf_train_2level: This fails on many platforms (avx2, conda installs). I could not find what the issue was. I had to increase the bounds slightly.
- test_index_accuracy.py: test_OIVFPQ: This fails just on AVX512. I had to increase the iterations from 10 to 12.
- test_local_search_quantizer.py and test_standalone_codec.py: Some of these break on Windows. Others needed lighter tolerances.

Debugging attempts: Especially for test_ivf_train_2level, I tried to unsuccessfully track down what might have changed. I suspected the type promotion change from numpy1 to 2.
```
- np.float32(3) + 3. now returns a float32 when it previously returned a float64.
- np.array([3], dtype=np.float32) + np.float64(3) will now return a float64 array. (The higher precision of the scalar is not ignored.)
```
I changed all datasets to be explicitly float64s, and checked for any constants in the code path. No luck.

Devmate was less than useless, it was actively a waste of time (as usual...)

Differential Revision: D79900113
mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request Oct 2, 2025
@facebook-github-bot
Upgrade Faiss OSS side to numpy2 (facebookresearch#4523)
5ee5802 
Summary:

This upgrades OSS Faiss to numpy 2.3.3. I tried to fix issues based on https://numpy.org/devdocs/numpy_2_0_migration_guide.html.

Overall, numpy2 causes some test issues on various platforms/optimization levels.

Test changes that are OK:
- test_contrib.py: test_float: changing to use item(): this is just the new way of getting the value, like recommended in the migration guide.

Test changes that indicate some problem:
- test_contrib.py: test_ivf_train_2level: This fails on many platforms (avx2, conda installs). I could not find what the issue was. I had to increase the bounds slightly.
- test_index_accuracy.py: test_OIVFPQ: This fails just on AVX512. I had to increase the iterations from 10 to 12.
- test_local_search_quantizer.py and test_standalone_codec.py: Some of these break on Windows. Others needed lighter tolerances.

Debugging attempts: Especially for test_ivf_train_2level, I tried to unsuccessfully track down what might have changed. I suspected the type promotion change from numpy1 to 2.
```
- np.float32(3) + 3. now returns a float32 when it previously returned a float64.
- np.array([3], dtype=np.float32) + np.float64(3) will now return a float64 array. (The higher precision of the scalar is not ignored.)
```
I changed all datasets to be explicitly float64s, and checked for any constants in the code path. No luck.

Devmate was less than useless, it was actively a waste of time (as usual...)

Differential Revision: D79900113
@meta-codesync Meta CodeSync
Copy link

meta-codesync bot commented Oct 2, 2025

@mnorris11 has exported this pull request. If you are a Meta employee, you can view the originating Diff in D79900113.

mnorris11 pushed a commit to mnorris11/faiss that referenced this pull request Oct 2, 2025
@facebook-github-bot
Upgrade Faiss OSS side to numpy2 (facebookresearch#4523)
ef1b1a7 
Summary:

This upgrades OSS Faiss to numpy 2.3.3. I tried to fix issues based on https://numpy.org/devdocs/numpy_2_0_migration_guide.html.

Overall, numpy2 causes some test issues on various platforms/optimization levels.

Test changes that are OK:
- test_contrib.py: test_float: changing to use item(): this is just the new way of getting the value, like recommended in the migration guide.

Test changes that indicate some problem:
- test_contrib.py: test_ivf_train_2level: This fails on many platforms (avx2, conda installs). I could not find what the issue was. I had to increase the bounds slightly.
- test_index_accuracy.py: test_OIVFPQ: This fails just on AVX512. I had to increase the iterations from 10 to 12.
- test_local_search_quantizer.py and test_standalone_codec.py: Some of these break on Windows. Others needed lighter tolerances.

Debugging attempts: Especially for test_ivf_train_2level, I tried to unsuccessfully track down what might have changed. I suspected the type promotion change from numpy1 to 2.
```
- np.float32(3) + 3. now returns a float32 when it previously returned a float64.
- np.array([3], dtype=np.float32) + np.float64(3) will now return a float64 array. (The higher precision of the scalar is not ignored.)
```
I changed all datasets to be explicitly float64s, and checked for any constants in the code path. No luck.

Devmate was less than useless, it was actively a waste of time (as usual...)

Differential Revision: D79900113
@facebook-github-bot
Upgrade Faiss OSS side to numpy2 (facebookresearch#4523)
afda341 
Summary:

This upgrades OSS Faiss to numpy 2.3.3. I tried to fix issues based on https://numpy.org/devdocs/numpy_2_0_migration_guide.html.

Overall, numpy2 causes some test issues on various platforms/optimization levels.

Test changes that are OK:
- test_contrib.py: test_float: changing to use item(): this is just the new way of getting the value, like recommended in the migration guide.

Test changes that indicate some problem:
- test_contrib.py: test_ivf_train_2level: This fails on many platforms (avx2, conda installs). I could not find what the issue was. I had to increase the bounds slightly.
- test_index_accuracy.py: test_OIVFPQ: This fails just on AVX512. I had to increase the iterations from 10 to 12.
- test_local_search_quantizer.py and test_standalone_codec.py: Some of these break on Windows. Others needed lighter tolerances.

Debugging attempts: Especially for test_ivf_train_2level, I tried to unsuccessfully track down what might have changed. I suspected the type promotion change from numpy1 to 2.
```
- np.float32(3) + 3. now returns a float32 when it previously returned a float64.
- np.array([3], dtype=np.float32) + np.float64(3) will now return a float64 array. (The higher precision of the scalar is not ignored.)
```
I changed all datasets to be explicitly float64s, and checked for any constants in the code path. No luck.

Devmate was less than useless, it was actively a waste of time (as usual...)

Differential Revision: D79900113
@seberg
Copy link

seberg commented Oct 2, 2025

Looks fine, cannot say quickly why the item calls are needed, but probably OK. Tolerance changes may mean that some internal computation previously went to float64 and now stay at float32, I guess. But that is presumably also fine.

@meta-codesync meta-codesync bot closed this in 4fab13c Oct 2, 2025
@facebook-github-bot
Copy link
Contributor

This pull request has been merged in 4fab13c.

@jakirkham
Copy link

Thanks @jakirkham for the review!
The flexible versions you suggest sound the best to me. This is the most user friendly.
We have experienced frequent nightly failures in the past when conda decides to pick up a different version (i.e. 2.1 instead of 2.3 etc), so by default I have been pinning exact versions. (I will change it to go with the flexible pinning for now, but we may have to go back to an exact version later.)

Happy to help 🙂

Thanks Michael! 🙏

If you run into issues, please let me know. Would be interested in taking a look at those when they come up

jakirkham
jakirkham reviewed Oct 2, 2025
View reviewed changes
.github/actions/build_cmake/action.yml
# Ensure starting packages are from conda-forge.
conda list --show-channel-urls
conda update -y -q conda
conda install -y -q "conda<=25.07"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Guessing this is due to an issue with conda version 25.09, do you have more details on the issue seen?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, seems like it just surfaced. Prior runs used 25.07 so I just pinned there.
https://github.com/facebookresearch/faiss/actions/runs/18175470029/job/51741940955
The last logs are:

Error while loading conda entry point: conda-libmamba-solver (cannot import name 'Spinner' from 'conda.common.io' (/home/runner/miniconda3/lib/python3.12/site-packages/conda/common/io.py))

CondaValueError: You have chosen a non-default solver backend (libmamba) but it was not recognized. Choose one of: classic

Error: Process completed with exit code 1.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Michael! 🙏

Think I can see what is going on. Have filed an upstream issue: conda/conda#15287

Also to help with debugging future Conda or conda-forge issues, have suggested a few commands we can add to CI in issue: #4599

This would help in providing additional detail when debugging. Additionally this would help in upstream bug reports

@jakirkham jakirkham mentioned this pull request Oct 2, 2025
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
1 more reviewer

@jakirkham jakirkham jakirkham left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
CLA Signed fb-exported ignore Merged meta-exported
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@mnorris11 @facebook-github-bot @jakirkham @seberg