Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Updating pylibraft pairwise_distance to cuvs#8847

Merged
kmaehashi merged 28 commits into
cupy:mainfrom
cjnolet:update_pylibraft_to_cuvs
Jan 23, 2025
Merged

Updating pylibraft pairwise_distance to cuvs#8847
kmaehashi merged 28 commits into
cupy:mainfrom
cjnolet:update_pylibraft_to_cuvs

Conversation

@cjnolet

@cjnolet cjnolet commented Dec 19, 2024 •
edited
Loading

Copy link
Copy Markdown
Member

This PR updates cupy so that it can make use of the cuvs library for pairwise distance, which the vector search APIs from pylibraft have recently moved. This PR maintains the backwards compatibility with pylibraft while will exposing double precision float support.

Closes NVIDIA/cuvs#542

bdice
bdice reviewed Dec 19, 2024
View reviewed changes
Comment thread tests/cupyx_tests/scipy_tests/spatial_tests/test_kdtree_cuvs.py Outdated
@leofang

leofang commented Dec 20, 2024

Copy link
Copy Markdown
Member

I assume these two files will need to change too, in order to test this in the CI:

@leofang leofang added the to-be-backported Pull-requests to be backported to stable branch label Dec 20, 2024
@kmaehashi kmaehashi self-assigned this Dec 20, 2024
@kmaehashi kmaehashi added cat:enhancement Improvements to existing features prio:high labels Dec 20, 2024
@kmaehashi

kmaehashi commented Dec 20, 2024

Copy link
Copy Markdown
Member

Thanks @cjnolet, LGTM!

Could you bump the base Docker image to the one with cuVS (I guess 24.12-cuda12.0-py3.10?) Then /test rapids can trigger the CI.

@dpshepherd dpshepherd mentioned this pull request Dec 21, 2024
Merged
@cjnolet

cjnolet commented Jan 7, 2025

Copy link
Copy Markdown
Member Author

/test rapids

@cjnolet

cjnolet commented Jan 14, 2025

Copy link
Copy Markdown
Member Author

/test rapids

1 similar comment
@cjnolet

cjnolet commented Jan 14, 2025

Copy link
Copy Markdown
Member Author

/test rapids

@cjnolet

cjnolet commented Jan 14, 2025

Copy link
Copy Markdown
Member Author

/test rapids

@cjnolet

cjnolet commented Jan 14, 2025

Copy link
Copy Markdown
Member Author

/test rapids

@cjnolet

cjnolet commented Jan 14, 2025

Copy link
Copy Markdown
Member Author

/test rapids

@cjnolet

cjnolet commented Jan 15, 2025

Copy link
Copy Markdown
Member Author

@kmaehashi I'm confused about an error I'm getting in CI:

[00:12:02.957337 STDOUT 1477]](https://ci.preferred.jp/cupy.linux.cuda-rapids/182672/#L683)	#7 [2/2] RUN export DEBIAN_FRONTEND=noninteractive &&     apt-get -qqy update &&     apt-get -qqy install build-essential cuda-toolkit-12-0 ccache git curl	4.3s
[00:12:07.305510 STDOUT 1477]](https://ci.preferred.jp/cupy.linux.cuda-rapids/182672/#L685)	#7 4.498 W: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.	1.1s
[00:12:08.370260 STDOUT 1477]](https://ci.preferred.jp/cupy.linux.cuda-rapids/182672/#L687)	#7 5.556 E: Unable to locate package cuda-toolkit-12-0

It started happening right after updating the link to the docker image (which uses 12.5 for all cuda 12 versions), but I'm not sure if it's related or if just a coincidence.

@cjnolet

cjnolet commented Jan 15, 2025

Copy link
Copy Markdown
Member Author

/test rapids

@cjnolet

cjnolet commented Jan 15, 2025

Copy link
Copy Markdown
Member Author

/test rapids

@cjnolet

cjnolet commented Jan 15, 2025

Copy link
Copy Markdown
Member Author

/test rapids

@cjnolet

cjnolet commented Jan 15, 2025

Copy link
Copy Markdown
Member Author

/test rapids

@cjnolet

cjnolet commented Jan 15, 2025

Copy link
Copy Markdown
Member Author

/test rapids

@cjnolet

cjnolet commented Jan 15, 2025

Copy link
Copy Markdown
Member Author

Okay, I've got CI running successfully and all tests are passing. Last piece here is waiting unti this cuVS PR is merged and the nightly is deployed: NVIDIA/cuvs#570. Then we can merge this PR.

@cjnolet

cjnolet commented Jan 16, 2025

Copy link
Copy Markdown
Member Author

/test rapids

@cjnolet

cjnolet commented Jan 16, 2025

Copy link
Copy Markdown
Member Author

/test rapids

@cjnolet

cjnolet commented Jan 17, 2025

Copy link
Copy Markdown
Member Author

@kmaehashi this is ready to merge whenever you are ready. Thanks so much!

@ramakrishnap-nv

ramakrishnap-nv commented Jan 22, 2025

Copy link
Copy Markdown

@kmaehashi Shall we merge this PR ?

@cjnolet

cjnolet commented Jan 22, 2025

Copy link
Copy Markdown
Member Author

/merge

@cjnolet

cjnolet commented Jan 22, 2025

Copy link
Copy Markdown
Member Author

@kmaehashi just a friendly ping. I don't think I have the power to merge this PR. Any chance we could get it merged?

@kmaehashi kmaehashi merged commit 8ce9ebe into cupy:main Jan 23, 2025
@kmaehashi

kmaehashi commented Jan 23, 2025

Copy link
Copy Markdown
Member

Sorry for the delay, thanks again!

@chainer-ci

chainer-ci commented Jan 23, 2025

Copy link
Copy Markdown
Member

@kmaehashi Failed to backport automatically.

Not cleanly cherry-picked

@kmaehashi kmaehashi added this to the v14.0.0a1 milestone Jan 23, 2025
@kmaehashi

kmaehashi commented Jan 23, 2025

Copy link
Copy Markdown
Member

I'll take care of backport to v13

kmaehashi added a commit to kmaehashi/cupy that referenced this pull request Jan 23, 2025
@kmaehashi
Merge pull request cupy#8847 from cjnolet/update_pylibraft_to_cuvs
28fcea1 
Updating pylibraft `pairwise_distance` to cuvs
@kmaehashi kmaehashi mentioned this pull request Jan 23, 2025
Merged
@kmaehashi kmaehashi mentioned this pull request Feb 26, 2025
Merged
@kmaehashi kmaehashi mentioned this pull request Jun 26, 2025
Merged
@kmaehashi kmaehashi mentioned this pull request Aug 15, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kmaehashi kmaehashi kmaehashi approved these changes
+1 more reviewer
@bdice bdice bdice left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@kmaehashi kmaehashi

Labels

cat:enhancement Improvements to existing features prio:high to-be-backported Pull-requests to be backported to stable branch

Projects

None yet

Milestone

v14.0.0a1

Development

Successfully merging this pull request may close these issues.

[BUG] Update cuPy to use cuvs.distance

6 participants

@cjnolet @leofang @kmaehashi @ramakrishnap-nv @chainer-ci @bdice