MAINT Introduce Pairwise Distances Reductions private submodule #22064

jjerphan · 2021-12-23T09:26:42Z

Reference Issues/PRs

⚠ This targets the upstream:pairwise-distances-argkmin feature branch not upstream:main.

What does this implement/fix? Explain your changes.

This introduces the necessary private implementations for a new
private submodule, i.e.:

DatasetsPair, an abstraction to wrap a pair of two datasets and
compute their vectors pairwise distances
DenseDenseDatasetsPair, a first implementation of DatasetsPair
for pair of two dense datasets
PairwiseDistancesReduction, an abstraction allowing computing
reductions efficiently in parallel and of
PairwiseDistancesArgkmin, a first implementation of
PairwiseDistancesReduction for k-Nearest Neighbors search

This introduces the neccessary private implementations for a new private submodule, i.e.: - DatasetsPair, an abstraction to wrap a pair of two datasets and compute their vectors pairwise distances - DenseDenseDatasetsPair, a first implementation of DatasetsPair for pair of two dense datasets - PairwiseDistancesReduction, an abstraction allowing computing reductions efficiently in parallel and of - PairwiseDistancesArgkmin, a first implementation of PairwiseDistancesReduction for k-Nearest Neighbors search

Co-authored-by: Thomas J. Fan <[email protected]>

lorentzenchr

A review round about _dist_metrics.pyx.

sklearn/metrics/_dist_metrics.pyx

lorentzenchr

Review on setup:wink:

sklearn/metrics/setup.py

Co-authored-by: Christian Lorentzen <[email protected]>

lorentzenchr

A pass on cdef class PairwiseDistancesReduction.

sklearn/utils/__init__.py

sklearn/metrics/_pairwise_distances_reduction.pyx

Co-authored-by: Christian Lorentzen <[email protected]>

This was introduced once a long time ago for a failure which was happening in a single configuration (see the comments). Let's see if this has been fixed. Co-authored-by: Christian Lorentzen <[email protected]>

Co-authored-by: Christian Lorentzen <[email protected]> Co-authored-by: Thomas J. Fan <[email protected]>

jjerphan

Add "I-hope-useful" precision for some comments regarding parallel sections.

sklearn/metrics/_pairwise_distances_reduction.pyx

ogrisel

Here is a batch of suggestions to help with https://github.com/scikit-learn/scikit-learn/pull/22064/files#r774754046.

If the changes suggested to improve the docstring on the strategy parameter are accepted, the _parallel_on_X and _parallel_on_Y docstrings should be updated accordingly.

sklearn/metrics/_pairwise_distances_reduction.pyx

Co-authored-by: Olivier Grisel <[email protected]>

ogrisel

Another pass of review. Once addressed (especially the comment on the tests), I think this PR LGTM.

sklearn/metrics/_pairwise_distances_reduction.pyx

sklearn/metrics/tests/test_pairwise_distances_reduction.py

sklearn/metrics/_pairwise_distances_reduction.pyx

sklearn/metrics/_dist_metrics.pyx

thomasjpfan

Minor comments, but I think this is ready to go.

There are two paths for follow up:

FastEuclideanPairwiseArgKmin
Interfacing implementation with scikit-learn interfaces

I am in favor of option 2 first, because with option 2 completed, we are merged directly into main. FastEuclideanPairwiseArgKmin would be a follow up performance improvement.

sklearn/metrics/_pairwise_distances_reduction.pyx

Co-authored-by: Thomas J. Fan <[email protected]> Co-authored-by: Olivier Grisel <[email protected]>

Co-authored-by: Thomas J. Fan <[email protected]>

To have argkmin_indices always be the first for consistency. Co-authored-by: Olivier Grisel <[email protected]>

This is more appropriate, especially for dynamic allocation. See: cython.readthedocs.io/en/latest/src/userguide/special_methods.html#initialisation-methods-cinit-and-init The __cinit__() method is where you should perform basic C-level initialisation of the object, including allocation of any C data structures that your object will own. Co-authored-by: Thomas J. Fan <[email protected]>

ogrisel · 2022-01-04T14:29:02Z

FastEuclideanPairwiseArgKmin would be a follow up performance improvement.

It might actually be needed to avoid a performance regression on machines with few cores as the current numpy code already uses the GEMM trick.

ogrisel

+1 for merge once the tests have been updated to remove the translation invariance check that is too complicated to get working in general and replaced by a unit test to compare the ArgKMin implementation (with both parallelism strategies) with an naive implementation based on cdist/argpartition on small-ish random data as discussed in https://github.com/scikit-learn/scikit-learn/pull/22064/files#r777541915

sklearn/metrics/_pairwise_distances_reduction.pyx

Co-authored-by: Olivier Grisel <[email protected]>

jjerphan · 2022-01-05T09:08:18Z

Minor comments, but I think this is ready to go.

There are two paths for follow up:
1. FastEuclideanPairwiseArgKmin

2. Interfacing implementation with scikit-learn interfaces
I am in favor of option 2 first, because with option 2 completed, we are merged directly into main. FastEuclideanPairwiseArgKmin would be a follow up performance improvement.

So do I. 👍

ogrisel · 2022-01-05T09:22:16Z

@jjerphan here is what I had in mind for the test: jjerphan#7

…ance

lorentzenchr

A typo from an unfinished further review pass. Thanks @ogrisel and @thomasjpfan for finishing your reviews. I got busy these days.

sklearn/metrics/_pairwise_distances_reduction.pyx

lorentzenchr · 2022-01-05T13:12:09Z

@jjerphan The failing CI due to several DeprecationWarning is maybe the missing piece before merge. Looking forward to it!

… invariance

Co-authored-by: Christian Lorentzen <[email protected]>

thomasjpfan · 2022-01-05T14:12:08Z

It might actually be needed to avoid a performance regression on machines with few cores as the current numpy code already uses the GEMM trick.

If there is a performance regression, then I agree that FastEuclideanPairwiseArgKmin should come first.

ogrisel · 2022-01-05T14:19:35Z

CI is blue! Merging!

…min` (feature branch) (#22134) * MAINT Introduce Pairwise Distances Reductions private submodule (#22064) * MAINT Introduce FastEuclideanPairwiseArgKmin (#22065) * fixup! Merge branch 'main' into pairwise-distances-argkmin Remove duplicated Bunch * MAINT Plug `PairwiseDistancesArgKmin` as a back-end (#22288) * Forward pairwise_dist_chunk_size in the configuration * Flip finalized results for PairwiseDistancesArgKmin The previous would have made the code more complex by introducing some boilerplate for the interface plugs. Having it this way actually simplifies the code. This also removes the haversine branch for test_pairwise_distances_argkmin * Plug PairwiseDistancesArgKmin as a back-end * Adapt test accordingly * Add whats_new entry * Change input validation order for kneighbors * Remove duplicated test_neighbors_distance_metric_deprecation * Adapt the documentation * Add mahalanobis case to test fixtures * Correct whats_new entry * CLN Remove unneeded private metric attribute This was needed when 'fast_sqeuclidean' and 'fast_euclidean' were present to choose the best implementation based on the user specification. Those metric have been removed since then, making this attribute useless. * TST Assert FutureWarning instead of DeprecationWarning in test_neighbors_metrics * MAINT Add use_pairwise_dist_activate to scikit-learn config * TST Add a test for the 'brute' backends' results' consistency Co-authored-by: Thomas J. Fan <[email protected]> * fixup! MAINT Add use_pairwise_dist_activate to scikit-learn config * fixup! fixup! MAINT Add use_pairwise_dist_activate to scikit-learn config * TST Filter FutureWarning for WMinkowskiDistance * MAINT pin numpydoc in arm for now (#22292) * fixup! TST Filter FutureWarning for WMinkowskiDistance * Revert keywords arguments removal for the GEMM trick for 'euclidean' * MAINT pin max numpydoc for now (#22286) * Add 'haversine' to CDIST_PAIRWISE_DISTANCES_REDUCTION_COMMON_METRICS * fixup! Add 'haversine' to CDIST_PAIRWISE_DISTANCES_REDUCTION_COMMON_METRICS * Apply suggestions from code review * MAINT Document some config parameters for maintenance Also rename one of them. * FIX Support and test one of 'sqeuclidean' specification Co-authored-by: Olivier Grisel <[email protected]> * FIX Various typos fix and correct haversine 'haversine' is not supported by cdist. * Directly use get_config * CLN Apply comments from review * Motivate swapped returned values * TST Remove mahalanobis from test fixtures * MNT Add comment regaduction functions' signatures * TST Complete test for `pairwise_distance_{argmin,argmin_min}` (#22371) * DOC Add sub-pull requests to the whats_new entry * DOC place comment inside functions * DOC move up whatsnew entry Co-authored-by: Thomas J. Fan <[email protected]> Co-authored-by: Christian Lorentzen <[email protected]> Co-authored-by: Olivier Grisel <[email protected]> Co-authored-by: Jérémie du Boisberranger <[email protected]>

…min` (feature branch) (scikit-learn#22134) * MAINT Introduce Pairwise Distances Reductions private submodule (scikit-learn#22064) * MAINT Introduce FastEuclideanPairwiseArgKmin (scikit-learn#22065) * fixup! Merge branch 'main' into pairwise-distances-argkmin Remove duplicated Bunch * MAINT Plug `PairwiseDistancesArgKmin` as a back-end (scikit-learn#22288) * Forward pairwise_dist_chunk_size in the configuration * Flip finalized results for PairwiseDistancesArgKmin The previous would have made the code more complex by introducing some boilerplate for the interface plugs. Having it this way actually simplifies the code. This also removes the haversine branch for test_pairwise_distances_argkmin * Plug PairwiseDistancesArgKmin as a back-end * Adapt test accordingly * Add whats_new entry * Change input validation order for kneighbors * Remove duplicated test_neighbors_distance_metric_deprecation * Adapt the documentation * Add mahalanobis case to test fixtures * Correct whats_new entry * CLN Remove unneeded private metric attribute This was needed when 'fast_sqeuclidean' and 'fast_euclidean' were present to choose the best implementation based on the user specification. Those metric have been removed since then, making this attribute useless. * TST Assert FutureWarning instead of DeprecationWarning in test_neighbors_metrics * MAINT Add use_pairwise_dist_activate to scikit-learn config * TST Add a test for the 'brute' backends' results' consistency Co-authored-by: Thomas J. Fan <[email protected]> * fixup! MAINT Add use_pairwise_dist_activate to scikit-learn config * fixup! fixup! MAINT Add use_pairwise_dist_activate to scikit-learn config * TST Filter FutureWarning for WMinkowskiDistance * MAINT pin numpydoc in arm for now (scikit-learn#22292) * fixup! TST Filter FutureWarning for WMinkowskiDistance * Revert keywords arguments removal for the GEMM trick for 'euclidean' * MAINT pin max numpydoc for now (scikit-learn#22286) * Add 'haversine' to CDIST_PAIRWISE_DISTANCES_REDUCTION_COMMON_METRICS * fixup! Add 'haversine' to CDIST_PAIRWISE_DISTANCES_REDUCTION_COMMON_METRICS * Apply suggestions from code review * MAINT Document some config parameters for maintenance Also rename one of them. * FIX Support and test one of 'sqeuclidean' specification Co-authored-by: Olivier Grisel <[email protected]> * FIX Various typos fix and correct haversine 'haversine' is not supported by cdist. * Directly use get_config * CLN Apply comments from review * Motivate swapped returned values * TST Remove mahalanobis from test fixtures * MNT Add comment regaduction functions' signatures * TST Complete test for `pairwise_distance_{argmin,argmin_min}` (scikit-learn#22371) * DOC Add sub-pull requests to the whats_new entry * DOC place comment inside functions * DOC move up whatsnew entry Co-authored-by: Thomas J. Fan <[email protected]> Co-authored-by: Christian Lorentzen <[email protected]> Co-authored-by: Olivier Grisel <[email protected]> Co-authored-by: Jérémie du Boisberranger <[email protected]>

github-actions bot added module:metrics module:utils cython labels Dec 23, 2021

jjerphan added the No Changelog Needed label Dec 23, 2021

jjerphan mentioned this pull request Dec 23, 2021

MAINT Introduce FastEuclideanPairwiseArgKmin #22065

Merged

Retrigger CI for failing Circle CI job

14106c4

jjerphan mentioned this pull request Dec 23, 2021

ENH Pairwise Distances ArgKmin #21462

Closed

TST Improve _get_dummy_metric_params_list

3cdd3a5

Co-authored-by: Thomas J. Fan <[email protected]>

jjerphan requested review from ogrisel and thomasjpfan December 23, 2021 10:17

lorentzenchr reviewed Dec 23, 2021

View reviewed changes

sklearn/metrics/setup.py Outdated Show resolved Hide resolved

Address review comments

31db785

Co-authored-by: Christian Lorentzen <[email protected]>

lorentzenchr reviewed Dec 23, 2021

View reviewed changes

Address review comments

b60e897

Co-authored-by: Christian Lorentzen <[email protected]>

jjerphan force-pushed the pairwise-distances-argkmin-necessary-implementations branch from 9ba06e4 to bd3f8fe Compare December 23, 2021 16:39

DEBUG TST Try removing handling of unstable OpenBLAS configuration

5d7ea09

This was introduced once a long time ago for a failure which was happening in a single configuration (see the comments). Let's see if this has been fixed. Co-authored-by: Christian Lorentzen <[email protected]>

jjerphan force-pushed the pairwise-distances-argkmin-necessary-implementations branch from bd3f8fe to 5d7ea09 Compare December 23, 2021 16:43

jjerphan and others added 2 commits December 23, 2021 18:28

TST Remove useless mahalanobis case

fb927e7

Factor the logic for computing last chunks indices

a2f7b6d

Co-authored-by: Christian Lorentzen <[email protected]> Co-authored-by: Thomas J. Fan <[email protected]>

jjerphan commented Jan 3, 2022

View reviewed changes

ogrisel reviewed Jan 3, 2022

View reviewed changes

Improve comments regarding strategies and parallel sections

e9acef7

Co-authored-by: Olivier Grisel <[email protected]>

ogrisel reviewed Jan 3, 2022

View reviewed changes

thomasjpfan approved these changes Jan 3, 2022

View reviewed changes

sklearn/metrics/_pairwise_distances_reduction.pyx Outdated Show resolved Hide resolved

sklearn/metrics/_pairwise_distances_reduction.pyx Show resolved Hide resolved

sklearn/metrics/_pairwise_distances_reduction.pyx Outdated Show resolved Hide resolved

jjerphan and others added 4 commits January 4, 2022 11:12

Address reviews' comments

51dad2b

Co-authored-by: Thomas J. Fan <[email protected]> Co-authored-by: Olivier Grisel <[email protected]>

Remove unused _sqeuclidean_row_norms

59b153c

Co-authored-by: Thomas J. Fan <[email protected]>

Swap argkmin_indices and argkmin_distances

395f92a

To have argkmin_indices always be the first for consistency. Co-authored-by: Olivier Grisel <[email protected]>

ogrisel approved these changes Jan 4, 2022

View reviewed changes

sklearn/metrics/_pairwise_distances_reduction.pyx Show resolved Hide resolved

sklearn/metrics/_pairwise_distances_reduction.pyx Outdated Show resolved Hide resolved

sklearn/metrics/_pairwise_distances_reduction.pyx Outdated Show resolved Hide resolved

ogrisel reviewed Jan 4, 2022

View reviewed changes

sklearn/metrics/_pairwise_distances_reduction.pyx Outdated Show resolved Hide resolved

jjerphan and others added 4 commits January 5, 2022 08:30

Improve docstring comment

f396a58

Co-authored-by: Olivier Grisel <[email protected]>

Improve comments

22f4f30

Co-authored-by: Olivier Grisel <[email protected]>

Add 'pairwise_dist_chunk_size' to scikit-learn config

234fb01

TST Adapt test for PairwiseDistancesArgKmin translation invariance

f89c65d

Co-authored-by: Olivier Grisel <[email protected]>

test_pairwise_distances_argkmin

fe17af1

ogrisel and others added 3 commits January 5, 2022 10:24

Simpler variable names

38715d2

Merge pull request #7 from ogrisel/test_pairwise_distances_argkmin

a461355

fixup! TST Adapt test for PairwiseDistancesArgKmin translation invari…

ce986d5

…ance

lorentzenchr reviewed Jan 5, 2022

View reviewed changes

sklearn/metrics/_pairwise_distances_reduction.pyx Outdated Show resolved Hide resolved

jjerphan and others added 2 commits January 5, 2022 14:28

fixup! fixup! TST Adapt test for PairwiseDistancesArgKmin translation…

70a28b7

… invariance

Use correct orthograph for 'Callback'

06ca869

Co-authored-by: Christian Lorentzen <[email protected]>

ogrisel merged commit 0532aab into scikit-learn:pairwise-distances-argkmin Jan 5, 2022

jjerphan deleted the pairwise-distances-argkmin-necessary-implementations branch January 5, 2022 14:37

jjerphan mentioned this pull request Feb 10, 2022

ENH Introduce PairwiseDistancesReduction and PairwiseDistancesArgKmin (feature branch) #22134

Merged

10 tasks

Uh oh!

MAINT Introduce Pairwise Distances Reductions private submodule #22064

MAINT Introduce Pairwise Distances Reductions private submodule #22064

Uh oh!

Conversation

jjerphan commented Dec 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Uh oh!

lorentzenchr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lorentzenchr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lorentzenchr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jjerphan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thomasjpfan left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ogrisel commented Jan 4, 2022

Uh oh!

ogrisel left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

jjerphan commented Dec 23, 2021 •

edited

Loading

thomasjpfan left a comment •

edited

Loading

ogrisel left a comment •

edited

Loading