Thanks to visit codestin.com
Credit goes to github.com

Skip to content

ENH reduce memory consumption in nan_euclidean_distances #15615

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Nov 18, 2019
Merged

ENH reduce memory consumption in nan_euclidean_distances #15615

merged 2 commits into from
Nov 18, 2019

Conversation

jnothman
Copy link
Member

@jnothman jnothman commented Nov 13, 2019 •
edited
Loading

This goes towards fixing #15604

Basic benchmark:

import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.metrics.pairwise import nan_euclidean_distances

calhousing = fetch_california_housing()

X = pd.DataFrame(calhousing.data, columns=calhousing.feature_names)
y = pd.Series(calhousing.target, name='house_value')

rng = np.random.RandomState(42)

density = 4  # one in 10 values will be NaN

mask = rng.randint(density, size=X.shape) == 0
X_na = X.copy()
X_na.values[mask] = np.nan

%time nan_euclidean_distances(X_na)

At master:

CPU times: user 33 s, sys: 9.42 s, total: 42.4 s
Wall time: 24 s

This branch:

CPU times: user 27.3 s, sys: 3.71 s, total: 31 s
Wall time: 14.8 s

Faster times could possibly be achieved with chunking.

@jnothman
Copy link
Member Author

The test failure is because of this expected distance matrix:

        X = np.array([[missing_value, missing_value], [0, 1]])
        exp_dist = np.array([[np.nan, np.nan], [np.nan, 0]])

This sets the standard that if a sample is all-nan, its euclidean distance to itself is nan rather than 0. My code currently sets the diagonal to 0 in all cases. What do we consider to be the right behaviour?

@thomasjpfan
Copy link
Member

Originally I had the diagonal at zero (when X is Y), but I was concerned with how the following were not equal:

X = np.array([[missing_value, missing_value], [0, 1]])

nan_euclidean_distances(X, X.copy())
nan_euclidean_distances(X, X)

@thomasjpfan thomasjpfan self-requested a review November 14, 2019 15:53
@glemaitre glemaitre self-requested a review November 14, 2019 17:50
@glemaitre
Copy link
Member

What do we consider to be the right behaviour?

At least it is documented nan.

Since the distance is a bit specific to handle nan, I would not be surprised that it returns nan while I would be surprised to have some nan in other metrics due to division by zero or stuff like that.

@jnothman Do you have a use case or any thought on what it should be 0 instead of nan.

@glemaitre
Copy link
Member

Apart from this LGTM

@glemaitre glemaitre mentioned this pull request Nov 15, 2019
Closed
@jnothman
Copy link
Member Author

jnothman commented Nov 17, 2019 via email

glemaitre
glemaitre approved these changes Nov 17, 2019
View reviewed changes
Copy link
Member

@glemaitre glemaitre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So LGTM. @thomasjpfan Do you want to give a look.

@thomasjpfan thomasjpfan merged commit f7ed72a into scikit-learn:master Nov 18, 2019
@thomasjpfan
Copy link
Member

Thank you @jnothman !

adrinjalali pushed a commit to adrinjalali/scikit-learn that referenced this pull request Nov 25, 2019
@jnothman @adrinjalali
ENH reduce memory consumption in nan_euclidean_distances (scikit-lear…
c8b0dc3 
…n#15615)
jnothman added a commit that referenced this pull request Nov 28, 2019
@jnothman
ENH reduce memory consumption in nan_euclidean_distances (#15615)
ddfc592
panpiort8 pushed a commit to panpiort8/scikit-learn that referenced this pull request Mar 3, 2020
@jnothman
ENH reduce memory consumption in nan_euclidean_distances (scikit-lear…
aa1558e 
…n#15615)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@thomasjpfan thomasjpfan thomasjpfan approved these changes

@glemaitre glemaitre glemaitre approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jnothman @thomasjpfan @glemaitre