Thanks to visit codestin.com
Credit goes to github.com

Skip to content

ENH reduce memory usage in make_blobs - #22412

Merged
glemaitre merged 6 commits into
scikit-learn:mainfrom
MaxwellLZH:enh/make-blob
Feb 10, 2022
Merged

ENH reduce memory usage in make_blobs#22412
glemaitre merged 6 commits into
scikit-learn:mainfrom
MaxwellLZH:enh/make-blob

Conversation

@MaxwellLZH

Copy link
Copy Markdown
Contributor

Reference Issues/PRs

This is a fix to issue #22244.

What does this implement/fix? Explain your changes.

Reduce memory usage in make_blobs by preallocating empty NumPy array instead of concatenating, as suggested by @glemaitre.

Also adds a test case, using memory_profiler to track the memory usage of the function.

@jeremiedbb jeremiedbb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @MaxwellLZH. Implementation looks good. I'm not sure if we want to begin testing memory usage in the test suite, I'm +0 for this addition. What's sure is that we never install memory_profiler in our CI so currently this test would never be run. We'd need to install it in at least one CI job.

Comment thread sklearn/datasets/_samples_generator.py
Comment thread sklearn/datasets/tests/test_samples_generator.py Outdated
Comment thread sklearn/datasets/tests/test_samples_generator.py Outdated
Comment thread sklearn/datasets/tests/test_samples_generator.py Outdated
@glemaitre

Copy link
Copy Markdown
Member

I think that we can avoid adding a test to check the performance. Instead, @ogrisel and @jeremiedbb think that we could add such utility in the ASV benchmark suite.

@MaxwellLZH Could you still add an entry in the changelog to acknowledge the no-copy behaviour.

Comment thread sklearn/datasets/_samples_generator.py Outdated
Co-authored-by: Jérémie du Boisberranger <[email protected]>

@jeremiedbb jeremiedbb left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks @MaxwellLZH !

@jeremiedbb jeremiedbb added the Quick Review For PRs that are quick to review label Feb 10, 2022
@glemaitre
glemaitre merged commit 0de3b0d into scikit-learn:main Feb 10, 2022
@glemaitre

Copy link
Copy Markdown
Member

Thanks @MaxwellLZH

@MaxwellLZH

Copy link
Copy Markdown
Contributor Author

Hi @glemaitre @jeremiedbb , I noticed that we can also improve the memory usage for datasets.make_s_curve in a similar way:

# before
x = np.sin(t)
y = 2.0 * generator.uniform(size=(1, n_samples))
z = np.sign(t) * (np.cos(t) - 1)
X = np.concatenate((x, y, z))

# after
X = np.empty(shape=(n_samples, 3), dtype=np.float64)
X[:, 0] = np.sin(t)
X[:, 1] = 2.0 * generator.uniform(size=n_samples)
X[:, 2] = np.sign(t) * (np.cos(t) - 1)

I've tested on my machine this indeed uses less memory (from 694MiB to 541MiB when n_samples = 10000000), shall I open a separate PR for this ?

@glemaitre

Copy link
Copy Markdown
Member

I've tested on my machine this indeed uses less memory (from 694MiB to 541MiB when n_samples = 10000000), shall I open a separate PR for this ?

Yes, it is better to do in a separate PR. Feel free to open one.

thomasjpfan pushed a commit to thomasjpfan/scikit-learn that referenced this pull request Mar 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

module:datasets Quick Review For PRs that are quick to review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants