ENH reduce memory usage in make_blobs - #22412
Conversation
jeremiedbb
left a comment
There was a problem hiding this comment.
Thanks @MaxwellLZH. Implementation looks good. I'm not sure if we want to begin testing memory usage in the test suite, I'm +0 for this addition. What's sure is that we never install memory_profiler in our CI so currently this test would never be run. We'd need to install it in at least one CI job.
|
I think that we can avoid adding a test to check the performance. Instead, @ogrisel and @jeremiedbb think that we could add such utility in the ASV benchmark suite. @MaxwellLZH Could you still add an entry in the changelog to acknowledge the no-copy behaviour. |
Co-authored-by: Jérémie du Boisberranger <[email protected]>
jeremiedbb
left a comment
There was a problem hiding this comment.
LGTM. Thanks @MaxwellLZH !
|
Thanks @MaxwellLZH |
|
Hi @glemaitre @jeremiedbb , I noticed that we can also improve the memory usage for # before
x = np.sin(t)
y = 2.0 * generator.uniform(size=(1, n_samples))
z = np.sign(t) * (np.cos(t) - 1)
X = np.concatenate((x, y, z))
# after
X = np.empty(shape=(n_samples, 3), dtype=np.float64)
X[:, 0] = np.sin(t)
X[:, 1] = 2.0 * generator.uniform(size=n_samples)
X[:, 2] = np.sign(t) * (np.cos(t) - 1)I've tested on my machine this indeed uses less memory (from 694MiB to 541MiB when |
Yes, it is better to do in a separate PR. Feel free to open one. |
Co-authored-by: Jérémie du Boisberranger <[email protected]>
Reference Issues/PRs
This is a fix to issue #22244.
What does this implement/fix? Explain your changes.
Reduce memory usage in
make_blobsby preallocating empty NumPy array instead of concatenating, as suggested by @glemaitre.Also adds a test case, using
memory_profilerto track the memory usage of the function.