FIX Missing num_threads in some HGBT pranges - #22566
Conversation
|
Thanks, I wanted to test if that would fix the performance problem on my local machine but unfortunately, I found another bug :) I reported it here: joblib/loky#354. It should be quite easy to fix. |
|
FYI I am currently building this branch in my user container the MOOC platform to give it a try. |
ogrisel
left a comment
There was a problem hiding this comment.
LGTM! I confirm this fixed the over-subscription problem observed on the MOOC platform!
Please add a changelog entry to document the fix.
|
Does that mean that each time |
I think so because the num_threads we are passing comes from _openmp_effective_n_threads which takes cgroups quota into account. |
|
OK, I am not 100% sure but it seems like there are other places where we don't do that, e.g. This is not using I was looking visually at |
|
Yes this one should also set num_threads. Let's do that in another PR since this one is specific to HGBT |
lorentzenchr
left a comment
There was a problem hiding this comment.
Seems to be the right thing to do, but couldn't test it.
Co-authored-by: Christian Lorentzen <[email protected]>
|
Merging thanks! |
Co-authored-by: Christian Lorentzen <[email protected]>
Probably the reason of the performance issue reported in INRIA/scikit-learn-mooc#586
in
for thread_idx in prange(n_threads, schedule='static', chunksize=1), n_threads is the number of tasks but the number of threads stays unspecified. Like this, the number of tasks might be smaller than the number of threads, but openmp will still spawn all threads even if some of them are idle.