Add joblib backend - #122
Conversation
Default:
import ipyparallel.backend
Specific View:
view.register_joblib_backend()
| In general working on larger data chunks is more efficient (less | ||
| scheduling overhead and better use of CPU cache prefetching heuristics) | ||
| as long as all the workers have enough work to do. | ||
| """ |
There was a problem hiding this comment.
Big monobloc docstring. Did you forgot aa white line ?
Doesn't seem bad if you want to keep it as is though. Just wondering.
There was a problem hiding this comment.
This is the upstream docstring unmodified.
|
Minor comment/questions, +1 regardless of answers . |
| return register_parallel_backend(name, IPythonParallelBackend, make_default=make_default) | ||
|
|
||
| # importing ipyparallel.joblib sets up the default `ipyparallel` backend | ||
| register() |
There was a problem hiding this comment.
Personally I don't like imports with implicit side effects. I would rather document the following idiom:
from ipyparallel.joblib import register; register()There was a problem hiding this comment.
Declaring a backend as available doesn't instantiate anything, does it? Is there a downside to joblib knowing about ipyparallel?
There was a problem hiding this comment.
But the user has to import that module explicitly to register the backend no? My point is that instead of asking the user to do:
from joblib import Parallel, delayed, parallel_backend
import ipyparallel.joblib # registered the ipyrallel joblib backend implicitly
with parallel_backend('ipyparallel'):
Parallel(n_jobs=4)(delayed(func)(task) in tasks)do the backend registration explicitly
from joblib import Parallel, delayed, parallel_backend
from ipyparallel.joblib import register; register()
with parallel_backend('ipyparallel'):
Parallel(n_jobs=4)(delayed(func)(task) in tasks)There was a problem hiding this comment.
Fair enough. Moved to ipp.register_joblib_backend()
removes implicit register on `ipyparallel.joblib`
joblib backend is a moving target
since it changes the behavior of other tests will open a new PR adding `use_pickle` method
In joblib/joblib#306, joblib adds support for custom backends.
This provides such a backend from a LoadBalancedView.
Simplest version using all the defaults:
Or registering an existing View: