Codecov Report

Merging #1055 into master will decrease coverage by 0.40%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #1055      +/-   ##
==========================================
- Coverage   94.23%   93.83%   -0.41%     
==========================================
  Files          47       47              
  Lines        6849     6889      +40     
==========================================
+ Hits         6454     6464      +10     
- Misses        395      425      +30     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dfc11fe...360e8d9. Read the comment docs.

@ogrisel in case you did not notice this PR (no pressure for the review, I just have not tagged you yet :))

#959 is not fixed by this PR only, we also need to tweak the scattering properties of dask, but this is orthogonal to this PR. However, the pattern that this PR fixes appeared in #959.

The linux_pypy3 failure is an instance of the random failure of test_nested_exception_dispatch[multiprocessing] already tracked in #1034 as is unrelated to the dask backend.

I pushed a new commit to also run the tests with a more recent version of dask/distributed (namely 2.17) and I can reproduce a failure I observed on my local machine when using recent distributed:

_____________________________ test_manual_scatter ______________________________

loop = <tornado.platform.asyncio.AsyncIOLoop object at 0x7f93ae9cce50>

                             f(z, z, x, d=z, e=y)]
                    expected = [func(*args, **kwargs)
                                for func, args, kwargs in tasks]
                    results = Parallel()(tasks)
    
                # Scatter must take a list/tuple
                with pytest.raises(TypeError):
                    with parallel_backend('dask', loop=loop, scatter=1):
                        pass
    
        assert results == expected
    
        # Scattered variables only serialized once
        assert x.count == 1
        assert y.count == 1
>       assert z.count == 6
E       assert 4 == 6
E         +4
E         -6

This failure disappears with distributed 2.13 (I have not tried other versions) or when running the tests with distributed 2.17 on master. So this related to an interaction of this PR and somewhat recent changes in distributed.

Edit: I just realized that on master we have assert z.count == 4 instead.

Maybe z is sometimes memoized in a Batch?

I merged master that has the fix to make AutoBatchingMixin work and improved the test to make it more exhaustive and rewrote some old inline comments that would no longer match the current state of the PR.

I think we could add a __repr__ to the Batch class to display the name of the wrapped function name and the number of batched tasks to make it more informative when using the dask dashboard.

Indeed, my conclusion was that somehow, memoization does not happen during the serialization of a function call submitted to distributed's Client.

So this PR actually changes the behavior of joblib when the user submits a function that modifies its inputs:

from distributed import Client, LocalCluster
from joblib import Parallel, delayed, parallel_backend

if __name__ == "__main__":
    cluster = LocalCluster(n_workers=2)
    client = Client(cluster)

    def f1(a, b):
        a.append(1)
        return a, b

    def f2(tuple_, b):
        a1, a2 = tuple_
        a1.append(1)
        return (a1, a2), b

    a = []

    print("f1 using distributed", client.submit(f1, a, a).result())

    a = []
    print("f1 using joblib+loky", Parallel()(delayed(f1)(a, a) for _ in range(1)))

    a = []
    with parallel_backend("dask"):
        print("f1 using joblib+dask", Parallel()(delayed(f1)(a, a) for _ in range(1)))

    a = []
    print("f2 using distributed", client.submit(f2, (a, a), a).result())

    a = []
    print("f2 using joblib+loky", Parallel()(delayed(f2)((a, a), a) for _ in range(1)))

    a = []
    with parallel_backend("dask"):
        print(
            "f2 using joblib+dask", Parallel()(delayed(f2)((a, a), a) for _ in range(1))
        )

gives:

f1 using distributed ([1], [])
f1 using joblib+loky [([1], [1])]
f1 using joblib+dask [([1], [])]
f2 using distributed (([1], []), [])
f2 using joblib+loky [(([1], [1]), [1])]
f2 using joblib+dask [(([1], []), [])]

on this branch, and

f1 using distributed ([1], [])
f1 using joblib+loky [([1], [1])]
f1 using joblib+dask [([1], [1])]
f2 using distributed (([1], []), [])
f2 using joblib+loky [(([1], [1]), [1])]
f2 using joblib+dask [(([1], [1]), [1])]

on master.

f2 highlights the fact that there is no memoization happening at all, even when serializing only one attribute (one could imagine that distribubuted separates the serialization of each args of a submitted functions).

one could imagine that distribubuted separates the serialization of each args of a submitted functions

That does not seem to be the case:

>>> from distributed import LocalCluster, Client
>>> client = Client(LocalCluster(processes=2, threads_per_worker=1))
>>> def physical_identity(a, b): 
...     return a is b
...
>>> o1 = o2 = object()
>>> client.submit(physical_identity, o1, o2).result()
True

I have no idea what's going on. It seems that objects are memoized but lists are not...

This is probably caused by the fact that top level lists are copied because they can contain future instances.

For the record, the failure in test_nested_exception_dispatch[loky] is caused by a side effect of the import of the distributed package in test_parallel.py but only in Python 3.8. Still investigating.

The test_nested_exception_dispatch[loky] failure is caused by tblib.pickling_support.install() used in distributed.

I can reproduce it with loky and tblib (without joblib and distributed) but only under a pytest session:

import tblib.pickling_support
from loky import ProcessPoolExecutor


tblib.pickling_support.install()


def raise_value_error():
    raise ValueError("The message")


def test_exception_cause():
    pe = ProcessPoolExecutor(max_workers=2)
    f = pe.submit(raise_value_error)
    try:
        f.result()
    except ValueError as e:
        assert e.__cause__ is not None


if __name__ == "__main__":
    test_exception_cause()

Interestingly enough, I cannot reproduce the problem when using concurrent.futures instead of loky. I suspect that this is because _ExceptionWithTraceback inherits from BaseException in loky but not in concurrent.futures and that causes a bad interaction with pytest and tblib for some reason I do not quite understand.

I will work on an issue / PR for loky.

Even if I do not fully understand what's going on with pytest and tblib, the fix I suggested above for loky actually works. I opened a PR for loky. In the mean time let's XFAIL this test in joblib.

I think we just need to improve Batch.__repr__ to be able to merge this PR.

test_thread_bomb_mitigation[loky] used to be a random failure but now seems to fail 100% on the time on this PR. This is possibly again because of a side effect of tblib: will need to investigate.

A simple joblib+dask example to showcase how joblib's Batch function now appear:

In [1]: import time
   ...: from distributed import LocalCluster, get_client, Client, as_completed
   ...: from joblib import Parallel, delayed, parallel_backend
   ...: import numpy as np
   ...:
   ...: cluster = LocalCluster(n_workers=2)
   ...: client = Client(cluster)
   ...:
   ...: def sleep_and_return(i):
   ...:     time.sleep(0.05)
   ...:     return i

then open http://localhost:8787/status

then

In [2]: with parallel_backend('dask'):
   ...:     res = Parallel(batch_size=1)(delayed(sleep_and_return)(x) for x in range(100))
   ...: print(res)

here is how the dask dashboard looks:

Note that for some unknown reason, in order to prevent the Batch name to be truncated in the dashboard, I had to use underscores and not dashes in the repr of the Batch to separate words. Weird but I don't have time to investigate.

Thanks @pierreglaser!

I had to use underscores and not dashes in the repr of the Batch to separate words. Weird but I don't have time to investigate.

This is fine I think.

Merged!

Great! Upon rebasing, #1061 should now fully fix #959.

I already merged master in #1601.