Coverage remained the same when pulling f4771ba on dblalock:pipeline-caching into 56ee99c on scikit-learn:master.

It's not a bad idea, but it has been discussed before extensively (e.g. #2086). I'm interested to see how you have implemented it, but expect some friction before any such feature is merged!

For an exercise, could you please set cache=True as default and report if existing tests fail? Thanks.

Coverage remained the same when pulling 724573d on dblalock:pipeline-caching into 56ee99c on scikit-learn:master.

Good suggestion. It initially failed two tests in the "working with text" doctests because the named_steps dict wasn't being updated, but everything passes now (branch with this experiment here). The only snag I observed when running it with this setting was that the caching version took much longer when manipulating the 20 Newsgroups dataset in the text analysis tutorial:

> ~/Desktop/code/scikit-learn$ nosetests --doctest-tests doc/tutorial/text_analytics/working_with_text_data.rst
.
----------------------------------------------------------------------
Ran 1 test in 5.864s        # no caching

OK
> ~/Desktop/code/scikit-learn$ nosetests --doctest-tests doc/tutorial/text_analytics/working_with_text_data.rst
.
----------------------------------------------------------------------
Ran 1 test in 64.028s       # cache=True, first run

OK
> ~/Desktop/code/scikit-learn$ nosetests --doctest-tests doc/tutorial/text_analytics/working_with_text_data.rst
.
----------------------------------------------------------------------
Ran 1 test in 29.057s       # cache=False, subsequent runs; +/- 1s
OK`

I.e., the text analysis runs significantly slower with cache=True than cache=False. This is unsurprising given that it's probably faster to count tokens than hash giant strings, as joblib is forced to do in this case. Clearly, caching should be off by default.

Thanks for that. I think it would be a good idea to modify all pipeline tests to ensure they work both with and without caching, but it's a lot of work for little glory.

Clearly, caching should be off by default.

Or this is the wrong model, and the user should be able to cache any particular estimator, rather than the whole pipeline. Yet I think that in terms of usability that covers the majority of cases, the present model excels.

I'd like to see tests with a mock estimator that counts the number of calls to each method so that we can actually tell when it's being called.

Apart from hashing text, I'm a little concerned that we're using entire models as cache keys (#2086 instead used the training data + the estimator parameters). Can you benchmark caching a data-heavy model like RandomTreesEmbedding? How long does the caching itself take?

Apart from efficiency, one reason for using parameters rather than the whole estimator object is that we might be able to annotate some estimator parameters as not affecting the model fitting (e.g. feature selection thresholds in most cases), and then exclude them from the cache key. But this certainly falls under the class of pie-in-the-sky enhancement, and could -- if the user really cares to -- be achieved in other ways.

Hi.

Is there any progress on this? There is also something on #2086. Is anyone currently working on this limitation of pipelines?

No one's working on this limitation of Pipelines atm, unfortunately. And my attempts (#5080) to make clone work better for generic memoising wrappers hav come to a standstill.

What a pity. This would make a huge impact on many of my projects (and surely on projects of others). But reading through all related issues and PRs, I don't believe I could do this myself.

I would use a subclass of Pipeline, rather than Pipeline itself. The reason being that it is probably a good idea to clone the objects before memoizing the fit (in order to maximize the cache hits). This is not what the standard Pipeline does, hence memoizing would have a slightly different behavior (though one fully consistent with scikit-learn's philosophy of not modifying input parameters).

@GaelVaroquaux yeah the not cloning is an interesting feature. Name suggestions? Cloneline? ;) [Actually we should be pretty careful with naming now, our names get adopted quickly]

I have to say that cloneline sounds awesome :-D

should actually be Cacheline ;)

yeah why not. That's easy to do once we have the class....

I will propose a PR based on the above comments and the discussion from the mailing list.

Fixed by #7990. Thanks @dblalock!