I like this idea in general, but I think there will be heavy resistance given how much it is used.

On a technical note, TfidfTransformer copies by default, and TfidfVectorizer gets around this by calling transform(..., copy=False). We may need to add a copy parameter to TfidfTransformer to have the same behavior:

vectorizer = Pipeline([
    ('vect', CountVectorizer(...)),
    ('idf', TfidfTransformer(..., copy=False))
])

I like this idea in general, but I think there will be heavy resistance given how much it is used.

+1, I prefer to keep it, but won't oppose if we decide to deprecate it. TFIDF is a basic text preprocessing technique and I guess users will be unhappy if they need to use a pipeline?

The main argument in favour of the deprecation is the complexity of the API and the number of parameters in the TfidfVectorizer class.

I don't think the API is complex. We mention in the doc that "TfidfVectorizer is equivalent to CountVectorizer followed by TfidfTransformer." and parameters in TfidfVectorizer = parameters in CountVectorizer + parameters in TfidfTransformer.

We may need to add a copy parameter to TfidfTransformer to have the same behavior:

+1, maybe also remove the copy parameter in transform.

TFIDF is a basic text preprocessing technique and I guess users will be unhappy if they need to use a pipeline?*

That is a risk yes, particularly with all the things that we have been deprecating lately.

I don't think the API is complex.

What I mean by complex is that it has 21 parameters and performs numerous tasks (pre-processing, tokenization, n-grams concatenation, counting, IDF weighting, L2 normalization) some of which users are not aware. As long as they use it as a black box it works fine, however, as soon as they need to tune any part of that pipeline (which is my experience happens often) problems start:

• the way you have to subclass vectorizers plug-in something into that pipeline, while re-using the rest is messy. Deprecating TfidfVectorizer is only one step in the direction of isolating that to a single CountVectorizer class, to be able to do something with it later.
• users don't realize the options they use (e.g. Added option to use standard idf term for TfidfTransformer and TfidfVectorizer #14748) because instead considering each stage of the pipeline, they are presented with these 21 parameters at once.

That is a risk yes, particularly with all the things that we have been deprecating lately.

I guess this risk is not trivial :)

What I mean by complex is that it has 21 parameters

That's because there're 17 parameters in CountVectorizer, TfidfTransformer only introduces 4 extra parameters.

performs numerous tasks (pre-processing, tokenization, n-grams concatenation, counting, IDF weighting, L2 normalization) some of which users are not aware.

I think this is a seperate issue? (e.g., split the vectorizer into several classes?)

users don't realize the options they use (e.g. #14748) because instead considering each stage of the pipeline, they are presented with these 21 parameters at once.

Not sure whether this is a good reason. I think the doc is clear enough. "TfidfVectorizer is equivalent to CountVectorizer followed by TfidfTransformer."

I like this idea in general, but I think there will be heavy resistance given how much it is used.

On a technical note, TfidfTransformer copies by default, and TfidfVectorizer gets around this by calling transform(..., copy=False). We may need to add a copy parameter to TfidfTransformer to have the same behavior:

vectorizer = Pipeline([
    ('vect', CountVectorizer(...)),
    ('idf', TfidfTransformer(..., copy=False))
])

Shouldn't adding a copy parameter to TfidfTransformer be a good issue to address regardless? As of now, if one wanted to call fit_transform on a TfIdfTransformer within a pipeline, they wouldn't be able to set the copy parameter. It is only useful when calling fit() and transform() separately atm.

Closing as it's unlikely to happen.

Do you remember why this was deemed "unlikely to happen" @rth?

My understanding is that in general we would be OK to add a copy param to TfidfTransformer and then deprecate TfidfVectorizer?

I was just closing my PRs that I didn't see being merged in the near future. But overall yes, I think it would still be good if this happened, and since there is indeed no specific issue for it and all discussion is here I'll re-open.

My understanding is that in general we would be OK to add a copy param to TfidfTransformer and then deprecate TfidfVectorizer?

That and having a reasonable way to get feature names in a make_pipeline(CountVectorizer(), TfidfTransformer()) without pipeline slicing.