Yes, but open does not accept an encoding parameter in Python 2; maybe initializing TfidfVectorizer with the input="filename" and letting it handle the encoding would be better?

Also what's the difference between the MLComp 20 newsgoups in this example and the 20 newgroups dataset used in the other text classification example obtained with fetch_20newsgroups. Isn't that the same dataset?

MLComp 20 newsgoups in this example and the 20 newgroups dataset used in the other text classification example obtained with `fetch_20newsgroups

I was wondering exactly the same thing and I don't know the answer.

@lesteve It looks like it is almost the same dataset, the MLComp 20 newsgroup example indicates that it uses the 20news-18828.tar.gz dataset from http://qwone.com/~jason/20Newsgroups/. This corresponds to

20news-18828.tar.gz 20 Newsgroups; duplicates removed, only "From" and "Subject" headers (18828 documents)

while the original Newsgroup dataset which can be obtained with sklearn.datasets.fetch_20newsgroups, is

20news-19997.tar.gz - Original 20 Newsgroups data set (18846 documents)

where headers can be removed with remove=(headers,).

I don't think that a few duplicates and differences in the kept header fields are fundamental to demonstrate text categorization in this example. Which makes the the MLComp 20 newsgroup example somewhat redundant with the other text classification example. Maybe the the MLComp 20 newsgroup example should be rewritten to use fetch_20newsgroups or made less redundant?

Also it looks like neither of those examples are run in the example gallery, is it because they need to download an external dataset and take longer to run?

Maybe the the MLComp 20 newsgroup example should be rewritten to use fetch_20newsgroups or made less redundant?

I would remove the MLComp 20 newsgroup example. Quickly looking at the source code both examples are very similar. I would be in favour of deprecating load_mlcomp which is not used anywhere else and is very cumbersome to use (need to manually download the data and set an environment variable) and does not have any added value compared to fetch_20newsgroups.

Also it looks like neither of those examples are run in the example gallery, is it because they need to download an external dataset and take longer to run?

Another thing worth investigating would be to look at the examples that are not run (i.e. the ones not starting with plot_) and checking how long they take to run. I tried a few and they were not taking too much time. Renaming these examples to start with plot_ would make them better tested and more accessible through a gallery thumbnail.

I would remove the MLComp 20 newsgroup example. Quickly looking at the source code both examples are very similar.

OK, I can make a PR for it.

I would be in favour of deprecating load_mlcomp which is not used anywhere else and is very cumbersome to use (need to manually download the data and set an environment variable) and does not have any added value compared to fetch_20newsgroups.

It is indeed redundant with fetch_20newsgroups for 20 newgroups dataset, however, presumably, this function can be used to load any dataset from http://mlcomp.org/datasets (well, according to the docstring). So maybe just raise a deprecation warning if the name_or_id corresponds to the 20newsgroup dataset?

this can be closed as the example was removed, right?