[MRG+1] Test to make sure deletion of `stop_words_` does not affect transformation. #4037

raghavrv · 2014-12-31T02:54:53Z

Addresses #4032

Tests to make sure excluding stop_words_ does not affect transforming.

@jnothman @amueller Please take a look...

jnothman · 2014-12-31T03:04:51Z

I'm -1 on this feature. The additional parameter just adds noise to what is already a huge and confusing parameter list on CountVectorizer, let alone TfidfVecotrizer (which I think you've not yet modified). We similarly don't have a parameter to automatically run sparsify on a linear model. The only case I can think of where this provides great benefit over deleting the attribute manually is where the entire model is cached. But if the user really cares that much, they can easily extend the class via inheritance.

raghavrv · 2014-12-31T03:11:08Z

So these tests would suffice for now?

jnothman · 2014-12-31T03:15:33Z

sklearn/feature_extraction/tests/test_text.py

@@ -857,6 +874,27 @@ def test_pickling_vectorizer():
        assert_array_equal(
            copy.fit_transform(JUNK_FOOD_DOCS).toarray(),
            orig.fit_transform(JUNK_FOOD_DOCS).toarray())
+
+    # Ensure that deleting the stop_words_ attribute doesn't affect pickling


Shouldn't really need a pickle in there. Just need to test that deletion doesn't affect transformation.

If we recommend to do so to users, that is.

Okay! Also just realized that transform calls fit_transform which is the method that sets the stop_word_ attribute... so should we need a test at all?

raghavrv · 2014-12-31T10:02:47Z

@jnothman Does this look okay...?

Also I would like to note that transform in both TfidfVectorizer and CountVectorizer calls fit_transform which is the method that sets the stop_words_ attributes. Hence I am not sure if this test is necessary... ( i.e even if stop_words_ is deleted (or set to None), it is restored back when calling transform ). Please take a look...
If you find it not necessary feel free to close this PR ( and also sorry for the additional noise by raising this PR without investigating properly :p )

jnothman · 2014-12-31T10:26:47Z

Also I would like to note that transform in both TfidfVectorizer and CountVectorizer calls fit_transform which is the method that sets the stop_words_ attributes

Not as far as I can see!

raghavrv · 2014-12-31T10:39:18Z

Ah sorry for the blunder, it was fit which called fit_transform not the transform... :p

raghavrv · 2014-12-31T10:40:00Z

sorry for the confusion :)

jnothman · 2015-01-07T09:35:39Z

sklearn/feature_extraction/tests/test_text.py

@@ -857,6 +860,19 @@ def test_pickling_vectorizer():
        assert_array_equal(
            copy.fit_transform(JUNK_FOOD_DOCS).toarray(),
            orig.fit_transform(JUNK_FOOD_DOCS).toarray())
+
+    # Ensure that deleting the stop_words_ attribute doesn't affect transform


Just make a separate test. This is no longer about pickling, and the current approach makes the above loop through instances more confusing.

jnothman · 2015-01-07T09:36:43Z

Please add a Notes section to the {Count,Tfidf}Vectorizer where you mention that stop_words_ can get large, but can be removed before pickling.

raghavrv · 2015-01-07T17:38:10Z

@jnothman Done! BTW why is Notes section not available in the html?

amueller · 2015-01-07T20:08:26Z

+0 on adding the tests....

The notes should show up in html, probably malformed docstrings.

raghavrv · 2015-01-07T20:41:38Z

@amueller Do you feel the test is unnecessary?

amueller · 2015-01-07T20:45:13Z

It is not even documented that you can remove it, right? I would rather document it than test for it. Or maybe document and also test. Then the test would ensure the documentation is correct.

raghavrv · 2015-01-07T20:46:20Z

@amueller Does this change look okay?

amueller · 2015-01-07T20:47:53Z

sklearn/feature_extraction/text.py

+
+    Notes
+    -----
+    The ``stop_words_`` attribute can get large which may not be desirable when


I would document it at the attribute documentation, saying "only for introspection and can savely be removed before pickling" or something like that.

jnothman · 2015-01-07T22:02:46Z

I thought documenting it at the model level made more sense because users aren't going to looking for things that can be removed. They're going to look for "why is this model so big?". But not too concerned either way.

amueller · 2015-01-07T22:13:29Z

I don't have a strong opinion either way, sorry if I created work.

raghavrv · 2015-01-07T22:18:16Z

Should I revert back to old form? ( Notes )

raghavrv · 2015-01-11T11:26:44Z

sklearn/feature_extraction/text.py

+
+        This attribute is provided only for introspection and can be safely
+        removed using delattr or set to None before pickling.
+


@jnothman Should I revert this to Notes?

raghavrv · 2015-01-11T11:49:02Z

I'll squash the 2nd and 3rd commits once the 3rd is approved!

raghavrv · 2015-01-12T22:23:56Z

Do you think the tests are moot? perhaps we could have the documentation alone?

( In which case I'll delete 1st commit and squash 2 and 3 ) ?

amueller · 2015-01-13T17:15:06Z

As I said, I don't have a strong opinion.

raghavrv · 2015-01-15T23:14:10Z

@jnothman your view on whether the test feels unnecessary?

jnothman · 2015-01-15T23:23:25Z

I think that if it's documented, the test is worthwhile. Don't promise your
users something without checking you can fulfil it.

On 16 January 2015 at 10:14, ragv [email protected] wrote:

@jnothman https://github.com/jnothman your view on whether the test
feels unnecessary?

—
Reply to this email directly or view it on GitHub
#4037 (comment)
.

raghavrv · 2015-01-15T23:24:58Z

Okay! thanks for the quick reply! :)

amueller · 2015-01-15T23:42:22Z

ok then lets merge.

raghavrv · 2015-02-01T18:32:34Z

@jnothman Could this be closed / merged ? ~~( If its good for merge, please let me know I'll squash all the commits to one)~~ I squashed it anyway...

DOC Add a line to {Count, Tfidf}Vectorizer about removal of stop_words_ DOC Add documentation of stop_words_ attr in TfidfVectorizer

coveralls · 2015-02-01T21:02:41Z

Coverage increased (+0.0%) to 94.79% when pulling 21369dd on ragv:make_storing_stop_words_optional into 94157fa on scikit-learn:master.

[MRG] Test to make sure deletion of `stop_words_` does not affect transformation.

jnothman · 2015-02-01T22:55:11Z

Thanks @ragv

raghavrv mentioned this pull request Dec 31, 2014

In feature extraction from text, what is the purpose stop_words_ attribute? #4032

Closed

jnothman reviewed Dec 31, 2014
View reviewed changes

raghavrv force-pushed the make_storing_stop_words_optional branch 2 times, most recently from 791366c to b9a2a54 Compare December 31, 2014 09:56

raghavrv changed the title ~~[MRG] Make storing stop words optional~~ [MRG] Test to make sure deletion of stop_words_ does not affect transformation. Dec 31, 2014

raghavrv force-pushed the make_storing_stop_words_optional branch 3 times, most recently from b2e899a to a8fa121 Compare January 4, 2015 00:21

jnothman reviewed Jan 7, 2015
View reviewed changes

raghavrv force-pushed the make_storing_stop_words_optional branch from a8fa121 to 7c02fe0 Compare January 7, 2015 14:17

amueller reviewed Jan 7, 2015
View reviewed changes

raghavrv force-pushed the make_storing_stop_words_optional branch 3 times, most recently from 75ef655 to 427fb5f Compare January 7, 2015 21:19

raghavrv changed the title ~~[MRG] Test to make sure deletion of stop_words_ does not affect transformation.~~ [MRG+1] Test to make sure deletion of stop_words_ does not affect transformation. Jan 7, 2015

raghavrv force-pushed the make_storing_stop_words_optional branch from 427fb5f to 0875593 Compare January 11, 2015 10:28

raghavrv reviewed Jan 11, 2015
View reviewed changes

raghavrv force-pushed the make_storing_stop_words_optional branch 2 times, most recently from 070f0b1 to 416fea3 Compare January 11, 2015 11:43

raghavrv force-pushed the make_storing_stop_words_optional branch from 5c81e60 to 155d575 Compare January 14, 2015 20:49

raghavrv force-pushed the make_storing_stop_words_optional branch from 155d575 to f71248a Compare January 16, 2015 18:19

raghavrv force-pushed the make_storing_stop_words_optional branch 2 times, most recently from 70c7435 to 43ad91a Compare January 28, 2015 19:16

raghavrv force-pushed the make_storing_stop_words_optional branch 2 times, most recently from ff36cb5 to cd8bd93 Compare February 1, 2015 18:58

TST Removal or modification of stop_words_ should not affect transform.

21369dd

DOC Add a line to {Count, Tfidf}Vectorizer about removal of stop_words_ DOC Add documentation of stop_words_ attr in TfidfVectorizer

raghavrv force-pushed the make_storing_stop_words_optional branch from cd8bd93 to 21369dd Compare February 1, 2015 20:50

jnothman added a commit that referenced this pull request Feb 1, 2015

Merge pull request #4037 from ragv/make_storing_stop_words_optional

eb52bb3

[MRG] Test to make sure deletion of `stop_words_` does not affect transformation.

jnothman merged commit eb52bb3 into scikit-learn:master Feb 1, 2015

raghavrv deleted the make_storing_stop_words_optional branch March 18, 2015 22:55

raghavrv mentioned this pull request May 22, 2015

[WIP] Adding tests for estimators implementing partial_fit and a few other related fixes / enhancements #3907

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MRG+1] Test to make sure deletion of `stop_words_` does not affect transformation. #4037

[MRG+1] Test to make sure deletion of `stop_words_` does not affect transformation. #4037

raghavrv commented Dec 31, 2014

jnothman commented Dec 31, 2014

raghavrv commented Dec 31, 2014

jnothman Dec 31, 2014

jnothman Dec 31, 2014

raghavrv Dec 31, 2014

raghavrv commented Dec 31, 2014

jnothman commented Dec 31, 2014

raghavrv commented Dec 31, 2014

raghavrv commented Dec 31, 2014

jnothman Jan 7, 2015

jnothman commented Jan 7, 2015

raghavrv commented Jan 7, 2015

amueller commented Jan 7, 2015

raghavrv commented Jan 7, 2015

amueller commented Jan 7, 2015

raghavrv commented Jan 7, 2015

amueller Jan 7, 2015

jnothman commented Jan 7, 2015

amueller commented Jan 7, 2015

raghavrv commented Jan 7, 2015

raghavrv Jan 11, 2015

raghavrv commented Jan 11, 2015

raghavrv commented Jan 12, 2015

amueller commented Jan 13, 2015

raghavrv commented Jan 15, 2015

jnothman commented Jan 15, 2015

raghavrv commented Jan 15, 2015

amueller commented Jan 15, 2015

raghavrv commented Feb 1, 2015

coveralls commented Feb 1, 2015

jnothman commented Feb 1, 2015


		This attribute is provided only for introspection and can be safely
		removed using delattr or set to None before pickling.

[MRG+1] Test to make sure deletion of stop_words_ does not affect transformation. #4037

[MRG+1] Test to make sure deletion of stop_words_ does not affect transformation. #4037

Conversation

raghavrv commented Dec 31, 2014

jnothman commented Dec 31, 2014

raghavrv commented Dec 31, 2014

jnothman Dec 31, 2014

Choose a reason for hiding this comment

jnothman Dec 31, 2014

Choose a reason for hiding this comment

raghavrv Dec 31, 2014

Choose a reason for hiding this comment

raghavrv commented Dec 31, 2014

jnothman commented Dec 31, 2014

raghavrv commented Dec 31, 2014

raghavrv commented Dec 31, 2014

jnothman Jan 7, 2015

Choose a reason for hiding this comment

jnothman commented Jan 7, 2015

raghavrv commented Jan 7, 2015

amueller commented Jan 7, 2015

raghavrv commented Jan 7, 2015

amueller commented Jan 7, 2015

raghavrv commented Jan 7, 2015

amueller Jan 7, 2015

Choose a reason for hiding this comment

jnothman commented Jan 7, 2015

amueller commented Jan 7, 2015

raghavrv commented Jan 7, 2015

raghavrv Jan 11, 2015

Choose a reason for hiding this comment

raghavrv commented Jan 11, 2015

raghavrv commented Jan 12, 2015

amueller commented Jan 13, 2015

raghavrv commented Jan 15, 2015

jnothman commented Jan 15, 2015

raghavrv commented Jan 15, 2015

amueller commented Jan 15, 2015

raghavrv commented Feb 1, 2015

coveralls commented Feb 1, 2015

jnothman commented Feb 1, 2015

[MRG+1] Test to make sure deletion of `stop_words_` does not affect transformation. #4037

[MRG+1] Test to make sure deletion of `stop_words_` does not affect transformation. #4037