FIX binary/multiclass jaccard_similarity_score and extend to handle averaging #13092

jnothman · 2019-02-04T21:25:48Z

Reference Issues/PRs

Fixes #7332. Supersedes #10083

What does this implement/fix? Explain your changes.

The current Jaccard implementation is ridiculous for binary and multiclass problems, returning accuracy. This makes Jaccard comparable to Precision, Recall and F-score, which are also fundamentally set-wise metrics.

Fixes scikit-learn#7332

this deals with both multilabel and multiclass problems

labels, sample_weight seems to be working fine, though haven't fully testing them again, will do in next commit

…ccard-sim

…accard-sim

jnothman · 2019-02-05T19:44:11Z

We might also consider renaming jaccard_similarity_score to jaccard_score, deprecating the old one, and making average='binary' the default as in Precision-Recall-Fscore

adrinjalali · 2019-02-06T05:54:26Z

sklearn/metrics/classification.py

@@ -577,7 +577,8 @@ class labels [2]_.
    return 1 - k


-def jaccard_similarity_score(y_true, y_pred, normalize=True,
+def jaccard_similarity_score(y_true, y_pred, labels=None, pos_label=1,
+                             average='samples', normalize='true-if-samples',


just a note that this is not backward compatible with users calling it with positional arguments [sigh]! But I'm not sure what we should do in these cases.

If we deprecate the current function and make jaccard_score that would solve it :)

sklearn/metrics/classification.py

adrinjalali · 2019-02-06T06:00:48Z

sklearn/metrics/classification.py

+        labels are column indices. By default, all labels in ``y_true`` and
+        ``y_pred`` are used in sorted order.
+
+    pos_label : str or int, 1 by default


-> (default=1)?

adrinjalali · 2019-02-06T06:06:12Z

sklearn/metrics/classification.py

+    ... # doctest: +ELLIPSIS
+    0.33...
+    >>> jaccard_similarity_score(y_true, y_pred, average='micro')
+    ... # doctest: +ELLIPSIS


I think this is redundant, it's already set above (and it generates the odd empty ... line in the output).

I think these flags are per-statement, so I don't see how "it's already set above"

The scope of those flags are at least per-block, example:

scikit-learn/sklearn/covariance/empirical_covariance_.py

Lines 128 to 132 in 8d10ba0

>>> cov.covariance_ # doctest: +ELLIPSIS

array([[0.7569..., 0.2818...],

[0.2818..., 0.3928...]])

>>> cov.location_

array([0.0622..., 0.0193...])

sklearn/metrics/classification.py

adrinjalali · 2019-02-06T06:25:29Z

sklearn/metrics/tests/test_classification.py

+def test_jaccard_similarity_score_validation():
+    y_true = np.array([0, 1, 0, 1, 1])
+    y_pred = np.array([0, 1, 0, 1, 1])
+    assert_raise_message(ValueError, "pos_label=2 is not a valid label: "


pytest.raises?

why pytest.raises? For readability? I don't think the error message is any better with pytest.raises for instance.

aren't we gradually moving away from assert_raise_message and move to with pytest.raises(...)? At least that was my impression.

adrinjalali · 2019-02-06T06:28:44Z

sklearn/metrics/tests/test_classification.py

+            "classification.")
+    assert_raise_message(ValueError, msg3, jaccard_similarity_score, y_true,
+                         y_pred, average='samples')
+    assert_raise_message(ValueError, msg3, jaccard_similarity_score, y_true,


duplicate of the above? seems like a copy/paste issue.

jnothman · 2019-02-06T06:39:29Z

Thanks for the review @adrinjalali!

adrinjalali · 2019-02-06T07:36:31Z

It's also probably a good idea to check if there need to be changes to examples/multioutput/plot_classifier_chain_yeast.py.

qinhanmin2014 · 2019-02-12T15:08:46Z

We might also consider renaming jaccard_similarity_score to jaccard_score, deprecating the old one, and making average='binary' the default as in Precision-Recall-Fscore

I'll vote +1 for this solution.

And I don't understand why we need normalize parameter.

adrinjalali · 2019-02-13T16:05:59Z

sklearn/metrics/classification.py

+        ``'samples'``:
+            Calculate metrics for each instance, and find their average (only
+            meaningful for multilabel classification).
+
    normalize : bool, optional (default=True)


default is true-if-samples and not True

adrinjalali · 2019-02-13T16:11:57Z

well, I guess #13151 is a better solution anyway :)

qinhanmin2014 · 2019-02-14T01:23:43Z

I think there's enough consensus to close this one. Let's try to merge #13151

gxyd added 30 commits November 8, 2017 01:25

multiclass jaccard similarity not equal to accurary_score

64e30d6

Fixes scikit-learn#7332

add space and fix input

a495cfc

score being a n_class size array and weight already taken care of

fcba7f0

add space to fix printing of doctest

d49ccab

add support for 'average' of type 'macro', 'micro', 'weighted'

615ac9a

add tests and make documentation changes

78b2a84

use 'average' for 'multilabel' classification

41f7e2b

introduce average='binary', average='samples'

a7d0111

show errors and warning before anything

057815a

this deals with both multilabel and multiclass problems

write separate functions

f1bd76f

completely okay API and improved doctest

581d540

fix lgtm error and better control flow

aefe921

add normalize in API

83df958

raise ValueError for not-providing 'avergae' in multiclass

041c668

fixed errors with multiclass for different average values

39b92b1

fix tests, use assert_raise_message instead

a0712b5

add common_test for jaccard_similarity_score

113072a

use average='none-samples' instead of 'normalize=False'

c52d577

average='micro' in multiclass case is equivalent to accuracy_score

2e2d762

fixes to multilabel case

5504a00

add error message for average='samples' for non-multilable case

b30ba53

add none-samples in common test

8d0ca20

add support for labels in multilabel classification

ce89b5f

fix multilablel classification

192bb2d

labels, sample_weight seems to be working fine, though haven't fully testing them again, will do in next commit

fix for multiclass

149af2a

corrected 'macro', 'weighted' for multiclass only 'micro' remains

40fca72

fix completely logic of average='micro', now only 'binary' remains

4b50447

remove 'warn' from API, after discussion on PR with jnothman

fd099e5

fix average='binary'

8c9c614

fix doctest, now test_common and lgtm remain to be fixed

a7d3b40

jnothman added 4 commits November 25, 2018 00:26

reuse warning code

46c1274

Merge branch 'master' of github.com:scikit-learn/scikit-learn into ja…

a779926

…ccard-sim

WIP

e082e62

Merge branch 'master' into jaccard-sim

1e9373e

jnothman added the Bug label Feb 4, 2019

jnothman added this to the 0.21 milestone Feb 4, 2019

jnothman mentioned this pull request Feb 4, 2019

[MRG] average parameter for jaccard_similarity_score #10083

Closed

jnothman added 2 commits February 5, 2019 08:27

Clean what's new

28dcca4

Merge branch 'master' into jaccard-sim

7fd7201

jnothman changed the title ~~FIX and extension of jaccard_similarity_score to handle multiclass/averaging properly~~ FIX multiclass jaccard_similarity_score and extend to handle averaging Feb 5, 2019

jnothman changed the title ~~FIX multiclass jaccard_similarity_score and extend to handle averaging~~ FIX binary/multiclass jaccard_similarity_score and extend to handle averaging Feb 5, 2019

jnothman added 2 commits February 5, 2019 23:40

FIX coax tests to pass

27cf502

Merge branch 'jaccard-sim' of github.com:jnothman/scikit-learn into j…

7943540

…accard-sim

adrinjalali reviewed Feb 6, 2019

View reviewed changes

sklearn/metrics/classification.py Show resolved Hide resolved

adrinjalali reviewed Feb 6, 2019

View reviewed changes

sklearn/metrics/classification.py Show resolved Hide resolved

adrinjalali reviewed Feb 6, 2019

View reviewed changes

Address Adrin's comments

47776c0

jnothman mentioned this pull request Feb 13, 2019

ENH/FIX Replace jaccard_similarity_score by sane jaccard_score #13151

Merged

adrinjalali reviewed Feb 13, 2019

View reviewed changes

qinhanmin2014 closed this Feb 14, 2019

	>>> cov.covariance_ # doctest: +ELLIPSIS
	array([[0.7569..., 0.2818...],
	[0.2818..., 0.3928...]])
	>>> cov.location_
	array([0.0622..., 0.0193...])

Uh oh!

FIX binary/multiclass jaccard_similarity_score and extend to handle averaging #13092

FIX binary/multiclass jaccard_similarity_score and extend to handle averaging #13092

Conversation

jnothman commented Feb 4, 2019

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Uh oh!

jnothman commented Feb 5, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jnothman commented Feb 6, 2019

Uh oh!

adrinjalali commented Feb 6, 2019

Uh oh!

qinhanmin2014 commented Feb 12, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adrinjalali commented Feb 13, 2019

Uh oh!

qinhanmin2014 commented Feb 14, 2019

Uh oh!

Uh oh!

jnothman commented Feb 5, 2019 •

edited

Loading