[MRG+2] Fix log loss bug #7239

nelson-liu · 2016-08-24T20:46:45Z

Reference Issue

original PRs at #6714 and #7166 .
Fixes:
metrics.log_loss fails when any classes are missing in y_true #4033
Fix a bug, the result is wrong when use sklearn.metrics.log_loss with one class, #4546
Log_loss is calculated incorrectly when only 1 class present #6703

What does this implement/fix? Explain your changes.

This PR is a cherrypicked, rebased, and squashed version of #7166. I addressed the comments in there, namely by renaming the single-letter variables, adding another ValueError saying that labels should have more than one unique label if len(lb.classes_) == 1 and labels is not None, and removing a commented out code block.

Any other comments?

@MechCoder @amueller anything else that needs to be done?

nelson-liu · 2016-08-24T20:50:09Z

sklearn/metrics/classification.py

    The logarithm used is the natural logarithm (base-e).
    """
    lb = LabelBinarizer()
-    T = lb.fit_transform(y_true)


this block was moved to line 1620 in the diff. the variable T was renamed to transformed_labels and the variable Y was renamed to y_pred

MechCoder · 2016-08-24T21:29:02Z

sklearn/metrics/classification.py

+    else:
+        lb.fit(y_true)
+
+    if labels is None and len(lb.classes_) == 1:


You could reorganize this into:

if len(lb.classes_) == 1: if labels is None: raise else: raise

Definitely, good catch

MechCoder · 2016-08-24T22:26:35Z

Could you also add a test for when there is more than one label in y_true but still len(np.unique(y_true)) != y_pred.shape[1] as a non-regression test for this (#4033) and as a sanity check?

MechCoder · 2016-08-24T22:43:14Z

Also, it is possible to move all the check_array and check_consistent_length checks to the top of the function. It is not clear why all those checks are necessary. (For instance, the fit and transform in the LabelBinarizer should internally call check_array as well.)

MechCoder · 2016-08-24T22:45:36Z

Thanks for wrapping this up. Seems like changing the names of variables has given us the opportunity to do an unintentional cleanup ;)

nelson-liu · 2016-08-24T23:12:37Z

Also, it is possible to move all the check_array and check_consistent_length checks to the top of the function. It is not clear why all those checks are necessary. (For instance, the fit and transform in the LabelBinarizer should internally call check_array as well.)

The check for transformed_labels that I didn't move to the top seems necessary, considering the error is thrown at all (and thus isn't picked up by LabelBinarizer). I've addressed your comments above (kept track of what I had finished with the 👍 emoji). if you can't find where I put a change, I'd be happy to point you to the associated place as this diff is pretty big.

MechCoder · 2016-08-24T23:57:26Z

sklearn/metrics/classification.py

+        raise ValueError("Unable to automatically cast y_pred to "
+                         "float. y_pred should be an array of floats.")
+    # sanity check
+    if y_pred.dtype != float:


So the correct way to do this is to pass a dtype argument to check_array that raises an error if it is unable to be cast to the provided dtype. But if I pass dtype=np.float32, it fails this test (https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/tests/test_classification.py#L1392) because check_array raises an error with a MockDataFrame and dtype=np.float32.

For now, I would suggest to either just remove these float checks (note that nothing useful was checked previously and np.clip should take care of string dtypes) or figure out what is going on with the MockDataframe and fix that (which is beyond the scope of this PR)
Wdyt?

seems reasonable, i agree that figuring out what's going on with MockDataframe is beyond the scope of the PR. do you want to raise an issue for that?

MechCoder · 2016-08-25T00:46:40Z

@nelson-liu That should be my last pass. LGTM pending comments.

nelson-liu · 2016-08-25T05:52:25Z

@MechCoder addressed your comments, let me know if i got them right (in particular the nonregression test).

MechCoder · 2016-08-25T07:28:14Z

sklearn/metrics/tests/test_classification.py

    assert_almost_equal(loss, 1.0383217, decimal=6)

+    # case when len(np.unique(y_true)) != y_pred.shape[1]
+    y_true = [1,2,2]


space this out. and space out labels 2 lines below

MechCoder · 2016-08-25T07:29:49Z

Thanks! You have my +1.

amueller · 2016-08-25T16:25:37Z

sklearn/metrics/classification.py

        Sample weights.

+    labels : array-like, optional (default=None)
+        If not provided, labels will be inferred from y_true


full stop at the end.

amueller · 2016-08-25T16:36:26Z

looks good apart from minor comments.

amueller · 2016-08-25T16:38:59Z

rebase?

nelson-liu · 2016-08-25T16:51:19Z

@amueller addressed your comments, can you look over them (in particular the docstrings clarifying assumptions made) to make sure I got what you wanted before I rebase and lose the history? I'll rebase after you verify it's ok.

enhance log_loss labels option feature log_loss changed test log_loss case u add ValueError in log_loss

fixes as per existing pull request scikit-learn#6714 fixed log_loss bug enhance log_loss labels option feature log_loss changed test log_loss case u add ValueError in log_loss fixes as per existing pull request scikit-learn#6714 fixed error message when y_pred and y_test labels don't match fixed error message when y_pred and y_test labels don't match corrected doc/whats_new.rst for syntax and with correct formatting of credits additional formatting fixes for doc/whats_new.rst fixed versionadded comment removed superfluous line removed superflous line

amueller · 2016-08-25T16:56:44Z

lgtm with additional sentence to docstring.

fix a typo in whatsnew refactor conditional and move dtype check before np.clip general cleanup of log_loss remove dtype checks edit non-regression test and wordings fix non-regression test misc doc fixes / clarifications + final touches fix naming of y_score2 variable specify log loss is only valid for 2 labels or more

nelson-liu · 2016-08-25T17:03:09Z

squashed my commits and rebased. do we want to merge all of the commits on this PR, seeing as there are 3 authors, instead of squashing?

amueller · 2016-08-25T17:12:01Z

one commit per author is fine.

nelson-liu · 2016-08-25T17:30:09Z

Perfect, just how I squashed it. Assuming CI passes, this is ready for merge on my side, let me know if anything else is needed

amueller · 2016-08-25T17:52:49Z

no, I think it's good to go.

nelson-liu · 2016-08-25T19:12:25Z

@amueller merge?

MechCoder · 2016-08-25T19:24:11Z

Thanks!

marconoe · 2016-12-08T21:36:03Z

hey guys im new to github and coding and wondering - how do i use the fix that you guys seem to have created above? I am getting the same issue using log_loss:

ValueError: y_true and y_pred have different number of classes 2, 3

thanks for working on this!
marco

amueller · 2016-12-08T21:46:19Z

@marconoe what version of scikit-learn are you using? This should be fixed in 0.18 and 0.181.

marconoe · 2016-12-09T02:26:10Z

hi @amueller,
Thanks that makes sense - I'm using 0.17.1. I just updated to 0.18.1

Now i get a new error but at least it's different so i can work on that.

cheers
marco

jnothman · 2016-12-09T03:33:46Z

Upgrade scikit-learn (e.g. pip install -U scikit-learn) and check if it's still a problem.

…

On 9 December 2016 at 08:36, Marco ***@***.***> wrote: hey guys im new to github and coding and wondering - how do i use the fix that you guys seem to have created above? I am getting the same issue using log_loss: ValueError: y_true and y_pred have different number of classes 2, 3 thanks marco — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#7239 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz6woJFV7Ua-JaiHIsQazacQozjKI8ks5rGHhEgaJpZM4Jsb6D> .

nelson-liu reviewed Aug 24, 2016
View reviewed changes

nelson-liu changed the title ~~Fix log loss bug~~ [MRG] Fix log loss bug Aug 24, 2016

MechCoder mentioned this pull request Aug 24, 2016

[MRG+1] Log loss bug fixed #7166

Closed

MechCoder added this to the 0.18 milestone Aug 24, 2016

MechCoder reviewed Aug 24, 2016
View reviewed changes

MechCoder reviewed Aug 25, 2016
View reviewed changes

amueller reviewed Aug 25, 2016
View reviewed changes

Harry040 and others added 2 commits August 25, 2016 09:52

fixed log_loss bug

6028e42

enhance log_loss labels option feature log_loss changed test log_loss case u add ValueError in log_loss

nelson-liu force-pushed the fix_log_loss_bug branch from 4ced594 to d97a25f Compare August 25, 2016 17:02

nelson-liu changed the title ~~[MRG+1] Fix log loss bug~~ [MRG+2] Fix log loss bug Aug 25, 2016

MechCoder merged commit 104e09a into scikit-learn:master Aug 25, 2016

ysk24ok mentioned this pull request Sep 17, 2016

Add a warning when one-class vector is passed to LabelBinarizer::fit() #6009

Closed

amueller mentioned this pull request Aug 6, 2020

pos_label in PrecisionRecallDisplay and RocCurveDisplay #18101

Open

Uh oh!

[MRG+2] Fix log loss bug #7239

[MRG+2] Fix log loss bug #7239

Uh oh!

Conversation

nelson-liu commented Aug 24, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference Issue

What does this implement/fix? Explain your changes.

Any other comments?

Uh oh!

nelson-liu Aug 24, 2016

Choose a reason for hiding this comment

Uh oh!

MechCoder Aug 24, 2016

Choose a reason for hiding this comment

Uh oh!

nelson-liu Aug 24, 2016

Choose a reason for hiding this comment

Uh oh!

MechCoder commented Aug 24, 2016

Uh oh!

MechCoder commented Aug 24, 2016

Uh oh!

MechCoder commented Aug 24, 2016

Uh oh!

nelson-liu commented Aug 24, 2016

Uh oh!

MechCoder Aug 24, 2016

Choose a reason for hiding this comment

Uh oh!

nelson-liu Aug 25, 2016

Choose a reason for hiding this comment

Uh oh!

MechCoder commented Aug 25, 2016

Uh oh!

nelson-liu commented Aug 25, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MechCoder Aug 25, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amueller Aug 25, 2016

Choose a reason for hiding this comment

Uh oh!

MechCoder commented Aug 25, 2016

Uh oh!

amueller Aug 25, 2016

Choose a reason for hiding this comment

Uh oh!

amueller commented Aug 25, 2016

Uh oh!

amueller commented Aug 25, 2016

Uh oh!

nelson-liu commented Aug 25, 2016

Uh oh!

amueller commented Aug 25, 2016

Uh oh!

nelson-liu commented Aug 25, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amueller commented Aug 25, 2016

Uh oh!

nelson-liu commented Aug 25, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amueller commented Aug 25, 2016

Uh oh!

nelson-liu commented Aug 25, 2016

Uh oh!

MechCoder commented Aug 25, 2016

Uh oh!

marconoe commented Dec 8, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amueller commented Dec 8, 2016

Uh oh!

marconoe commented Dec 9, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

nelson-liu commented Aug 24, 2016 •

edited

Loading

nelson-liu commented Aug 25, 2016 •

edited

Loading

MechCoder Aug 25, 2016 •

edited

Loading

nelson-liu commented Aug 25, 2016 •

edited

Loading

nelson-liu commented Aug 25, 2016 •

edited

Loading

marconoe commented Dec 8, 2016 •

edited

Loading

marconoe commented Dec 9, 2016 •

edited

Loading