Thanks to visit codestin.com
Credit goes to github.com

Skip to content

FIX roc_auc_curve: Return np.nan instead of 0.0 for single class #30103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
glemaitre merged 5 commits into from
Oct 29, 2024
Merged

FIX roc_auc_curve: Return np.nan instead of 0.0 for single class #30103

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
0637f13
roc_auc_curve: Return np.nan instead of 0.0 for single class
janezd Oct 18, 2024
da2809d
roc_auc_curve: Use math.isnan instead of np.isnan; add attribution to…
janezd Oct 22, 2024
ef8fdba
Update sklearn/metrics/tests/test_ranking.py
glemaitre Oct 29, 2024
4d33b35
Update sklearn/metrics/tests/test_ranking.py
glemaitre Oct 29, 2024
9ea9001
duplicate entry changelog
glemaitre Oct 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions doc/whats_new/upcoming_changes/sklearn.metrics/27412.fix.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
- :func:`metrics.roc_auc_score` will now correctly return 0.0 and
- :func:`metrics.roc_auc_score` will now correctly return np.nan and
warn user if only one class is present in the labels.
By :user:`Gleb Levitski <glevv>`
By :user:`Gleb Levitski <glevv>` and :user:`Janez Demšar <janezd>`
Copy link
Member

@adrinjalali adrinjalali Oct 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lesteve this PR number is not gonna show up here. What do we do?

Copy link
Member

@ogrisel ogrisel Oct 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can duplicate the changelog entry into two files with different PR numbers in their name but the same contents.

Then both entries will be merged by towncrier when we aggregate the changelog of a given release.

Copy link
Member

@lesteve lesteve Oct 23, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that's the towncrier's way of doing it, creating two files with same content that only differ by the PR number in the filename.

I originally thought it was a hidden towncrier feature but it is mentioned in the tutorial

$ towncrier create --content 'Can also be ``rst`` as well!' 3456.doc.rst
# You can associate multiple issue numbers with a news fragment by giving them the same contents.
$ towncrier create --content 'Can also be ``rst`` as well!' 7890.doc.rst

I discovered this by looking at the towncrier source code. Note that it only works if both fragments have the same type (fix in this case) IIRC.

There is twisted/towncrier#599 to try to make it a bit more convenient.

Copy link
Member

@adrinjalali adrinjalali Oct 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@glemaitre you've enabled automerge before resolving this though.

Copy link
Member

@glemaitre glemaitre Oct 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I duplicated the changelog.

3 changes: 3 additions & 0 deletions doc/whats_new/upcoming_changes/sklearn.metrics/30013.fix.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
- :func:`metrics.roc_auc_score` will now correctly return np.nan and
warn user if only one class is present in the labels.
By :user:`Gleb Levitski <glevv>` and :user:`Janez Demšar <janezd>`
5 changes: 2 additions & 3 deletions sklearn/metrics/_ranking.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -375,12 +375,11 @@ def _binary_roc_auc_score(y_true, y_score, sample_weight=None, max_fpr=None):
warnings.warn(
(
"Only one class is present in y_true. ROC AUC score "
"is not defined in that case. The score is set to "
"0.0."
"is not defined in that case."
Copy link
Member

@ogrisel ogrisel Oct 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be helpful to store np.unique(y_true) in a local variable and then display the value of that class label in the warning message.

Copy link
Contributor Author

@janezd janezd Oct 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To my understanding and exploration of the code, y_true can already be swapped at this point, so the warning would be misleading. (See #30079 (comment)).

I would prefer to keep this as it is, but if anybody wants to change it - go ahead. :)

Copy link
Contributor

@glevv glevv Oct 23, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the idea here is different: we are not going to say that it is a positive or a negative class, we will just show the value that we found, i.e

f"Only one class is present in y_true: {y_true[0]!r}. "
f"ROC AUC score is not defined in that case."

Or something like that.

Copy link
Contributor Author

@janezd janezd Oct 23, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please let us stop this.

As I wrote, classes are renumerated. I know next to nothing about sklearn's code, but if caller gives y_true=[5, 2, 5] to roc_auc_score, the y_true that is passed to this internal function will be [1, 0, 1]. If caller gives [1, 1, 1], this function gets [0, 0, 0]. Any y_true composed of equal values is renumerated to y_true's composed of 0s.

If I change the function to

def _binary_roc_auc_score(y_true, y_score, sample_weight=None, max_fpr=None):
    if len(np.unique(y_true)) != 2:
        print("all values equal", y_true[0])
    return

I get the following

>>> import sklearn.metrics
sklearn.metrics.roc_auc_score([0, 0, 0], [0.3, 0.4, 0.5])
all values equal 0
>>> sklearn.metrics.roc_auc_score([1, 1, 1], [0.3, 0.4, 0.5])
all values equal 0
>>> sklearn.metrics.roc_auc_score([5, 5, 5], [0.3, 0.4, 0.5])
all values equal 0

To my possibly incomplete understanding of the code, len(np.unique(y_true)) != 2 may be equivalent to not np.any(y_true).

This function cannot give a more descriptive warning, because it doesn't have sufficient information for it. A better warning could be given by the function that calls it, but it would require a big refactoring because a (curried) _binary_roc_auc_score is given as an argument to _average_binary_score.

Changing a single 0.0 into np.nan and adding two lines of tests has taken me several hours spread across five days, mostly because of these warnings that could, imho, stay as they were. Not a great experience that I would want to repeat. :)

),
UndefinedMetricWarning,
)
return 0.0
return np.nan

fpr, tpr, _ = roc_curve(y_true, y_score, sample_weight=sample_weight)
if max_fpr is None or max_fpr == 1:
Expand Down
5 changes: 3 additions & 2 deletions sklearn/metrics/tests/test_common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import math
from functools import partial
from inspect import signature
from itertools import chain, permutations, product
Expand Down Expand Up @@ -843,9 +844,9 @@ def test_format_invariance_with_1d_vectors(name):
):
if "roc_auc" in name:
# for consistency between the `roc_cuve` and `roc_auc_score`
# 0.0 is returned and an `UndefinedMetricWarning` is raised
# np.nan is returned and an `UndefinedMetricWarning` is raised
with pytest.warns(UndefinedMetricWarning):
assert metric(y1_row, y2_row) == pytest.approx(0.0)
assert math.isnan(metric(y1_row, y2_row))
else:
with pytest.raises(ValueError):
metric(y1_row, y2_row)
Expand Down
7 changes: 5 additions & 2 deletions sklearn/metrics/tests/test_ranking.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import math
import re

import numpy as np
Expand Down Expand Up @@ -370,7 +371,8 @@ def test_roc_curve_toydata():
"ROC AUC score is not defined in that case."
)
with pytest.warns(UndefinedMetricWarning, match=expected_message):
roc_auc_score(y_true, y_score)
auc = roc_auc_score(y_true, y_score)
assert math.isnan(auc)

# case with no negative samples
y_true = [1, 1]
Expand All @@ -388,7 +390,8 @@ def test_roc_curve_toydata():
"ROC AUC score is not defined in that case."
)
with pytest.warns(UndefinedMetricWarning, match=expected_message):
roc_auc_score(y_true, y_score)
auc = roc_auc_score(y_true, y_score)
assert math.isnan(auc)

# Multi-label classification task
y_true = np.array([[0, 1], [0, 1]])
Expand Down