FIX `cross_validate` with multimetric scoring returns the non-failed scorers results if there is a failing scorer #23101

simonandras · 2022-04-10T15:37:52Z

Reference Issues/PRs

What does this implement/fix? Explain your changes.

In _scorer.py in the _MultimetricScorer class the __call__ method is updated. Now instead of raising an exception of a failing scorer it passes the exception to the scores to handle later.

In _validation.py in the _score function the exception is handled at the line 789, based on the suggestion of @thomasjpfan.

Any other comments?

Do we need a test for the one-failing scorer case for the cross_validate function?

sklearn/metrics/_scorer.py

thomasjpfan

Thank you for the PR!

Do we need a test for the one-failing scorer case for the cross_validate function?

Yes we need to test the new behavior. One as a non-regression test for cross_validate and another one for _MultimetricScorer itself.

Note: Changing the behavior of _MultimetricScorer to not raise anymore may require updating some unit tests. This is okay since _MultimetricScorer is a private class.

sklearn/metrics/_scorer.py

…one to the new parts.

sklearn/model_selection/_validation.py

…ssage will show up there.

sklearn/model_selection/_validation.py

thomasjpfan

Thanks for the update!

This PR still requires a non-regression test for the original issue and test to test the behavior of _MultimetricScorer returning a dictionary with exceptions.

…O: a non-regression test for the original issue.

…n-failing multimetric case.

simonandras · 2022-04-18T12:35:53Z

Updated the test_cross_validate_failing_scorer to cover the failing and non-failing multimetric case (the case of the original issue). I think its better this way than writing a new test. @thomasjpfan what do you think about this?
Also added a new test for _MultimetricScorer returning a dictionary.

thomasjpfan

Please add an entry to the change log at doc/whats_new/v1.1.rst with tag |Fix|. Like the other entries there, please reference this pull request with :pr: and credit yourself (and other contributors if applicable) with :user:.

sklearn/metrics/_scorer.py

sklearn/model_selection/tests/test_validation.py

…iling_scorer

…s is controlled by the new bool parameter. The _score function is changed accordingly. The test for the _MultimetricScorer which tested the exception returning case is replaced with the test_multimetric_scorer_exception_handling test. The test_cross_validate_failing_scorer function needs no modification.

sklearn/metrics/_scorer.py

…itional (dict) instead of a depacked dict using **.

…oss_validate_failing_scorer: checking for '_score_2' instead of 'score_2'.

doc/whats_new/v1.1.rst

thomasjpfan

Thanks for the update. I'm mostly okay with using a string to send the exception information back to the caller. Note other reviewers, may have a different opinion on the matter.

doc/whats_new/v1.1.rst

sklearn/model_selection/_validation.py

simonandras · 2022-05-21T14:15:09Z

@thomasjpfan I do not understand the failing test. Do you know what is the issue there?

thomasjpfan · 2022-05-21T17:33:46Z

Merging with main should fix the circleci issue.

simonandras · 2022-06-04T12:12:32Z

@thomasjpfan how do you see, do we need any more modification to the changes?

thomasjpfan · 2022-06-04T15:12:11Z

Assuming you ran this to set the remote url:

git remote add upstream [email protected]:scikit-learn/scikit-learn.git

you can update this PR to fix the circleci issue:

git checkout cross_validate__with_multiple_scorer_nan_issue
git fetch upstream
git merge upstream/main

…h_multiple_scorer_nan_issue

simonandras · 2022-06-04T18:58:03Z

Thank you, it wokred!

thomasjpfan

Thank you for the update! I am mostly on board with this. Let's see what other reviewers think.

thomasjpfan · 2022-06-05T15:49:49Z

sklearn/metrics/_scorer.py

+                if self._raise_exc:
+                    raise e
+                else:
+                    scores[name] = format_exc()


For future reviewers, a string is used to pass the information back to the caller.

simonandras · 2022-07-23T13:06:19Z

@thomasjpfan could you please mark this as "waiting for review"?

doc/whats_new/v1.3.rst

sklearn/metrics/_scorer.py

sklearn/metrics/tests/test_score_objects.py

glemaitre

Since the changes that I requested are only about style, I merged them directly and put my +1.

I will merge when the CI turns green. Thanks @simonandras

…scorers results even if some fail (scikit-learn#23101) Co-authored-by: Guillaume Lemaitre <[email protected]>

…scorers results even if some fail (#23101) Co-authored-by: Guillaume Lemaitre <[email protected]>

Formatting

d49b41c

github-actions bot added module:metrics module:model_selection labels Apr 10, 2022

simonandras added 3 commits April 10, 2022 18:32

If all scorer fails, raise an error.

82e6dc1

Update warning message.

056b40c

Update raised exception

2aa8ac7

simonandras commented Apr 10, 2022

View reviewed changes

sklearn/metrics/_scorer.py Outdated Show resolved Hide resolved

thomasjpfan reviewed Apr 10, 2022

View reviewed changes

sklearn/metrics/_scorer.py Outdated Show resolved Hide resolved

Making the changes of the rewiev. TODO: update the tests and add new …

30363eb

…one to the new parts.

simonandras commented Apr 10, 2022

View reviewed changes

sklearn/model_selection/_validation.py Outdated Show resolved Hide resolved

thomasjpfan reviewed Apr 10, 2022

View reviewed changes

sklearn/model_selection/_validation.py Outdated Show resolved Hide resolved

sklearn/model_selection/_validation.py Outdated Show resolved Hide resolved

simonandras added 2 commits April 12, 2022 21:19

Update to a list.

5caba43

Added formatted exception to the warnings. All the failing scorers me…

e88cfde

…ssage will show up there.

simonandras commented Apr 16, 2022

View reviewed changes

sklearn/model_selection/_validation.py Outdated Show resolved Hide resolved

Calling the function with positional arguments instead.

69c61e8

thomasjpfan reviewed Apr 17, 2022

View reviewed changes

simonandras added 2 commits April 18, 2022 12:47

added test_multimetric_scorer_returning_exceptions_in_dictianary. TOD…

01ccc1a

…O: a non-regression test for the original issue.

Update test_cross_validate_failing_scorer to cover the failing and no…

218f5bd

…n-failing multimetric case.

thomasjpfan reviewed Apr 24, 2022

View reviewed changes

sklearn/metrics/_scorer.py Outdated Show resolved Hide resolved

sklearn/model_selection/tests/test_validation.py Outdated Show resolved Hide resolved

simonandras added 3 commits April 30, 2022 13:38

Changing '2' to more explicit 'score_2' in the test_cross_validate_fa…

1ab52b2

…iling_scorer

Remove not used function from the imports.

b56b585

thomasjpfan reviewed May 3, 2022

View reviewed changes

sklearn/metrics/_scorer.py Outdated Show resolved Hide resolved

simonandras and others added 6 commits May 11, 2022 21:28

Changing _MultimetricScorer API: now the 'scorers' parameter is a pos…

ea793c9

…itional (dict) instead of a depacked dict using **.

Adding changelog.

a6e05c7

small changes to changelelog

8100661

Changing the changelog again.

176d406

Merge branch 'main' into cross_validate__with_multiple_scorer_nan_issue

84a0808

Changing the number in the changelogand a small change in the test_cr…

fd616b9

…oss_validate_failing_scorer: checking for '_score_2' instead of 'score_2'.

simonandras commented May 11, 2022

View reviewed changes

doc/whats_new/v1.1.rst Outdated Show resolved Hide resolved

Correct doc.

f7857ad

thomasjpfan reviewed May 13, 2022

View reviewed changes

simonandras added 2 commits May 21, 2022 13:49

Changing changelog based on rewiev.

70c59e0

Removing assertions from _validation.py and leaving comments instead.

03d33c8

Merge remote-tracking branch 'upstream/main' into cross_validate__wit…

1f3e60b

…h_multiple_scorer_nan_issue

thomasjpfan approved these changes Jun 5, 2022

View reviewed changes

thomasjpfan added the Waiting for Reviewer label Jul 23, 2022

cmarmo added Waiting for Second Reviewer First reviewer is done, need a second one! and removed Waiting for Reviewer labels Oct 20, 2022

glemaitre self-requested a review December 28, 2022 13:42

glemaitre added 2 commits December 28, 2022 14:42

Merge branch 'main' into cross_validate__with_multiple_scorer_nan_issue

94fab7e

DOC move entry changelog

89cf53e

glemaitre reviewed Dec 28, 2022

View reviewed changes

doc/whats_new/v1.3.rst Outdated Show resolved Hide resolved

sklearn/metrics/_scorer.py Outdated Show resolved Hide resolved

sklearn/metrics/tests/test_score_objects.py Outdated Show resolved Hide resolved

sklearn/metrics/tests/test_score_objects.py Outdated Show resolved Hide resolved

Apply suggestions from code review

7be9ca1

glemaitre approved these changes Dec 28, 2022

View reviewed changes

glemaitre enabled auto-merge (squash) December 28, 2022 14:04

glemaitre merged commit d17d0f9 into scikit-learn:main Dec 28, 2022

jjerphan pushed a commit to jjerphan/scikit-learn that referenced this pull request Jan 3, 2023

FIX cross_validate with multimetric scoring returns the non-failed …

ce0581a

…scorers results even if some fail (scikit-learn#23101) Co-authored-by: Guillaume Lemaitre <[email protected]>

jjerphan pushed a commit to jjerphan/scikit-learn that referenced this pull request Jan 20, 2023

FIX cross_validate with multimetric scoring returns the non-failed …

6a62f43

…scorers results even if some fail (scikit-learn#23101) Co-authored-by: Guillaume Lemaitre <[email protected]>

jjerphan pushed a commit to jjerphan/scikit-learn that referenced this pull request Jan 20, 2023

FIX cross_validate with multimetric scoring returns the non-failed …

381ee65

…scorers results even if some fail (scikit-learn#23101) Co-authored-by: Guillaume Lemaitre <[email protected]>

jjerphan pushed a commit to jjerphan/scikit-learn that referenced this pull request Jan 23, 2023

FIX cross_validate with multimetric scoring returns the non-failed …

5f01a31

…scorers results even if some fail (scikit-learn#23101) Co-authored-by: Guillaume Lemaitre <[email protected]>

adrinjalali pushed a commit that referenced this pull request Jan 24, 2023

FIX cross_validate with multimetric scoring returns the non-failed …

9585ed0

…scorers results even if some fail (#23101) Co-authored-by: Guillaume Lemaitre <[email protected]>

Uh oh!

FIX cross_validate with multimetric scoring returns the non-failed scorers results if there is a failing scorer #23101

FIX cross_validate with multimetric scoring returns the non-failed scorers results if there is a failing scorer #23101

Uh oh!

Conversation

simonandras commented Apr 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Any other comments?

Uh oh!

Uh oh!

thomasjpfan left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thomasjpfan left a comment

Choose a reason for hiding this comment

Uh oh!

simonandras commented Apr 18, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

thomasjpfan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

thomasjpfan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

simonandras commented May 21, 2022

Uh oh!

thomasjpfan commented May 21, 2022

Uh oh!

simonandras commented Jun 4, 2022

Uh oh!

thomasjpfan commented Jun 4, 2022

Uh oh!

simonandras commented Jun 4, 2022

Uh oh!

thomasjpfan left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thomasjpfan Jun 5, 2022

Choose a reason for hiding this comment

Uh oh!

simonandras commented Jul 23, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

glemaitre left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

FIX `cross_validate` with multimetric scoring returns the non-failed scorers results if there is a failing scorer #23101

FIX `cross_validate` with multimetric scoring returns the non-failed scorers results if there is a failing scorer #23101

simonandras commented Apr 10, 2022 •

edited

Loading

thomasjpfan left a comment •

edited

Loading

simonandras commented Apr 18, 2022 •

edited

Loading

thomasjpfan left a comment •

edited

Loading