[MRG + 2] Rename scorers like `mse` to `neg_mse` #7261

betatim · 2016-08-27T09:25:13Z

Reference Issue

Fixes #2439

What does this implement/fix? Explain your changes.

Renaming scorers for which smaller is better (like MSE) to neg_mse so that they fit the idea of "bigger is better".

mean_squared_error
mean_absolute_error
log_loss
median_absolute_error

Rename scorers like MSE to neg_MSE so that it is less surprising that they return negative values.

get_scrorer now warns if you use an old name for a scorer and tests have been updated to use new naming convention.

ogrisel · 2016-08-27T13:13:56Z

sklearn/metrics/scorer.py

-mean_absolute_error_scorer = make_scorer(mean_absolute_error,
-                                         greater_is_better=False)
-median_absolute_error_scorer = make_scorer(median_absolute_error,
-                                           greater_is_better=False)


Because those variables do not have a leading _ the can unfortunately be considered public API even if not in the documentation. I think we should maintain backward compat aliases until 0.20:

# Backward compat alias to keep until the end of the deprecation period (0.20) mean_squared_error_scorer = neg_mean_squared_error_scorer mean_absolute_error_scorer = neg_mean_absolute_error_scorer median_absolute_error_scorer = neg_median_absolute_error_scorer

Maybe we could add a deprecation_msg option to make_scorer to make sure that the deprecation warning is raised when the user actually call the scorer instead of raising the warning in get_scorer that only covers a subset of the public API.

neg_median_absolute_error_scorer = make_scorer(median_absolute_error, greater_is_better=False) median_absolute_error_scorer = make_scorer( median_absolute_error, greater_is_better=False, deprecation_msg='Scoring method median_absolute_error was renamed to ' 'neg_median_absolute_error in version 0.18 and will be ' 'removed in 0.20.' )

why not deprecated(make_scorer(median_absolute_error, greater_is_better=False)) ?

Ah, it's an object, not a function, so it doesn't go into the right branch in deprecated, right? We could still wrap the __call__ method, right?

deprecated("message")._decorate_fun(make_scorer(median_absolute_error, greater_is_better=False))
seems to be the right solution imho.

Neither of the deprecated()._decorate_* work because make_scorer doesn't return a class nor a function :-/ To get the __name__ of a class instance you have to do obj.__class__.__name__.

I like handling the deprecation stuff as a wrapper instead of extra argument. I made a first proposal in 09cd4c7. It still has a few problems but it would allow you to do:

deprecated('hello')(make_scorer(median_absolute_error, greater_is_better=False))

This will however print Thing _PredictScorer in deprecated... and I think it will be quite hard to find out the correct name to print in a generic way inside deprecated() ... but do we want to cook something that is special to handling make_scorer??

I would rather avoid making the sklearn.utils.deprecation.deprecated wrapper too complex. I think it's good enough to not use it at all in this case and just use a _deprecation_msg attribute on _BaseScorer as suggested in #7261 (comment).

I wouldn't want to make the wrapper to complex, but I feel that deprecating the call method is really the easiest thing. We also don't need to change the wrapper for that. It will then say __call__ is deprecated and we can let the message explain that.

Maintain old names during the deprecation period and update tests to use better variables names.

ogrisel · 2016-08-29T08:53:01Z

There is a failed doctest in model_evaluation.rst:

https://travis-ci.org/scikit-learn/scikit-learn/jobs/155644998#L3998

ogrisel · 2016-08-29T08:55:27Z

Also line 204 of get_scorer should also filter out the deprecated scores from the list of valid options in its error message.

betatim · 2016-08-29T09:20:52Z

Thanks. I had updated the expected output but not the actual code ... both fixed now.

amueller · 2016-08-29T22:12:59Z

sklearn/metrics/scorer.py

               median_absolute_error=median_absolute_error_scorer,
               mean_absolute_error=mean_absolute_error_scorer,
               mean_squared_error=mean_squared_error_scorer,
               accuracy=accuracy_scorer, roc_auc=roc_auc_scorer,
               average_precision=average_precision_scorer,
-               log_loss=log_loss_scorer,


we can't remove that, right? It's public API.

Correct. Wasn't paying enough attention apparently :(

amueller · 2016-08-29T22:16:52Z

thanks for working on this, it's very important I think.

jnothman · 2016-08-29T22:29:27Z

Sorry I've not been attuned to this thread. As I've proposed before, I think all deprecation should be in get_scorer: that's where the warning should be raised. The scorers themselves are not public API. Is that mistaken?

amueller · 2016-08-29T22:32:09Z

Hm the scorer objects don't have underscores in their names, and the SCORERS dict that contains the instances is mentioned in the docs. I considered them public API.

jnothman · 2016-08-29T23:39:27Z

SCORERS yes, but I think that was a bad design choice :) if we deprecate SCORERS at the same time, do we win? I don't think the scorers themselves were ever intended as public. They were never included in classes.rst, they are as useful as their names, and they should be immutable. In practice we have lots of names imported or declared in non-underscored modules that are not assumed to be public; these are mere module-level variables. Next you'll argue that sklearn.metrics.scorer.qualified_name is also public API!

Treat (callable) instances different from functions in the deprecated decorator

betatim · 2016-08-30T06:55:18Z

My original implementation was via get_scorer (ef64969) but after chatting with @ogrisel we decided that people will have been "naughty" and used both the sklearn.metrics.scorer.qualified_name and SCORERS from the outside (and that they should get a deprecation warning).

I think if we can organise for those users to get a warning as well without too much acrobatics then we should. No opinion on whether that warning should be the rename deprecation warning or a "stop using these internals" warning.

GaelVaroquaux · 2016-08-30T09:36:42Z

sklearn/metrics/scorer.py

@@ -33,14 +34,15 @@


 class _BaseScorer(six.with_metaclass(ABCMeta, object)):
-    def __init__(self, score_func, sign, kwargs):
+    def __init__(self, score_func, sign, kwargs, deprecation_msg=None):


I think that we will want to get rid of the "deprecation_msg" in two release. Maybe add a comment here saying that this should be removed for 0.20.

Maybe we could rename that parameter to _deprecation_msg to make it explicit that this is not public API.

Or even not put _deprecation_msg in the __init__ at all but just as None initialized attribute on the _BaseScorer class and manually use

mean_scorer_error._deprecation_msg = "the message"

to activate the deprecation.

GaelVaroquaux · 2016-08-30T09:46:48Z

I've finished my pass. I am 👍 with this PR once my small remarks are addressed.

Also, flake8 is unhappy (check travis). I guess that you should address that.

jnothman · 2016-09-05T12:16:45Z

sklearn/metrics/scorer.py


-    @abstractmethod


I think you can keep this.

betatim · 2016-09-06T06:39:33Z

bump

jnothman · 2016-09-06T06:45:23Z

:)

jnothman · 2016-09-06T06:48:30Z

Examples not yet updated?

examples/model_selection/plot_underfitting_overfitting.py
examples/plot_kernel_ridge_regression.py

ogrisel · 2016-09-06T08:51:54Z

The fix it-self looks good to me. +1 for merge once the examples are updated.

betatim · 2016-09-06T09:28:22Z

Updated.

jnothman · 2016-09-06T10:01:18Z

examples/plot_kernel_ridge_regression.py


-plt.plot(train_sizes, test_scores_svr.mean(1), 'o-', color="r",
+plt.plot(train_sizes, -test_scores_svr.mean(1), 'o-', color="r",


I think you should leave off this negation and change the ylabel to say "Negative".

But what you have here is okay; better than what's currently plotted at http://scikit-learn.org/stable/auto_examples/plot_kernel_ridge_regression.html.

@ogrisel, any preference for plotting -MSE vs plotting MSE?

I find plotting MSE and seeing it decrease easier on my brain than having to think "error, smaller is better but here it is negated so going up means closer to zero ..."

The question from my perspective is whether it's worth encouraging users to negate output. Or whether, regardless of scoring function we'd like to encourage the convention that better is up.

I have no opinion on this beyond previous comment. Who is going to arbitrate/decide?

No strong opinion here. I'm fine with the -

Merge then?

jnothman · 2016-09-06T10:03:01Z

LGTM other than that too

lesteve · 2016-09-08T13:26:29Z

It would be nice to add tests to make sure that accessing the old scorers (via the different ways mentioned above) does give DeprecationWarning as expected.

@betatim bonus points if you feel like tackling my comment from a week ago which got lost in other (very likely more important) considerations.

betatim · 2016-09-08T15:36:57Z

Tested.

I skipped testing the from sklearn.metrics.scorer import mean_squared_error_scorer kind of access. I think it would require a bit of import acrobatics to do based on a string name and not sure that is worth it.

amueller · 2016-09-08T16:16:09Z

merge?

betatim · 2016-09-08T18:13:21Z

Make it so.

GaelVaroquaux · 2016-09-08T18:40:43Z

Yey!

(I didn't do much, but I pressed the green button. What a pleasure!)

GaelVaroquaux · 2016-09-08T18:40:56Z

Thanks heaps @betatim !

amueller · 2016-09-08T18:42:07Z

awesome, thanks! this was was overdue

betatim · 2016-09-08T20:22:37Z

Thanks a lot for all the comments and extra 👀!

raghavrv · 2016-09-09T21:36:14Z

Yay 🍻 Much needed PR!

raghavrv · 2016-09-12T14:02:56Z

@jnothman This also closes #6028 #5023 right?

amueller · 2016-09-12T14:14:53Z

It does. Closed those. Thanks @raghavrv

betatim changed the title ~~Rename scorers like mse to neg_mse~~ [WIP] Rename scorers like mse to neg_mse Aug 27, 2016

betatim added 2 commits August 27, 2016 12:06

Rename smaller-is-better scoring metrics

9b2e53e

Rename scorers like MSE to neg_MSE so that it is less surprising that they return negative values.

Update tests to new scorer names

7e079c0

betatim changed the title ~~[WIP] Rename scorers like mse to neg_mse~~ [MRG] Rename scorers like mse to neg_mse Aug 27, 2016

Introduce deprecation warning and fix tests

ef64969

get_scrorer now warns if you use an old name for a scorer and tests have been updated to use new naming convention.

ogrisel reviewed Aug 27, 2016
View reviewed changes

betatim added 2 commits August 27, 2016 21:11

Keep old metric names for deprecation period

9d695d7

Maintain old names during the deprecation period and update tests to use better variables names.

Keep old metric names in tests

d464972

ogrisel added this to the 0.18 milestone Aug 29, 2016

Do not show deprecated scorers in list of valid names

42fe520

amueller reviewed Aug 29, 2016
View reviewed changes

amueller added the Blocker label Aug 29, 2016

betatim added 2 commits August 30, 2016 07:34

Do not remove log_loss scorer during deprecation period

3d16c47

Distinguish instances from functions in deprecated

09cd4c7

Treat (callable) instances different from functions in the deprecated decorator

GaelVaroquaux reviewed Aug 30, 2016
View reviewed changes

jnothman reviewed Sep 5, 2016
View reviewed changes

sklearn/metrics/scorer.py

@abstractmethod

Copy link

Member

jnothman Sep 5, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can keep this.

Keep abstractmethod decorator

2ba3478

Update examples to new scorer names

6673f75

jnothman reviewed Sep 6, 2016
View reviewed changes

jnothman changed the title ~~[MRG + 1] Rename scorers like mse to neg_mse~~ [MRG + 2] Rename scorers like mse to neg_mse Sep 6, 2016

Test that warning is raised when using deprecated scorer

3935404

GaelVaroquaux merged commit 4b2304f into scikit-learn:master Sep 8, 2016

betatim deleted the negative-scorers branch September 8, 2016 20:22

MechCoder mentioned this pull request Sep 12, 2016

Suppress warning in auto-generated notebooks scikit-optimize/scikit-optimize#221

Closed

This was referenced Sep 12, 2016

More intuitive scoring argument for loss and error #5023

Closed

Deprecated negative valued scorers #6028

Closed

qinhanmin2014 mentioned this pull request Sep 10, 2019

[MRG] API Replace scorer brier_score_loss with neg_brier_score #14898

Merged


		plt.plot(train_sizes, test_scores_svr.mean(1), 'o-', color="r",
		plt.plot(train_sizes, -test_scores_svr.mean(1), 'o-', color="r",

Uh oh!

[MRG + 2] Rename scorers like mse to neg_mse #7261

[MRG + 2] Rename scorers like mse to neg_mse #7261

Uh oh!

Conversation

betatim commented Aug 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Reference Issue

What does this implement/fix? Explain your changes.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ogrisel Aug 27, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amueller Aug 29, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amueller Aug 29, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ogrisel commented Aug 29, 2016

Uh oh!

ogrisel commented Aug 29, 2016

Uh oh!

betatim commented Aug 29, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amueller commented Aug 29, 2016

Uh oh!

jnothman commented Aug 29, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

amueller commented Aug 29, 2016

Uh oh!

jnothman commented Aug 29, 2016

Uh oh!

betatim commented Aug 30, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

GaelVaroquaux commented Aug 30, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

betatim commented Sep 6, 2016

Uh oh!

jnothman commented Sep 6, 2016

Uh oh!

jnothman commented Sep 6, 2016

Uh oh!

ogrisel commented Sep 6, 2016

Uh oh!

betatim commented Sep 6, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

[MRG + 2] Rename scorers like `mse` to `neg_mse` #7261

[MRG + 2] Rename scorers like `mse` to `neg_mse` #7261

betatim commented Aug 27, 2016 •

edited

Loading

ogrisel Aug 27, 2016 •

edited

Loading

amueller Aug 29, 2016 •

edited

Loading

amueller Aug 29, 2016 •

edited

Loading

jnothman commented Aug 29, 2016 •

edited

Loading