The general approach looks good to me, I don't have a good overview of how it might interact with @jnothman's PRs, though. I'm kinda busy, so don't wait for my 👍 to go ahead. Btw, @ogrisel might also have an opinion, as the original interface was pretty much his idea.

I'm definitely +1 for getting rid of needs_threshold. It didn't make much sense checking needs_threshold on __call__ anyway when it was a property of the metric.

Was there a reason for not using -loss previously?

And as to my PRs, I'm only require that there be a way to get more information than a single objective score (multiple class scores; P/R/F) back from the scorer. E.g. the scorer could optionally return a tuple of (objective_score, score_dict). You might then use the following, which I don't think is very pretty but does the job:

score = scorer(estimator, X, y)
try:
    score, score_data = score
except TypeError:
    score_data = {'score': score}

:s

I know it's still WIP, but I thought I'd just mention this assert fail from travis so long:

======================================================================
FAIL: sklearn.tests.test_cross_validation.test_cross_val_score_with_score_func_regression
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/travis/build/scikit-learn/scikit-learn/sklearn/tests/test_cross_validation.py", line 403, in test_cross_val_score_with_score_func_regression
    assert_array_almost_equal(mse_scores, expected_mse, 2)
  File "/usr/lib/python2.7/dist-packages/numpy/testing/utils.py", line 800, in assert_array_almost_equal
    header=('Arrays are not almost equal to %d decimals' % decimal))
  File "/usr/lib/python2.7/dist-packages/numpy/testing/utils.py", line 636, in assert_array_compare
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 2 decimals
(mismatch 100.0%)
 x: array([ -763.0785884 ,  -553.16968926,  -274.38203955,  -273.26281775,
       -1681.99954932])
 y: array([  763.07,   553.16,   274.38,   273.26,  1681.99])
>>  raise AssertionError('\nArrays are not almost equal to 2 decimals\n\n(mismatch 100.0%)\n x: array([ -763.0785884 ,  -553.16968926,  -274.38203955,  -273.26281775,\n       -1681.99954932])\n y: array([  763.07,   553.16,   274.38,   273.26,  1681.99])')

That asside, looks good to me so far. 👍

Strange, make test passed on my box. But this is to be expected, of course.

Yeah passed on mine too when I checked this branch. Travis is annoying like that sometimes haha

@jnothman the reason not to use -loss was that I didn't like returning negative mean squared errors to the user. That is just very counter-intuitive.

I guess I was asking whether there was more than that: I actually find it quite clear to report negative loss being climbed positively towards 0. At least, loss is strictly non-negative (as opposed to some ranking metrics) so it should be unambiguous. The confusion is that it's the opposite of what you get when you use the metric function directly.

@larsmans, despite your fix for CV over MSE, there's a more substantial deprecation issue: If someone passes a loss_func (deprecated) to cross_val_score or GridSearchCV you actually need to negate the output for them. Might as well add to the DeprecationWarning where loss_func is passed that the equivalent scoring parameter will return a negated version of the result.

Or we could have scorers returning tuples so this doesn't happen.

@jnothman You mean an (mse, 'minimize') tuple? I'm not too fond of that. It's possible to screw that up by returning different values from different calls. Always maximize is a robust and stable strategy that requires very little documentation (my favorite metric of complexity). I'm willing to jump through the deprecation hoops on this one if we can really simplify the API.

Compare the situation to that in scipy.optimize. That has an "always minimize" strategy, which is so obvious that when you screw it up, you're so ashamed that you fix the bug without posting issues about it ;)

No, I mean a (-mse, mse) tuple, or (-mse, {'loss': mse}). Certainly not simple; but again I'd really like to see P, R and F being returned somehow from a scorer, and that will add complexity somewhere, it just needs to be minimal...

(And I didn't mean to imply that the deprecation was excessively imposing,
but that you'd need to add it to your PR)

Yes, I agree with the tuple return value. For PRF I'd say we return values of type

_PRF1Tuple = namedtuple("_PRF1Tuple", "f1 precision recall")

That's simpler than a (scalar, dict) pair.

Yes, I see some advantages in the namedtuple. I had thought at one point a dict where the field 'score' was required, and if a scalar is returned it's interpreted as such. If it returns a namedtuple, we'll probably land up calling _asdict() given something like #2079, anyway, but I have no real objection to assuming the first element of a namedtuple is the objective.

(One further annoyance with the namedtuple for PRF is that the precision_recall_fscore_support function returns things in that order, not fscore first.)

We could name the scorer differently, like fscorer_with_precision_recall.

I had a quick look and my general feeling is that I like this approach and think that it is an improvement over the current code.

The good news is that the current code wasn't in a released version, right, So we won't have to go through deprecation and users hating us, right?

No scorer-related stuff was ever released, so we're safe.

Do we want to rename Scorer to Evaluator? I find Scorer rather ambiguous.

Rebased on master. I can haz reviews?

I don't particularly mind Evaluator... but most users won't see this name anyway.

Would you advocate changing the scoring param in *SearchCV and cross_val_score to evaluator or evaluation? Are those names less ambiguous?

Remember also that the default Scorer is estimator.__class__.score, so it's extending on an existing API naming convention.

Do we want to rename Scorer to Evaluator?

Sounds very French to me (Lars, are we having a bad influence on you?).

I actually like Scorer as it relates well to 'cross_val_score', and the
score method.

Very well :)

This looks like a nice improvement!

Once my comments above regarding the changes in F1 scores are addressed, I am +1 for merge.

One think I liked with the previous API that made need_threshold and score_func public attributes of the scorer instance is that it was possible to introspect the scoring object in a custom fit_grid_point implementation to combine several operations on the raw predictions that could be in a third party model evaluation tool such as hyperopt.

For instance I would like to have grid search fit_grid_point calls be able to:

• record the raw prediction time (the wall clock duration of the call to predict / predict_proba or / decision_function
• dump the raw predictions (or predict probas output) on the drive (for manual introspection of the common classification mistakes or to implement ensemble strategies such as Caruana et al., 2004 for instance)
• compute ROC-AUC, PR-AUC, F1, Precision, Recall, at the same time (for binary classification for instance) without calling predict_proba twice on the fitted model

The new public API hides too many implementation details to be able to implement this efficiently from a third party tool like hyperopt or even to implement this as options in the *SearchCV using the public scoring API.

You can do all those things with custom scorers. They're allowed to return tuples holding arbitrary data, and they're still arbitrary Python callables. Partial example:

Results = namedtuple("Results", "f1 precision recall time".split())

def super_scorer(estimator, X, y_true):
    t0 = time()
    y_pred = estimator.predict(X)
    t = time() - t0
    return Results(f1_score(y_true, y_pred), precision(y_true, y_pred), recall(y_true, y_pred), t)

You can do all those things with custom scorers. They're allowed to return tuples holding arbitrary data, and they're still arbitrary Python callables.

Also I forgot one very important use case: compute the same scores on the training split of the data to be able to detect underfitting and overfitting at the same time.

But the meta custom scorer would have to know (hardcoding) if the subscorers "need thresholds" and whether greater is better and so on.

Ideally I would like to have the default GridSearchCV compute the scores on both the train and test splits at the same time.

I think it would be need if it would be possible to have: GridSearchCV(model, param_grid, cv=5, scoring=('roc_auc', 'pr_auc', 'f1', 'precision', 'recall')) supported by the default sklearn tools without having to write a custom score class each time.

Also for the multi-class classification problem, one would also like to collect the confusion matrix (and / or) the detailed per-class P/R/F1 prior to multiclass averaging.

Also make the scorer responsible for dumping the raw predictions proba to the hard drive sounds like a design error (separation of concerns).

I'm not sure if I'm happy with the scope of the PR.
Could you explain in simple words what it does?

I just realized the issue message actually is pretty descriptive, but it is not entirely clear which changes are for the last point and which are for the rest.

I'm not sure if I'm happy with the scope of the PR.

I have suggested the handling of multiple metrics should be a separate PR (as much as I am anxious to have the feature, I want it to be given some thought), and @ogrisel seemed to concur, as we need to consider its interaction with other extensibility in grid search output.

I agree. We should move forward with the scorer refactoring. I plan to work on the multiple metrics features the next couple of days. We want to release soon and we should get stuff in a state that is future-ready but doesn't make changes that we might regret afterwards (like using named tuples ;)

Global +1 on @jnothman and @amueller's latest batch of comments.

@amueller @ogrisel @jnothman The scope of the PR is to simplify the scorers so that they are just functions. There are currently only two commits, the first of which establishes a simple API, the second exemplifies how it can be extended. If we want something that works right now, we can just pull the first commit and leave the structured score stuff for later.

As for @ogrisel's requirements of passing lots of information around and the complaint about separation of concerns: that's independent of the exact API. When you adorn scorers with all kinds of attributes to store generic information about the CV procedure, you're not doing proper separation of concerns.

+1 for merging the first commit.

the function is still missing from the references.

the function is still missing from the references.

I was about to point it out :)

Yes, and the description of the scorer protocol mentioned tuples while it shouldn't for now. Will force-push a new version when I'm confident the tests pass on current master.

Ok, pushed the changes. The test failures are in the second commit.

Ok, pushed the first commit to master. I'm closing this PR because we'll have to rethink or at least discuss the structured score stuff.

Thanks a lot @larsmans :)

Thank you for setting this up, Andy. This API turns out to be exactly what I needed to do better optimization :)