Use cross_validation.cross_val_score with metrics.precision_recall_fscore_support #1837

SolomonMg · 2013-04-03T21:03:59Z

I'd like to use cross_validation.cross_val_score with metrics.precision_recall_fscore_support so that I can get all relevant cross-validation metrics without having to run my cross-validation once for accuracy, once for precision, once for recall, and once for f1. But when I try this I get a ValueError:

from sklearn.datasets import fetch_20newsgroups

from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn import metrics
from sklearn import cross_validation
import numpy as np

data_train = fetch_20newsgroups(subset='train', #categories=categories,
                                shuffle=True, random_state=42)
clf = LinearSVC(loss='l1', penalty='l2')
vectorizer = TfidfVectorizer(
  sublinear_tf=False, 
  max_df=0.5,
  min_df=2, 
  ngram_range = (1,1),
  use_idf=False,
  stop_words='english')

X_train = vectorizer.fit_transform(data_train.data)

# Cross-validate:
scores = cross_validation.cross_val_score(
  clf, X_train, data_train.target, cv=5, 
  scoring=metrics.precision_recall_fscore_support)

amueller · 2013-04-04T11:24:22Z

You would need to do scoring=AsScorer(metrics.precision_recall_fscore_support) but that also doesn't work. Currently the scoring interface only supports returning a single value. I gotta fix that. There is also some work by @jnothman that may help you.

jnothman · 2013-04-04T12:00:19Z

Yes, @solomonm, something like this was my initial motivation for #1768. Allowing scoring functions to return a tuple, as you have attempted here, would suffice for some cases, but not for *SearchCV where the scores need to be summable and comparable.

In that PR I implemented an approach in which Scorer objects may provide a collection of named scores, providing a PRFScorer that would handle your use-case; I only made changes affecting GridSearchCV and RandomizedSearchCV.

I would still like something like this available, but it needs a fresh understanding of what a Scorer does and its API (see discussion at #1774); and to be useful in a parameter search, it requires extensible output from *SearchCV.fit(...) as proposed in #1787, which is awaiting feedback.

amueller · 2013-04-04T13:16:31Z

I think for for @solomonm a quick hack in cross_val_score would probably help (as I think he doesn't want to compare scores). I am really really sorry I didn't have more time to look at your work and it is really high up in my sklearn priority queue.

SolomonMg · 2013-04-04T22:15:58Z

@andreas Totally, this is really only useful to help the analyst assess performance on multiple measures (very common use case). I doubt using this for param tuning will be a common use case.
—
Sent from Mailbox for iPhone

On Thu, Apr 4, 2013 at 6:16 AM, Andreas Mueller [email protected]
wrote:

I think for for @solomonm a quick hack in cross_val_score would probably help (as I think he doesn't want to compare scores). I am really really sorry I didn't have more time to look at your work and it is really high up in my sklearn priority queue.

Reply to this email directly or view it on GitHub:
#1837 (comment)

amueller · 2013-04-05T08:48:32Z

@solomonm just a quick github reminder: if you use @, someone will get a ping / email. You just pinged some random guy ;)

Do you know what to change to make this work? I will try to do it on the weekend.

arjoly · 2013-07-25T13:16:03Z

It is addressed by the new scorer interface.

amueller · 2013-07-25T13:20:23Z

err not by the one that was already merged, right?

arjoly · 2013-07-25T13:33:33Z

I have closed this a bit fast.

jnothman · 2013-07-25T13:33:43Z

the multi-metric support was not merged, pending further discussion of what it should look like.

mwjackson · 2015-07-08T10:17:10Z

What is the status of this?

amueller · 2015-07-11T21:11:27Z

@mwjackson It is part of @rvraghav93's google summer of code this year. nothing finished yet, though.

arnavsharma93 · 2015-08-30T22:21:15Z

Any update on this?

amueller · 2015-08-31T18:35:29Z

same as in july ;) getting there.

jnothman · 2015-08-31T23:30:52Z

I think the elided "now that GSoC has concluded" might have been intended!

On 1 September 2015 at 04:35, Andreas Mueller [email protected]
wrote:

same as in july ;) getting there.

—
Reply to this email directly or view it on GitHub
#1837 (comment)
.

raghavrv · 2015-10-23T15:53:33Z

I take the full blame for not having finished this during my GSoC... :)

Now that #4294 is merged, I will start working on it shortly and hope to get this merged ASAP!!

hlin117 · 2016-04-07T04:49:04Z

Progress?

jnothman · 2016-04-07T04:51:39Z

See #2759 (comment) for the latest!

jnothman · 2017-02-07T06:25:27Z

Although #7388 appears close to being accepted for merge, I'd like to discuss the broader question of whether extending scoring is the right way to go about this, or whether a simpler, more generic callback for retrieving (or storing) experimental artefacts is appropriate.

When I suggested such a hook for diverse diagnostics years ago, someone -- @GaelVaroquaux I assume -- suggested it's not so hard to roll your own CV/search so why need such a callback? Many features have been added since to *SearchCV (cv_results_ including timing and training score, safe indexing for different input types, on_error, potentially use_warm_start in #8230, etc.) by which I would suggest that roll-your-own is no longer the right answer.

Advantages of extending scoring as in #7388:

intuitive specification of additional scorers.
results include mean, std, rank.
avoids user effort/error in terms of weighted averaging (wrt iid) and calling prediction methods. (I don't even think we provide the user with test_sample_counts` to make that averaging easy to DIY.)
can still be hacked to store arbitrary data in global variables or out of memory as long as float is returned (although X_test needs to be mapped back to CV split index if associating the data with a particular split is important).
it's implemented; and it's still possible to implement a more general diagnostic callback

Disadvantages:

restricted to single float per scorer, and even if we extend this to handle arrays or tuples, we will probably require their shape to be fixed across all calls.
scoring is no longer semantically appropriate, as we only use one metric to score, and the others for diagnostics. refit becomes a bit awkward.
limited to being a function of (estimator, X_test, y_test).
I fear that if we want to start seamlessly returning per-class results or avoid duplicated prediction work (particularly useful for things like KNN, kernel SVM), we will be engineering a beast.

What's the right compromise?

jnothman · 2017-02-08T22:58:07Z

I've added a couple of points to my previous comment.

jnothman mentioned this issue Apr 11, 2013

Cross-validation returning multiple scores #1850

Closed

jnothman mentioned this issue Jul 1, 2013

MRG add log loss (cross-entropy loss) to metrics #2013

Closed

jnothman mentioned this issue Jul 17, 2013

allow cross_val_score to return arbitrary types #2157

Closed

arjoly closed this as completed Jul 25, 2013

arjoly reopened this Jul 25, 2013

raghavrv mentioned this issue Apr 22, 2016

[MRG+3] ENH Restructure grid_scores_ into a dict of 1D (numpy) (masked) arrays that can be imported into pandas as a DataFrame. #6697

Merged

10 tasks

raghavrv mentioned this issue Sep 11, 2016

[MRG + 2] ENH Allow cross_val_score, GridSearchCV et al. to evaluate on multiple metrics #7388

Merged

16 tasks

amueller added this to the 1.0 milestone Oct 27, 2016

jnothman mentioned this issue Feb 8, 2017

[WIP] ENH arbitrary diagnostic_func callback in GridSarchCV #8317

Closed

amueller closed this as completed in #7388 Jul 7, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use cross_validation.cross_val_score with metrics.precision_recall_fscore_support #1837

Use cross_validation.cross_val_score with metrics.precision_recall_fscore_support #1837

SolomonMg commented Apr 3, 2013

amueller commented Apr 4, 2013

jnothman commented Apr 4, 2013

amueller commented Apr 4, 2013

SolomonMg commented Apr 4, 2013

I think for for @solomonm a quick hack in cross_val_score would probably help (as I think he doesn't want to compare scores). I am really really sorry I didn't have more time to look at your work and it is really high up in my sklearn priority queue.

amueller commented Apr 5, 2013

arjoly commented Jul 25, 2013

amueller commented Jul 25, 2013

arjoly commented Jul 25, 2013

jnothman commented Jul 25, 2013

mwjackson commented Jul 8, 2015

amueller commented Jul 11, 2015

arnavsharma93 commented Aug 30, 2015

amueller commented Aug 31, 2015

jnothman commented Aug 31, 2015

raghavrv commented Oct 23, 2015

hlin117 commented Apr 7, 2016

jnothman commented Apr 7, 2016

jnothman commented Feb 7, 2017 •

edited

Loading

jnothman commented Feb 8, 2017

Use cross_validation.cross_val_score with metrics.precision_recall_fscore_support #1837

Use cross_validation.cross_val_score with metrics.precision_recall_fscore_support #1837

Comments

SolomonMg commented Apr 3, 2013

amueller commented Apr 4, 2013

jnothman commented Apr 4, 2013

amueller commented Apr 4, 2013

SolomonMg commented Apr 4, 2013

I think for for @solomonm a quick hack in cross_val_score would probably help (as I think he doesn't want to compare scores). I am really really sorry I didn't have more time to look at your work and it is really high up in my sklearn priority queue.

amueller commented Apr 5, 2013

arjoly commented Jul 25, 2013

amueller commented Jul 25, 2013

arjoly commented Jul 25, 2013

jnothman commented Jul 25, 2013

mwjackson commented Jul 8, 2015

amueller commented Jul 11, 2015

arnavsharma93 commented Aug 30, 2015

amueller commented Aug 31, 2015

jnothman commented Aug 31, 2015

raghavrv commented Oct 23, 2015

hlin117 commented Apr 7, 2016

jnothman commented Apr 7, 2016

jnothman commented Feb 7, 2017 • edited Loading

jnothman commented Feb 8, 2017

jnothman commented Feb 7, 2017 •

edited

Loading