Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Simultaneous evaluation of several scorers when building validation and learning curves #4330

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jmarinero opened this issue Mar 3, 2015 · 3 comments
Closed

Simultaneous evaluation of several scorers when building validation and learning curves #4330

jmarinero opened this issue Mar 3, 2015 · 3 comments
Labels
New Feature

Comments

@jmarinero
Copy link

Fitting a model several times when building a validation or a learning curve can be costly while the evaluation of the scorer can be very fast.

It would be interesting if it would be possible to evaluate a list of scorers given to:

sklearn.learning_curve.learning_curve
sklearn.learning_curve.validation_curve

as the argument scoring = [scorer1, scorer2, scorer3, ... ]

As a workaround, I'm going to see if my scorer can return a float with extra info in the form of extra properties, like the extra scorers that I want to evaluate (in my particular case, I'm even going to try including confusion matrixes), although I'm quite new at Python, and I don't know if such a thing is possible/easy. Nevertheless such an approach seems unnecessarily twisted.

Learning curve and validation curves are the two functions that are relevant to me. I don't know whether there are any other methods which may be susceptible of this enhancement.

Would you consider as feasible expanding scoring functionality to accept lists of scorers?
@amueller
Copy link
Member

amueller commented Mar 3, 2015

Yes, we definitely want that, and for all cross-validation, gridsearch, etc... methods.
There is some work here: #2759, but it stalled somewhat. We are currently working on a release, but afterwards that is one of my top priorities.

@raghavrv
Copy link
Member

raghavrv commented Mar 4, 2015

Also related #1850

@jnothman
Copy link
Member

jnothman commented Mar 4, 2015

I think this should be closed as a duplicate of #1850; thank you @VoidBrained for voicing your support. I also a couple of years ago toyed with something that could be interpreted as a float (using the __float__ magic method) but also had other information, but it is too hacky: the system needs to just provide a way to do this.

@jnothman jnothman closed this as completed Mar 4, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
New Feature
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jnothman @amueller @jmarinero @raghavrv