Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Precision/Recall AUC scoring function #5992

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
halfak opened this issue Dec 9, 2015 · 5 comments
Closed

Precision/Recall AUC scoring function #5992

halfak opened this issue Dec 9, 2015 · 5 comments

Comments

@halfak
Copy link

halfak commented Dec 9, 2015

I'd like to run GridSearch optimizing for area under the precision/recall curve. It's great that "roc_auc" is readily available, but I'd like to be able to optimize for "pr_auc" too. See http://pages.cs.wisc.edu/~jdavis/davisgoadrichcamera2.pdf for a discussion of the meaningful differences between PR_AUC and ROC_AUC.

See also precision_recall_curve() and auc().

@raamana
Copy link
Contributor

raamana commented Dec 10, 2015

Have you looked at sklearn.metrics.make_scorer ?

There is an example also:

 >>> from sklearn.metrics import fbeta_score, make_scorer
    >>> ftwo_scorer = make_scorer(fbeta_score, beta=2)
    >>> ftwo_scorer
    make_scorer(fbeta_score, beta=2)
    >>> from sklearn.model_selection import GridSearchCV
    >>> from sklearn.svm import LinearSVC
    >>> grid = GridSearchCV(LinearSVC(), param_grid={'C': [1, 10]},
    ...                     scoring=ftwo_scorer)

@halfak
Copy link
Author

halfak commented Dec 10, 2015

Ahh yes. In fact, I am using make_scorer and a custom implementation of pr_auc right now. Here's my code (feedback welcome).

def pr_auc_score(y_true, y_score):
    """
    Generates the Area Under the Curve for precision and recall.
    """
    precision, recall, thresholds = \
        precision_recall_curve(y_true, y_score[:, 1])
    return auc(recall, precision, reorder=True)

pr_auc_scorer = make_scorer(pr_auc_score, greater_is_better=True,
                            needs_proba=True)

It just seems to me that, if roc_auc is directly available and it's easy to implement, pr_auc should be available too.

@amueller
Copy link
Member

that is "average_precision" right?

@amueller
Copy link
Member

@halfak
Copy link
Author

halfak commented Dec 10, 2015

Oh great! I did not realize that. Thank you for pointing it out.

From the docs:

This score corresponds to the area under the precision-recall curve.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants