I think this bug was introduced in #9980, pos_label=1 is the culprit, because it overwrites the logic in _binary_clf_curve that requires the user to explicitly set a pos_label.

ping @qinhanmin2014.
We should probably also error, or at least warn if pos_label is not present.

Thanks for the issue and apologies for the inconvenience. Some quick replies, will further investigate and fix this one later this day:
(1) Yes, we need to raise an error when there're two classes and pos_label is not among them (like what we've done in P/R/F)
(2) The NaN bug is not related to my PR but I'll try to look into it. It won't be hard to handle it separately.
(3) I use pos_label=1 because pos_label=None is ill-defined in _binary_clf_curve (and in the repo, see #10010). You might expect that pos_label=None will take the larger label but it actually means pos_label=1 in _binary_clf_curve.
(4) I guess it's acceptable that if users need to set a parameter, they need to rely on make_scorer instead of the default scorer?

Regarding the nan issue, we have a PR #8280. I can take that one but before that @amueller , I want your decision about the edge cases of precision & recall.
(1) (not related to current fix) How to define precision when TP + FP = 0? In precision_score, we raise a warning and return 0. In average_precision_score, we return 0 (in my PR #9980). But in a SO question provided by the contributor of #8280, it's defined as 1.
(2) (related to current fix) How to define recall when TP + FN = 0? In recall_score, we raise a warning and return 0. In average_precision_score, we raise a warning and return nan (the nan issue here). In #8280, the contributor return 1 in average_precision_score (consistent with the SO question he provided), which is inconsistent with our recall_score.
Also note that the nan issue is actually tested by us currently, so it's actually the expected behavior now.

I'm not sure what we should advise. We could ask users also to just do y == pos_label. That would remove the need to call make_scorer. So I think we should either recommend that to users in the error message, or we could try to use the minority class as positive - but we could only do that if we had the training set available, which we don't.

I'm not sure what you mean by "the nan issue".
It's a consequence of an undefined pos_label here, right? I think it's probably the right thing to do if only one label is present.

I'm not sure what you mean by "the nan issue".

The nan issue means that if we only have negative classes in y_true, we'll get nan (along with a warning). The issue here is whether we allow metrics & scorers to output nan and how to solve it.

We could ask users also to just do y == pos_label.

Apologies I don't understand here. I think we add pos_label here so that users don't need to do things like y == pos_label.

I've added the error message in #12313. If you think the nan output you got in this issue is fine, then that PR will close this issue.

Apologies I don't understand here. I think we add pos_label here so that users don't need to do things like y == pos_label.

Well doing y == pos_label is much easier than using make_scorer though.

I feel like all the issues I raised are fixed by #12313. But it's still awkward to actually use this with the scorer interface and I'm not sure what we should be suggesting.

I feel like all the issues I raised are fixed by #12313. But it's still awkward to actually use this with the scorer interface and I'm not sure what we should be suggesting.

If you don't like make_scorer, then we need to define pos_label=None. The difficult thing is how to define it when there's only one label (i.e., should we treat it as positive label or negative label?).

Seems that you don't worry about getting nan, so I'm going to close this one. For pos_label related discussions, we have an issue #10010.

Reopen if you disagree.

no I think this is good.