I would need a changelog entry for 1.5.1 but the changelog file needs to be updated in main to add the missing section first. Let's do that in another PR.

EDIT: this section is being added in #29078 that is likely to merged first.

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 0af3d32. Link to the linter CI: here}

For the first case, did you succeed to trigger it on a specific use-case. I thought that we would cover it with constant prediction but maybe the case we have probability of 0.0 and 1.0 then we get a constant score.

In main, passing scoring="roc_auc" does not result in constant scores as I would have intuitively expected: the ROC AUC metric is evaluated on binary predictions for all possible thresholds instead of unthresholded predicted probabilities. The result is really hard to make any sense of. I think we should not allow this usage of TunedThresholdClassifierCV because it's really confusing. Computing ROC AUC or log loss on binary predictions should probably be considered a methodological mistake and giving tools to users that do that silently is a really bad usability pitfall.

My main conclusion is that we should make an official distinction of scores/metrics: probabilistic ones and decision metrics.

My main conclusion is that we should make an official distinction of scores/metrics: probabilistic ones and decision metrics.

There are also:

• non-probabilistic classification confidence scores (e.g. decision_function of an SVM classifier)
• continuous/regression metrics

I don't like that here we're accessing private _response_method as a proxy to see if it's the right kind of scorer or not. Scorers should have a public API for this (cc @StefanieSenger )

I agree but defining a good public API for scorers is way beyond the scope of this fix and would probably require a SLEP. We can always refactor TunedThresholdClassifierCV to use that improved public API once it exists.

Another think that scorer need is a public string name that we could use to report result of model evaluations (e.g. data frame column names) or in error and warning messages.

Also I would like to decouple greater_is_better flag from taking the negative value. This is so confusing to cross-validate on MSE and have to multiply by -1 to be able to report meaningful MSE numbers in a figure, table or notebook cell output.

I updated the PR to issue an informative warning for a pitfall Christian and I fell into when working on this the release highlights for this new class.

Arguably, passing a constant scoring score is very unlikely, but it would be far more likely to craft a scoring metric that does not have a trade-off between FP and FN and as a result would lead to dummy classifiers without noticing it.

I also added a TODO comment on the line in the code that would deserve to be updated once we improve the scoring API in 5a464e4.

Please let me know what you think.

Does this "bug" not exist anywhere else in the codebase? AFAIK we don't do a similar check in other places. My point is not that we shouldn't fix it. I'm saying I don't think an adhoc fix is the way to go.

I think we should fix incrementally and refactor on the way to trim code redundancy otherwise the perfect is the enemy of the good.

In an IRL chat with @glemaitre we figured it's the scorer that should be raising this error when it's being applied to an invalid data or estimator since it would have access to estimator as well, and the scorer knows its own internals well. It would also fix the issue in all other cases that we have.

Guillaume said he already has an old PR open about it.

Guillaume said he already has an old PR open about it.

Going back to the old PRs, the scope a bit different: the idea was to get the metric associated with the use case: #17889, #17930

While we would benefit from having the discussion around the API of scores/scorers and decoupling the signs, I think that we could improve the situation to have a kind of metric registry where we can define a callable, the supported y_type (for the y_true) and the compatible response_method. I think that just registry would allow to solve the problem of the original PR (providing some metrics depending on the problem) and the issue that we have here because we can track what are the requirements linked to a callable.

This would be limited to the scikit-learn metrics at first. Maybe there is a way in the future to expose something such that one can add their metric to the registry. Maybe make_scorer could have an option for it to automatically register.

Not sure why a registry would be better than public methods / attributes on the scorer object itself.

Not sure why a registry would be better than public methods / attributes on the scorer object itself.

I just thought about this exact question when I posted my previous message. If we don't want to touch the scorers API then the registry is an option but it does not mean this is a clean way. IMO, I would prefer to be part of the scorer API.

This is the reason I would merge the PR and trigger a new PR where we can discuss the right API and alternative.

I don’t intent to sound harsh, still: Could we incorporate Adrin‘s comments, do a bit less magic warnings here for the moment and discuss scorer API redesign elsewhere?

Could we incorporate Adrin‘s comments

That's what I tried in #29082 (comment).

Do you have more specific suggestions for things to improve?

do a bit less magic warnings here for the moment

So only keep the ValueError and remove the 2 warnings? I can at least split the PR in two if people prefer.

I think what I'm suggesting now doesn't require a SLEP, and divides the PR into pieces which we should be able to tackle easily and quickly. WDYT @ogrisel

Let me update the PR accordingly. I will split it in two so that we can have decoupled review discussions about the scorer object attribute and the warnings about trivial thresholds / constant thresholds.