@NicolasHug Could you have a look at this PR. We quite change the __init__ and I would like to be sure that we did not overlook something.

Are the changes to refit needed?
I personally don't find the static method to be much more readable but maybe there's a good reason?

self.refit != refit passed during __init__ and it looks like against our API contract.

we'll need to remove _refit_callable I think.

I forgot to remove it indeed. Here, the idea was just to move at as a static method to be overwritten in successive halving.

And we might as well ensure that self.refit is a bool for SH.

Right, we can do that and add a test. A question regarding this point: is there a reason for not supporting multimeric?

self.refit != refit passed during init and it looks like against our API contract.

Hm, fair. Could we just set self.refit = refit after the call to super().__init__ then? No strong opinion, I'm fine with the proposed changes as well

is there a reason for not supporting multimeric?

I think it's because with multimetric scoring we would not know which scorer to use at each round to select the "survivors" for the next iteration. It sounds like refit could do that, but not really: what if you just want multimetric scoring with no refit?

Hm, fair. Could we just set self.refit = refit after the call to super().init then? No strong opinion, I'm fine with the proposed changes as well

Would not it be better to pass it to super (as now), and then add the check that we only support single metric with boolean in the parameter checks?

I think it's because with multimetric scoring we would not know which scorer to use at each round to select the "survivors" for the next iteration. It sounds like refit could do that, but not really: what if you just want multimetric scoring with no refit?

In grid-search, we use scoring to specify the multiple metrics and refit to choose which one to use to refit.
Indeed, I was just interested to know if methodologically there was a blocker. It seems more like an enhancement feature to have if we would like it at some point. This is enough of an answer for my curiosity :)

and then add the check that we only support single metric with boolean in the parameter checks?

yes I think this check will be useful regardless of the strategy!

In grid-search, we use scoring to specify the multiple metrics and refit to choose which one to use to refit.
Indeed, I was just interested to know if methodologically there was a blocker

There kinda is IMHO: we use refit in grid-search indeed but refit='accuracy' says "use this scorer when refitting the best estimator, at the very end of the search". Here we need something that says "use this scorer to select the k best candidates at each iteration". While it's similar to what refit does, the semantics and different enough for refit not to be a good solution I think

Also, are all of them needed? I would assume that column_or_1d could be enough here, but maybe I'm wrong

probably check_classification_targets as well. But it is true that the two others can be delayed when the validation will take place in the underlying classifier.

Let me make the changes.

Thanks @glemaitre and everybody else!