Reference Issues/PRs

Fix #20678

What does this implement/fix? Explain your changes.

Parameter combinations that have a score of NaN should not be ranked higher than solutions with an actual score.

This switches to using np.nanargmax() to find the highest scores that are not NaN. In addition, when selecting the top-k parameter combinations we now rank combinations with a score of NaN lower than any solution with a score.

Any other comments?

I am not sure exactly how to provoke the failure. I've tested this with a fake estimator like the following:

class RandomlyFailingClassifier(BaseEstimator, ClassifierMixin):
    def __init__(self, n_estimators=10, some_param=2):
        self.some_param = some_param
        self.n_estimators = n_estimators
    def fit(self, X, y):
        if False: #random.random() < 0.5 and self.n_estimators != 10:
            raise Exception("waaahhh")
        else:
            return self
    def predict(self, X):
        if False and random.random() < .3:
            raise Exception("muuuuuuuh")
        y = random.choices([0, 1], k=X.shape[0])
        return np.array(y)

which allows you to have fit() and/or predict() fail at random. I think for a non regression test we need something better. Does someone have an idea?

Another thing I've noticed is that np.nanargmaxraises an exception if all elements passed to it are NaNs. Not quite sure what we should do in that case. Pick a random order?? Raise?