We want a solution that can accept a single random_state. It can generate random states for each KFold instance.

I also don't get why you should accept a KFold instance to the constructor of RepeatedKFold.

Yes, it would be better to accept n_splits to the constructor of RepeatedKFold and create an instance of KFold inside. Likewise for RepeatedStratifiedKFold.

I'll make the changes.

we are not certain it's the best design, but currently all the splitters do this in split, not in __init__

Is there any other way to achieve this other than initializing self.random_states = [] in __init__ and generate random states when split is called for the first time? The code 946-950 will move to split inside an if condition?

I think you should just mention RepeatStratifiedKFold here and under StratifiedKFold. Also in the "see also"s of relevant classes.

I.e. I think this is overkill

"for an arbitrary randomized CV splitter"

with different randomization

Why is this an ABC?

Removing.

why?

I think only KFold and StratifiedKFold use random_state. That's why I added this check. Should I remove it? And also, is there a way to check if cv is an instance of cross-validator with randomized split functionality?

Yes, remove it.

It's a private class, it doesn't require such validation.

I don't think we need this. n_repeats is already an attribute.

Removing from all classes.

I wonder if instead we should be constructing new CV objects in here (i.e. in split). Thus _RepeatedSplits.__init__ would take a constructor for cv rather than cv.

It might be nice as it would remove the dependency of KFold from RepeatedKFold in terms of parameters. I am thinking something like def __init__(self, cv, n_repeats=5, random_state=None, **cvargs):. It will be called as _RepeatedSplits(KFold, n_repeats, random_state, n_splits=n_splits). This is what you meant, right?

I have one doubt though. Since shuffle should always be True and random_state will be generated inside split function, would it be okay to just mention that user should not pass these arguments and one random_state that is passed does not correspond to random_state parameter of KFold?

You can have **cvargs in the _RepeatedSplits class while still only allowing specified named args in RepeatedKFold

please also check that a second call to split produces the same sets.

add a comment to explain why this is repeated.
perhaps use a loop or a helper function to avoid duplicated code.

you also don't check here that the iterator is exhausted after 4 elements

do this in a loop together with the RepeatedKFold case.

please remove blank line

add a comment to explain why this is repeated.
perhaps use a loop or a helper function to avoid duplicated code.

you also don't check here that the iterator is exhausted after 4 elements

Never check values in __init__. Move it to split.

Shouldn't error be thrown at the construction time if there is some discrepancy with the parameters passed? In _BaseKFold also, values are checked in __init__.

In sklearn for the estimator, we never check the error in init because of set_params but these classes are not estimators. I imagine this rule is not applied here.

@jnothman As I'm not 100% sure can you confirm that ?

For now, at least, CV splitters are a bit special in this regard. Checking in __init__ is consistent with other splitters.

ok thx @jnothman

Not an obligation as _RepeatedSplits is private but can you raise an error in split to check that ?

Maybe I'm missing something but why directly send rng to random_state ?

Integer random state generated by rng is sent as random_state. Do you have any other way in mind?

Sorry I was not clear.
Can you remove the line random_state = rng.randint(np.iinfo(np.int32).max) and change random_state by rng later?

Do you mean after creating the object for cv? If yes, how would it make a difference?

I mean : cv = self.cv(random_state=rng, shuffle=True, **self.cvargs)

Your code is similar to the code following :

In [1]: from sklearn.utils import check_random_state

In [2]: class Foo:
   ...:     def __init__(self, random_state):
   ...:         self.rng = check_random_state(random_state)
   ...:         
   ...:     def fit(self):
   ...:         print(self.rng.randint(1000))
   ...:         

In [3]: rng = check_random_state(0)

In [4]: f1 = Foo(rng)

In [5]: f2 = Foo(rng)

In [6]: f1.fit()
684

In [7]: f1.fit()
559

In [8]: f2.fit()
629

In [9]: f2.fit()
192

rng is an object and it will be modified every time you call cv.split. So for me is not necessary to generate a specific random_state at each iteration. Maybe I am missing something ?

check_random_state does not create a copy of rng if rng is a random_state.

I can't find any case which we'll miss by this approach. But I think current implementation keeps the use of random_state clean. I am not really sure.

@jnothman thoughts?

I think passing rng directly should be okay. If it's not okay, we need to be able to construct a test case that proves so!

I am not able to find any such testcase. So making the changes. Thanks!

This is consistent with the other estimators but seem pretty useless in practice. I would make 10 times 10-fold the default. Why would you want to do 3x5 instead of 10 fold?

Changing it to 5x10 (n_splits x n_repeats) by default. Is that fine?

yeah

I would say "it can be used to run KFold multiple times to increase the fidelity of the estimate? Or can we say to decrease the variance? Is that accurate?

Isn't this the cross validation class itself? That seems more natural than passing the __init__ method.

We are not passing the __init__ method. It's called as RepeatedSplits(KFold, n_repeats, random_state, n_splits=n_splits). I am changing description to "Cross-validator class.".

I would probably do 10x10 by default, or maybe 5 times 10 fold. This is something you use when you care about accuracy but not necessarily time.

Changing default values of n_splits to 5 and n_repeats to 10.

if set(cvargs).intersection({'random_state', 'shuffle'})? Though not really shorter :-/

Keeping the same as both are of equal length. :P