Rename to n_splits across the board? 😛

Hi @jnothman, are you serious?
For me, it sounds like a good solution 😄

I'm serious in that that's how I might name it it in hindsight, especially since we now have get_n_splits. Such change and deprecation should generally be avoided to not force users to change. However, if we were to make that change, now would be the time, given that users of this part of the codebase need to update their code anyway (sklearn.model_selection doesn't exist in master).

Note however, that there's also the further inconsistency with the ShuffleSplit family's use of n_iter for the same notion as n_folds. Indeed, for a consistent and unambiguous interface, [get_]n_iter may be preferable to [get_]n_splits. Unless there is wholesale agreement among devs that we should piggyback further consistency improvements on the model_selection change, I think the choice of parameter name should be limited to the new splitter.

Ordinarily, @GaelVaroquaux righteously champions a conservative position on such change, so it's useful to have his input on whether that piggybacking is at all worth considering.

Any update on this?

Would @GaelVaroquaux please share your opinions?
Thanks!

Sorry to interrupt, @jnothman and @MechCoder .
I think the functionality of time series cross validation is done.

Problem remains now is that get_n_splits() will return self.n_folds -1, which may be confused since users used to see self.n_folds as how many test folds.

See #7169, which I should have left to you to post, @yenchenlin !

@jnothman thanks! 😄

Please add to classes.rst

I wonder whether there's any benefit in allowing the user to sub-sample from the training set so that its size is not variable. You could imagine not wanting to choose model parameters that maximise the mean score, but a mean weighted to those with a larger sample of training data: if the amount of training data has an effect, cross-validation scores may not be normally distributed. Allowing the option to sub-sample to a fixed training set size does not fix this, though. It may be worth leaving a note for the user that due to learning curve effects the mean over CV scores might not be as informative as otherwise...

This needs some narrative documentation (i.e. content in cross_validation.rst), whose role is to guide the user to the kinds of problems involved in choosing a cross-validation strategy, and the available solutions.

Test failures pending rebase on #7187, I hope.

Ideally, this PR would also be supported by an example. Do we know of / have datasets where this would be appropriate?

Before this is merged, I think that it is very important that there is a section / subsection added in the model selection documentation on time series data. It doesn't have to be long, but it should point out the particularities of cross-validation on time series, and point to this object.

I temporarily pushing commits that rebase #7187 to make sure CI looks good.
Will resolve all these commits after #7187 is merged.

I have to admit that I'm not familiar with time-series data, and the only time-series data I've played with is a private property of a startup.

I remembered @MechCoder has some experiences with this, would you please suggest some toy dataset?

I have to admit that I'm not familiar with time-series data, and the only time-series data I've played with is a private property of a startup.

You don't really need to be familiar with time-series data explicitly, as long as you understand why something like KFold is problematic and why this solution is better. "A note on shuffling" already introduces problems related to non-iid data. Similarly describe the time series case where samples are not identically distributed, rather the distribution changes as a function of the dataset ordering.

I think we can merge after those minor changes, and a New Feature entry in whats_new.

We should also create an issue quoting @amueller's suggestion of a restructure of the docs. You are welcome to do this.

@yenchenlin , can I suggest you install a pep8 checker?
flake8 tends to work well.

@jnothman I feel embarrassed, I do have one ...

I had no intention to embarrass you. Here's a further hint:

~/repos/scikit-learn$ cat .git/hooks/pre-commit
#!/bin/sh

FILES=$(git diff --cached --name-status | grep -v ^D | awk '$1 $2 { print $2}' | grep -e .py$)
if [ -n "$FILES" ]; then
pep8 -r $FILES
fi

use git commit -n to bypass its warnings. (Though I'm sure that could be done without the greps and I can't be bothered to change that now...)

Hurrah! 🍻

Ugh. forgot to add to classes.rst

Now that this is merged, I've realised that it's a bit unconventional for us to have the suffix CV on a splitter, and it has an altogether different conventional meaning. Should this be TimeSeriesSplit or TimeSeriesFold? @amueller??

True. I'd say TimeSeriesSplit

Wanna add to classes and do the rename?

Thanks a lot @yenchenlin . Yes, I agree that it should be TimeSeriesSplit

@yenchenlin are you able to make a patch for these afterthoughts?

@jnothman sure!

TimeSeriesSplitter maybe?

I'm not sure I understand what HeterogeneousTimeSeries means. It was mentioned here and in #6322. I think it means that the time stamps are not equally spaced.

I've implemented what I'm calling MultiTimeSeriesSplit for the case of multiple time-series with different time stamps present in different time-series (or, said differently, with missing data). I compute splits by calculating equally spaced quantiles of the distribution of time stamps.

Here's an illustration for a small dataset. There are four time-series (four countries), and not all countries have data in every year (e.g., it's common to have countries be "born" in the middle of a dataset).

Would this be of interest?

@yenchenlin what's the status of HeterogeneousTimeSeriesSplit (or whatever is the latest name)?

Update (March 2019): The MultiTimeSeriesSplitter I wrote is here. It ended up using quantiles using this function to decide where to make splits, with the option of making the first split larger (e.g., "I want the first fold to be the earliest 60% of the data, the second fold to be the earliest 70% of the data, etc."). This class has the possibly undesirable behavior of putting some records at time t in one fold and other records at time t in another fold. The benefit, however, was that the numbers of records in the folds are more similar. The code got a bit complicated and more tied to my use case than I would have liked.

@jnothman also check out the discussion here: #6322